Decision Workspace
tpt-tokenizers-fast vs splintr vs toktok-rs
Side-by-side comparison of Rust crates
41
tpt-tokenizers-fast
experimentalv0.1.0
Lightweight, no_std-compatible BPE/SentencePiece tokenization
51
splintr
experimentalv0.21.0
Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings
61
toktok-rs
experimentalv0.1.4
Fast, exact BPE tokenizer for OpenAI encodings (cl100k_base, o200k_base, o200k_harmony) — tiktoken-identical ids, no dependencies
Core Metrics
| tpt-tokenizers-fast | splintr | toktok-rs | |
|---|---|---|---|
| Health Score | 41 | 51 | 61 |
| Total Downloads | 15 | 35.0K | 1.1K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 0 | 23 | 0 |
| Releases | 1 | 36 | 4 |
| Last Updated | 79d ago | 12d ago | 37d ago |
| Age | 2m | 10m | 1m |
Health Breakdown
tpt-tokenizers-fast
Maintenance
9
Quality
10
Community
6
Popularity
1
Documentation
15
splintr
Maintenance
18
Quality
8
Community
10
Popularity
5
Documentation
10
toktok-rs
Maintenance
17
Quality
14
Community
11
Popularity
4
Documentation
15
Technical Details
| tpt-tokenizers-fast | splintr | toktok-rs | |
|---|---|---|---|
| Version | 0.1.0 | 0.21.0 | 0.1.4 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT OR Apache-2.0 | MIT | MIT |
| Dependencies | 1 | 31 | 0 |
| Crate Size | 5KB | 612KB | 2.3MB |
| Features | 4 | 23 | 2 |
| Yanked % | 100.0% | 13.9% | 0.0% |
| Edition | 2021 | 2021 | 2021 |
| MSRV | 1.75 | — | 1.70 |
| Owners | 1 | 1 | 2 (team) |
Links
Quick Verdict
- •toktok-rs leads with a health score of 61/100, but none of the options score above 80.
- •splintr has the most downloads (35.0K), suggesting wider adoption.