Decision Workspace
trustformers-tokenizers vs kitoken vs tokie
Side-by-side comparison of Rust crates
55
trustformers-tokenizers
experimentalv0.2.1
Tokenizers for TrustformeRS
54
kitoken
growingv0.11.0
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
54
tokie
experimentalv0.1.4
Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace
Core Metrics
| trustformers-tokenizers | kitoken | tokie | |
|---|---|---|---|
| Health Score | 55 | 54 | 54 |
| Total Downloads | 793 | 46.9K | 1.9K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 18 | 23 | 36 |
| Releases | 10 | 3 | 17 |
| Last Updated | 45d ago | 152d ago | 77d ago |
| Age | 11m | 1y 9m | 6m |
Health Breakdown
trustformers-tokenizers
Maintenance
17
Quality
11
Community
9
Popularity
3
Documentation
15
kitoken
Maintenance
10
Quality
16
Community
10
Popularity
6
Documentation
12
tokie
Maintenance
15
Quality
12
Community
10
Popularity
4
Documentation
13
Technical Details
| trustformers-tokenizers | kitoken | tokie | |
|---|---|---|---|
| Version | 0.2.1 | 0.11.0 | 0.1.4 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | Apache-2.0 | BSD-2-Clause | MIT OR Apache-2.0 |
| Dependencies | 35 | 24 | 21 |
| Crate Size | 571KB | 64KB | 185KB |
| Features | 9 | 21 | 3 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2024 | 2024 |
| MSRV | 1.89 | 1.86.0 | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •trustformers-tokenizers leads with a health score of 55/100, but none of the options score above 80.
- •kitoken has the most downloads (46.9K), suggesting wider adoption.