Decision Workspace
kitoken vs tokenizers vs sentencepiece-rs
Side-by-side comparison of Rust crates
58
kitoken
growingv0.11.0
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
64
tokenizers
growingv0.23.1
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
55
sentencepiece-rs
experimentalv0.2.2
Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding.
Core Metrics
| kitoken | tokenizers | sentencepiece-rs | |
|---|---|---|---|
| Health Score | 58 | 64 | 55 |
| Total Downloads | 38.7K | 17.0M | 759 |
| 30d Downloads | 0 | 2.5M | 0 |
| Dependents | 9 | 5.5K | 11 |
| Releases | 3 | 40 | 4 |
| Last Updated | 15d ago | 28d ago | 5d ago |
| Age | 1y 5m | 6y 9m | 15d |
Health Breakdown
kitoken
Maintenance
16
Quality
16
Community
9
Popularity
5
Documentation
12
tokenizers
Maintenance
16
Quality
12
Community
16
Popularity
8
Documentation
12
sentencepiece-rs
Maintenance
16
Quality
15
Community
8
Popularity
3
Documentation
13
Technical Details
| kitoken | tokenizers | sentencepiece-rs | |
|---|---|---|---|
| Version | 0.11.0 | 0.23.1 | 0.2.2 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | BSD-2-Clause | Apache-2.0 | Apache-2.0 |
| Dependencies | 24 | 33 | 0 |
| Crate Size | 64KB | 196KB | 22KB |
| Features | 21 | 6 | 0 |
| Yanked % | 0.0% | 2.5% | 0.0% |
| Edition | 2024 | 2018 | 2024 |
| MSRV | 1.86.0 | — | — |
| Owners | 1 | 4 | 1 |
Links
Quick Verdict
- •tokenizers leads with a health score of 64/100, but none of the options score above 80.
- •tokenizers is depended on by 5.5K crates — strongest ecosystem trust.