Decision Workspace
sentencepiece-rs vs kitoken vs tokenizers
Side-by-side comparison of Rust crates
55
sentencepiece-rs
experimentalv0.2.2
Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding.
58
kitoken
growingv0.11.0
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
64
tokenizers
growingv0.23.1
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Core Metrics
| sentencepiece-rs | kitoken | tokenizers | |
|---|---|---|---|
| Health Score | 55 | 58 | 64 |
| Total Downloads | 759 | 38.7K | 17.0M |
| 30d Downloads | 0 | 0 | 2.5M |
| Dependents | 11 | 9 | 5.5K |
| Releases | 4 | 3 | 40 |
| Last Updated | 5d ago | 15d ago | 28d ago |
| Age | 15d | 1y 5m | 6y 9m |
Health Breakdown
sentencepiece-rs
Maintenance
16
Quality
15
Community
8
Popularity
3
Documentation
13
kitoken
Maintenance
16
Quality
16
Community
9
Popularity
5
Documentation
12
tokenizers
Maintenance
16
Quality
12
Community
16
Popularity
8
Documentation
12
Technical Details
| sentencepiece-rs | kitoken | tokenizers | |
|---|---|---|---|
| Version | 0.2.2 | 0.11.0 | 0.23.1 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | Apache-2.0 | BSD-2-Clause | Apache-2.0 |
| Dependencies | 0 | 24 | 33 |
| Crate Size | 22KB | 64KB | 196KB |
| Features | 0 | 21 | 6 |
| Yanked % | 0.0% | 0.0% | 2.5% |
| Edition | 2024 | 2024 | 2018 |
| MSRV | — | 1.86.0 | — |
| Owners | 1 | 1 | 4 |
Links
Quick Verdict
- •tokenizers leads with a health score of 64/100, but none of the options score above 80.
- •tokenizers is depended on by 5.5K crates — strongest ecosystem trust.