Decision Workspace
rust_tokenizers vs sentencepiece-model vs rusty-tokenizer
Side-by-side comparison of Rust crates
49
rust_tokenizers
growingv8.1.1
High performance tokenizers for Rust
46
sentencepiece-model
growingv0.1.4
SentencePiece model parser generated from the SentencePiece protobuf definition
45
rusty-tokenizer
experimentalv0.1.0
A Rust implementation of word, character, and Byte Pair Encoding (BPE) tokenizers.
Core Metrics
| rust_tokenizers | sentencepiece-model | rusty-tokenizer | |
|---|---|---|---|
| Health Score | 49 | 46 | 45 |
| Total Downloads | 452.3K | 747.5K | 9 |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 117 | 9 | 0 |
| Releases | 34 | 5 | 1 |
| Last Updated | 1105d ago | 731d ago | 1d ago |
| Age | 6y 7m | 2y 10m | 1d |
Health Breakdown
rust_tokenizers
Maintenance
6
Quality
18
Community
9
Popularity
7
Documentation
9
sentencepiece-model
Maintenance
4
Quality
16
Community
9
Popularity
7
Documentation
10
rusty-tokenizer
Maintenance
13
Quality
15
Community
6
Popularity
1
Documentation
10
Technical Details
| rust_tokenizers | sentencepiece-model | rusty-tokenizer | |
|---|---|---|---|
| Version | 8.1.1 | 0.1.4 | 0.1.0 |
| Stable (≥1.0) | ✓ Yes | ✗ No | ✗ No |
| License | Apache-2.0 | BSD-2-Clause | MIT |
| Dependencies | 17 | 4 | 0 |
| Crate Size | 134KB | 8KB | 6KB |
| Features | 4 | 2 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2018 | 2021 | 2024 |
| MSRV | — | — | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •rust_tokenizers leads with a health score of 49/100, but none of the options score above 80.
- •sentencepiece-model has the most downloads (747.5K), suggesting wider adoption.
- •rust_tokenizers is depended on by 117 crates — strongest ecosystem trust.
- •⚠ rust_tokenizers, sentencepiece-model have not been updated in over a year.
- •sentencepiece-model, rusty-tokenizer are pre-1.0 — API may change.