Decision Workspace
rust_tokenizers vs sentencepiece-model vs fibpetokenizer
Side-by-side comparison of Rust crates
49
rust_tokenizers
growingv8.1.1
High performance tokenizers for Rust
46
sentencepiece-model
growingv0.1.4
SentencePiece model parser generated from the SentencePiece protobuf definition
46
fibpetokenizer
experimentalv0.1.0
A blazing fast Byte Pair Encoding (BPE) tokenizer library with Python bindings
Core Metrics
| rust_tokenizers | sentencepiece-model | fibpetokenizer | |
|---|---|---|---|
| Health Score | 49 | 46 | 46 |
| Total Downloads | 361.3K | 311.3K | 14 |
| 30d Downloads | 7.9K | 238.7K | 14 |
| Dependents | 117 | 6 | 0 |
| Releases | 34 | 5 | 1 |
| Last Updated | 908d ago | 535d ago | 24d ago |
| Age | 6y 1m | 2y 4m | 24d |
Health Breakdown
rust_tokenizers
Maintenance
6
Quality
18
Community
9
Popularity
7
Documentation
9
sentencepiece-model
Maintenance
5
Quality
16
Community
8
Popularity
7
Documentation
10
fibpetokenizer
Maintenance
11
Quality
15
Community
6
Popularity
1
Documentation
13
Technical Details
| rust_tokenizers | sentencepiece-model | fibpetokenizer | |
|---|---|---|---|
| Version | 8.1.1 | 0.1.4 | 0.1.0 |
| Stable (≥1.0) | ✓ Yes | ✗ No | ✗ No |
| License | Apache-2.0 | BSD-2-Clause | MIT OR Apache-2.0 |
| Dependencies | 17 | 4 | 6 |
| Crate Size | 134KB | 8KB | 24KB |
| Features | 4 | 2 | 2 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2018 | 2021 | 2021 |
| MSRV | — | — | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •rust_tokenizers leads with a health score of 49/100, but none of the options score above 80.
- •rust_tokenizers is depended on by 117 crates — strongest ecosystem trust.
- •⚠ rust_tokenizers, sentencepiece-model have not been updated in over a year.
- •sentencepiece-model, fibpetokenizer are pre-1.0 — API may change.