Decision Workspace
sentencepiece-model vs rust_tokenizers vs fibpetokenizer
Side-by-side comparison of Rust crates
46
sentencepiece-model
growingv0.1.4
SentencePiece model parser generated from the SentencePiece protobuf definition
49
rust_tokenizers
growingv8.1.1
High performance tokenizers for Rust
46
fibpetokenizer
experimentalv0.1.0
A blazing fast Byte Pair Encoding (BPE) tokenizer library with Python bindings
Core Metrics
| sentencepiece-model | rust_tokenizers | fibpetokenizer | |
|---|---|---|---|
| Health Score | 46 | 49 | 46 |
| Total Downloads | 311.3K | 361.3K | 14 |
| 30d Downloads | 238.7K | 7.9K | 14 |
| Dependents | 6 | 117 | 0 |
| Releases | 5 | 34 | 1 |
| Last Updated | 535d ago | 908d ago | 24d ago |
| Age | 2y 4m | 6y 1m | 24d |
Health Breakdown
sentencepiece-model
Maintenance
5
Quality
16
Community
8
Popularity
7
Documentation
10
rust_tokenizers
Maintenance
6
Quality
18
Community
9
Popularity
7
Documentation
9
fibpetokenizer
Maintenance
11
Quality
15
Community
6
Popularity
1
Documentation
13
Technical Details
| sentencepiece-model | rust_tokenizers | fibpetokenizer | |
|---|---|---|---|
| Version | 0.1.4 | 8.1.1 | 0.1.0 |
| Stable (≥1.0) | ✗ No | ✓ Yes | ✗ No |
| License | BSD-2-Clause | Apache-2.0 | MIT OR Apache-2.0 |
| Dependencies | 4 | 17 | 6 |
| Crate Size | 8KB | 134KB | 24KB |
| Features | 2 | 4 | 2 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2018 | 2021 |
| MSRV | — | — | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •rust_tokenizers leads with a health score of 49/100, but none of the options score above 80.
- •rust_tokenizers is depended on by 117 crates — strongest ecosystem trust.
- •⚠ sentencepiece-model, rust_tokenizers have not been updated in over a year.
- •sentencepiece-model, fibpetokenizer are pre-1.0 — API may change.