Decision Workspace
bbpe vs tokenizers vs tokenizers-enfer
Side-by-side comparison of Rust crates
51
bbpe
experimentalv0.6.3
Binary byte pair encoding (BPE) trainer and CLI compatible with Hugging Face tokenizers
60
tokenizers
growingv0.22.2
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
38
tokenizers-enfer
growingv0.21.1
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Core Metrics
| bbpe | tokenizers | tokenizers-enfer | |
|---|---|---|---|
| Health Score | 51 | 60 | 38 |
| Total Downloads | 999 | 12.7M | 3.8K |
| 30d Downloads | 15 | 2.0M | 5 |
| Dependents | 0 | 3.6K | 0 |
| Releases | 13 | 39 | 3 |
| Last Updated | 124d ago | 115d ago | 477d ago |
| Age | 5m | 6y 7m | 1y 3m |
Health Breakdown
bbpe
Maintenance
13
Quality
13
Community
6
Popularity
4
Documentation
15
tokenizers
Maintenance
12
Quality
12
Community
16
Popularity
8
Documentation
12
tokenizers-enfer
Maintenance
6
Quality
13
Community
3
Popularity
4
Documentation
12
Technical Details
| bbpe | tokenizers | tokenizers-enfer | |
|---|---|---|---|
| Version | 0.6.3 | 0.22.2 | 0.21.1 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 |
| Dependencies | 22 | 33 | 31 |
| Crate Size | 88KB | 186KB | 185KB |
| Features | 2 | 6 | 5 |
| Yanked % | 0.0% | 2.6% | 0.0% |
| Edition | 2021 | 2018 | 2018 |
| MSRV | 1.74 | — | — |
| Owners | 1 | 4 | 1 |
Links
Quick Verdict
- •tokenizers leads with a health score of 60/100, but none of the options score above 80.
- •tokenizers is depended on by 3.6K crates — strongest ecosystem trust.
- •⚠ tokenizers-enfer has not been updated in over a year.