Decision Workspace
riptoken vs oxibonsai-tokenizer vs toklab-core
Side-by-side comparison of Rust crates
53
riptoken
experimentalv0.3.0
Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken
53
oxibonsai-tokenizer
experimentalv0.1.4
Pure Rust BPE tokenizer for OxiBonsai (MeCrab-compatible)
49
toklab-core
experimentalv0.1.1
Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings.
Core Metrics
| riptoken | oxibonsai-tokenizer | toklab-core | |
|---|---|---|---|
| Health Score | 53 | 53 | 49 |
| Total Downloads | 3.9K | 286 | 27 |
| 30d Downloads | 3.8K | 0 | 0 |
| Dependents | 0 | 18 | 0 |
| Releases | 8 | 5 | 2 |
| Last Updated | 44d ago | 9d ago | 15d ago |
| Age | 1m | 1m | 16d |
Health Breakdown
riptoken
Maintenance
14
Quality
14
Community
6
Popularity
4
Documentation
15
oxibonsai-tokenizer
Maintenance
15
Quality
13
Community
9
Popularity
3
Documentation
13
toklab-core
Maintenance
13
Quality
13
Community
6
Popularity
2
Documentation
15
Technical Details
| riptoken | oxibonsai-tokenizer | toklab-core | |
|---|---|---|---|
| Version | 0.3.0 | 0.1.4 | 0.1.1 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT | Apache-2.0 | MIT OR Apache-2.0 |
| Dependencies | 9 | 6 | 4 |
| Crate Size | 41KB | 83KB | 7KB |
| Features | 3 | 0 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2021 | 2021 |
| MSRV | 1.85 | — | 1.75 |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •riptoken leads with a health score of 53/100, but none of the options score above 80.