Decision Workspace
toklab-core vs riptoken vs tiktoken
Side-by-side comparison of Rust crates
46
toklab-core
experimentalv0.1.1
Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings.
59
riptoken
experimentalv0.3.0
Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken
55
tiktoken
growingv3.5.1
A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Core Metrics
| toklab-core | riptoken | tiktoken | |
|---|---|---|---|
| Health Score | 46 | 59 | 55 |
| Total Downloads | 34 | 99.9K | 18.7K |
| 30d Downloads | 0 | 47.1K | 3.8K |
| Dependents | 0 | 0 | 77 |
| Releases | 2 | 8 | 15 |
| Last Updated | 64d ago | 92d ago | 36d ago |
| Age | 2m | 3m | 2y 11m |
Health Breakdown
toklab-core
Maintenance
10
Quality
13
Community
6
Popularity
2
Documentation
15
riptoken
Maintenance
18
Quality
14
Community
6
Popularity
6
Documentation
15
tiktoken
Maintenance
16
Quality
10
Community
11
Popularity
5
Documentation
13
Technical Details
| toklab-core | riptoken | tiktoken | |
|---|---|---|---|
| Version | 0.1.1 | 0.3.0 | 3.5.1 |
| Stable (≥1.0) | ✗ No | ✗ No | ✓ Yes |
| License | MIT OR Apache-2.0 | MIT | MIT OR Apache-2.0 |
| Dependencies | 4 | 9 | 9 |
| Crate Size | 7KB | 41KB | 6.1MB |
| Features | 0 | 3 | 2 |
| Yanked % | 0.0% | 0.0% | 46.7% |
| Edition | 2021 | 2024 | 2024 |
| MSRV | 1.75 | 1.85 | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •riptoken leads with a health score of 59/100, but none of the options score above 80.
- •toklab-core, riptoken are pre-1.0 — API may change.