Decision Workspace
tiktoken vs toktok-rs vs toklab-core
Side-by-side comparison of Rust crates
61
tiktoken
stablev4.1.2
A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
61
toktok-rs
experimentalv0.1.4
Fast, exact BPE tokenizer for OpenAI encodings (cl100k_base, o200k_base, o200k_harmony) — tiktoken-identical ids, no dependencies
46
toklab-core
experimentalv0.1.1
Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings.
Core Metrics
| tiktoken | toktok-rs | toklab-core | |
|---|---|---|---|
| Health Score | 61 | 61 | 46 |
| Total Downloads | 57.8K | 1.1K | 40 |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 245 | 0 | 0 |
| Releases | 26 | 4 | 2 |
| Last Updated | 57d ago | 37d ago | 152d ago |
| Age | 3y 1m | 1m | 5m |
Health Breakdown
tiktoken
Maintenance
16
Quality
13
Community
13
Popularity
6
Documentation
13
toktok-rs
Maintenance
17
Quality
14
Community
11
Popularity
4
Documentation
15
toklab-core
Maintenance
10
Quality
13
Community
6
Popularity
2
Documentation
15
Technical Details
| tiktoken | toktok-rs | toklab-core | |
|---|---|---|---|
| Version | 4.1.2 | 0.1.4 | 0.1.1 |
| Stable (≥1.0) | ✓ Yes | ✗ No | ✗ No |
| License | MIT OR Apache-2.0 | MIT | MIT OR Apache-2.0 |
| Dependencies | 9 | 0 | 4 |
| Crate Size | 5.3MB | 2.3MB | 7KB |
| Features | 23 | 2 | 0 |
| Yanked % | 26.9% | 0.0% | 0.0% |
| Edition | 2024 | 2021 | 2021 |
| MSRV | — | 1.70 | 1.75 |
| Owners | 1 | 2 (team) | 1 |
Links
Quick Verdict
- •tiktoken leads with a health score of 61/100, but none of the options score above 80.
- •tiktoken is depended on by 245 crates — strongest ecosystem trust.
- •toktok-rs, toklab-core are pre-1.0 — API may change.