Decision Workspace
tiktoken vs tiktokenx vs tiktoken-rs
Side-by-side comparison of Rust crates
55
tiktoken
growingv3.1.2
A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
42
tiktokenx
experimentalv0.1.0
A high-performance Rust implementation of OpenAI's tiktoken library
59
tiktoken-rs
growingv0.9.1
Library for encoding and decoding with the tiktoken library in Rust
Core Metrics
| tiktoken | tiktokenx | tiktoken-rs | |
|---|---|---|---|
| Health Score | 55 | 42 | 59 |
| Total Downloads | 5.7K | 399 | 5.3M |
| 30d Downloads | 1.3K | 3 | 823.9K |
| Dependents | 12 | 0 | 3.8K |
| Releases | 8 | 1 | 32 |
| Last Updated | 20d ago | 217d ago | 138d ago |
| Age | 2y 7m | 7m | 3y 1m |
Health Breakdown
tiktoken
Maintenance
17
Quality
10
Community
9
Popularity
4
Documentation
15
tiktokenx
Maintenance
4
Quality
14
Community
6
Popularity
3
Documentation
15
tiktoken-rs
Maintenance
12
Quality
12
Community
12
Popularity
8
Documentation
15
Technical Details
| tiktoken | tiktokenx | tiktoken-rs | |
|---|---|---|---|
| Version | 3.1.2 | 0.1.0 | 0.9.1 |
| Stable (≥1.0) | ✓ Yes | ✗ No | ✗ No |
| License | MIT | MIT | MIT |
| Dependencies | 9 | 8 | 9 |
| Crate Size | 6.1MB | 33KB | 3.8MB |
| Features | 2 | 3 | 2 |
| Yanked % | 87.5% | 0.0% | 0.0% |
| Edition | 2024 | 2024 | 2021 |
| MSRV | 1.94 | 1.85.0 | 1.61.0 |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •tiktoken-rs leads with a health score of 59/100, but none of the options score above 80.
- •tiktoken-rs is depended on by 3.8K crates — strongest ecosystem trust.
- •tiktokenx, tiktoken-rs are pre-1.0 — API may change.