Decision Workspace
wordchipper vs tiktoken-rs vs tiktoken
Side-by-side comparison of Rust crates
49
wordchipper
experimentalv0.9.2
HPC Rust LLM Tokenizer Library
65
tiktoken-rs
growingv0.12.1
Library for encoding and decoding with the tiktoken library in Rust
61
tiktoken
stablev4.1.2
A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Core Metrics
| wordchipper | tiktoken-rs | tiktoken | |
|---|---|---|---|
| Health Score | 49 | 65 | 61 |
| Total Downloads | 2.4K | 20.2M | 57.8K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 80 | 8.7K | 245 |
| Releases | 29 | 36 | 26 |
| Last Updated | 148d ago | 15d ago | 57d ago |
| Age | 8m | 3y 8m | 3y 1m |
Health Breakdown
wordchipper
Maintenance
12
Quality
13
Community
8
Popularity
4
Documentation
12
tiktoken-rs
Maintenance
18
Quality
12
Community
12
Popularity
8
Documentation
15
tiktoken
Maintenance
16
Quality
13
Community
13
Popularity
6
Documentation
13
Technical Details
| wordchipper | tiktoken-rs | tiktoken | |
|---|---|---|---|
| Version | 0.9.2 | 0.12.1 | 4.1.2 |
| Stable (≥1.0) | ✗ No | ✗ No | ✓ Yes |
| License | MIT | MIT | MIT OR Apache-2.0 |
| Dependencies | 30 | 8 | 9 |
| Crate Size | 265KB | 3.8MB | 5.3MB |
| Features | 13 | 2 | 23 |
| Yanked % | 0.0% | 0.0% | 26.9% |
| Edition | 2024 | 2024 | 2024 |
| MSRV | 1.93.0 | 1.85.0 | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •tiktoken-rs leads with a health score of 65/100, but none of the options score above 80.
- •tiktoken-rs is depended on by 8.7K crates — strongest ecosystem trust.
- •wordchipper, tiktoken-rs are pre-1.0 — API may change.