Decision Workspace
tokens vs itok vs simd-tokenizer
Side-by-side comparison of Rust crates
40
tokens
experimentalv0.1.0
Whitespace and quoted-strings tokenizer
51
itok
experimentalv0.3.1
context-cost estimator: token and window budgets for files and changes
51
simd-tokenizer
experimentalv0.1.2
Safe SWAR-accelerated whitespace token estimator with an optional exact tiktoken (cl100k_base, o200k_base) backend.
Core Metrics
| tokens | itok | simd-tokenizer | |
|---|---|---|---|
| Health Score | 40 | 51 | 51 |
| Total Downloads | 27 | 270 | 1.2K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 0 | 8 | 0 |
| Releases | 1 | 3 | 3 |
| Last Updated | 112d ago | 49d ago | 28d ago |
| Age | 3m | 1m | 1m |
Health Breakdown
tokens
Maintenance
7
Quality
15
Community
3
Popularity
2
Documentation
13
itok
Maintenance
11
Quality
13
Community
9
Popularity
3
Documentation
15
simd-tokenizer
Maintenance
12
Quality
15
Community
5
Popularity
4
Documentation
15
Technical Details
| tokens | itok | simd-tokenizer | |
|---|---|---|---|
| Version | 0.1.0 | 0.3.1 | 0.1.2 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT OR Apache-2.0 | MIT | Apache-2.0 |
| Dependencies | 0 | 10 | 5 |
| Crate Size | 7KB | 286KB | 35KB |
| Features | 0 | 4 | 3 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2024 | 2021 |
| MSRV | — | 1.95 | 1.85 |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •itok leads with a health score of 51/100, but none of the options score above 80.
- •simd-tokenizer has the most downloads (1.2K), suggesting wider adoption.