Decision Workspace
char-token-est vs tokenx vs utokenizer
Side-by-side comparison of Rust crates
43
char-token-est
experimentalv0.1.0
Tokenless byte/char-based token-count estimator for LLM prompts. Per-model-family calibration for Claude, GPT, Gemini, Llama. Zero deps.
43
tokenx
experimentalv0.2.0
vocab-free heuristic LLM token-count estimation
42
utokenizer
experimentalv0.1.1
CLI tool for building a local model-tokenizer registry and counting input tokens across model families.
Core Metrics
| char-token-est | tokenx | utokenizer | |
|---|---|---|---|
| Health Score | 43 | 43 | 42 |
| Total Downloads | 27 | 49 | 43 |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 0 | 0 | 0 |
| Releases | 1 | 2 | 2 |
| Last Updated | 146d ago | 94d ago | 176d ago |
| Age | 4m | 3m | 5m |
Health Breakdown
char-token-est
Maintenance
6
Quality
15
Community
5
Popularity
2
Documentation
15
tokenx
Maintenance
12
Quality
13
Community
6
Popularity
2
Documentation
10
utokenizer
Maintenance
9
Quality
12
Community
6
Popularity
2
Documentation
13
Technical Details
| char-token-est | tokenx | utokenizer | |
|---|---|---|---|
| Version | 0.1.0 | 0.2.0 | 0.1.1 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT OR Apache-2.0 | MIT | MIT |
| Dependencies | 0 | 1 | 9 |
| Crate Size | 8KB | 153KB | 33KB |
| Features | 0 | 0 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2021 | 2021 |
| MSRV | 1.74 | — | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •char-token-est leads with a health score of 43/100, but none of the options score above 80.
- •tokenx has the most downloads (49), suggesting wider adoption.