Decision Workspace
tokenfold-core vs tokenfold-cli vs llm-tokenizer
Side-by-side comparison of Rust crates
52
tokenfold-core
experimentalv0.5.1
Token-aware compression for LLM payloads: shrink JSON tool-call bodies, command output, and diffs with exact tiktoken accounting and a typed safety report.
47
tokenfold-cli
experimentalv0.5.1
Portable CLI for token-aware compression of LLM payloads.
65
llm-tokenizer
growingv1.9.0
LLM tokenizer library with caching and chat template support
Core Metrics
| tokenfold-core | tokenfold-cli | llm-tokenizer | |
|---|---|---|---|
| Health Score | 52 | 47 | 65 |
| Total Downloads | 356 | 138 | 2.0M |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 26 | 0 | 44 |
| Releases | 10 | 8 | 15 |
| Last Updated | 7d ago | 7d ago | 1d ago |
| Age | 2m | 2m | 9m |
Health Breakdown
tokenfold-core
Maintenance
14
Quality
13
Community
10
Popularity
3
Documentation
12
tokenfold-cli
Maintenance
14
Quality
12
Community
6
Popularity
3
Documentation
12
llm-tokenizer
Maintenance
22
Quality
14
Community
11
Popularity
8
Documentation
10
Technical Details
| tokenfold-core | tokenfold-cli | llm-tokenizer | |
|---|---|---|---|
| Version | 0.5.1 | 0.5.1 | 1.9.0 |
| Stable (≥1.0) | ✗ No | ✗ No | ✓ Yes |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 |
| Dependencies | 9 | 10 | 24 |
| Crate Size | 169KB | 66KB | 242KB |
| Features | 2 | 0 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2024 | 2021 |
| MSRV | 1.98 | 1.98 | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •llm-tokenizer leads with a health score of 65/100, but none of the options score above 80.
- •tokenfold-core, tokenfold-cli are pre-1.0 — API may change.