Decision Workspace
miktik vs llm-tokenizer vs dynamo-tokenizers
Side-by-side comparison of Rust crates
57
miktik
experimentalv0.4.3
A unified, multi-backend tokenizer library for LLMs
65
llm-tokenizer
growingv1.9.0
LLM tokenizer library with caching and chat template support
63
dynamo-tokenizers
experimentalv3.1.0
Standalone HuggingFace, tiktoken, fastokens, and Baseten tokenizer implementations for LLM inference serving.
Core Metrics
| miktik | llm-tokenizer | dynamo-tokenizers | |
|---|---|---|---|
| Health Score | 57 | 65 | 63 |
| Total Downloads | 5.6K | 2.0M | 818.0K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 0 | 44 | 51 |
| Releases | 11 | 15 | 24 |
| Last Updated | 36d ago | 1d ago | 3d ago |
| Age | 7m | 9m | 5m |
Health Breakdown
miktik
Maintenance
17
Quality
16
Community
5
Popularity
4
Documentation
15
llm-tokenizer
Maintenance
22
Quality
14
Community
11
Popularity
8
Documentation
10
dynamo-tokenizers
Maintenance
20
Quality
14
Community
12
Popularity
7
Documentation
10
Technical Details
| miktik | llm-tokenizer | dynamo-tokenizers | |
|---|---|---|---|
| Version | 0.4.3 | 1.9.0 | 3.1.0 |
| Stable (≥1.0) | ✗ No | ✓ Yes | ✓ Yes |
| License | MIT | Apache-2.0 | Apache-2.0 |
| Dependencies | 4 | 24 | 16 |
| Crate Size | 24KB | 242KB | 76KB |
| Features | 6 | 0 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2021 | 2024 |
| MSRV | 1.86 | — | — |
| Owners | 1 | 1 | 4 |
Links
Quick Verdict
- •llm-tokenizer leads with a health score of 65/100, but none of the options score above 80.
- •miktik is pre-1.0 — API may change.