Decision Workspace
llm-tokenizer vs dynamo-tokenizers vs miktik
Side-by-side comparison of Rust crates
65
llm-tokenizer
growingv1.9.0
LLM tokenizer library with caching and chat template support
63
dynamo-tokenizers
experimentalv3.1.0
Standalone HuggingFace, tiktoken, fastokens, and Baseten tokenizer implementations for LLM inference serving.
57
miktik
experimentalv0.4.3
A unified, multi-backend tokenizer library for LLMs
Core Metrics
| llm-tokenizer | dynamo-tokenizers | miktik | |
|---|---|---|---|
| Health Score | 65 | 63 | 57 |
| Total Downloads | 2.0M | 818.0K | 5.6K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 44 | 51 | 0 |
| Releases | 15 | 24 | 11 |
| Last Updated | 1d ago | 3d ago | 36d ago |
| Age | 9m | 5m | 7m |
Health Breakdown
llm-tokenizer
Maintenance
22
Quality
14
Community
11
Popularity
8
Documentation
10
dynamo-tokenizers
Maintenance
20
Quality
14
Community
12
Popularity
7
Documentation
10
miktik
Maintenance
17
Quality
16
Community
5
Popularity
4
Documentation
15
Technical Details
| llm-tokenizer | dynamo-tokenizers | miktik | |
|---|---|---|---|
| Version | 1.9.0 | 3.1.0 | 0.4.3 |
| Stable (≥1.0) | ✓ Yes | ✓ Yes | ✗ No |
| License | Apache-2.0 | Apache-2.0 | MIT |
| Dependencies | 24 | 16 | 4 |
| Crate Size | 242KB | 76KB | 24KB |
| Features | 0 | 0 | 6 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2024 | 2024 |
| MSRV | — | — | 1.86 |
| Owners | 1 | 4 | 1 |
Links
Quick Verdict
- •llm-tokenizer leads with a health score of 65/100, but none of the options score above 80.
- •miktik is pre-1.0 — API may change.