Decision Workspace
dynamo-tokenizers vs llm-tokenizer vs miktik
Side-by-side comparison of Rust crates
63
dynamo-tokenizers
experimentalv3.1.0
Standalone HuggingFace, tiktoken, fastokens, and Baseten tokenizer implementations for LLM inference serving.
65
llm-tokenizer
growingv1.9.0
LLM tokenizer library with caching and chat template support
57
miktik
experimentalv0.4.3
A unified, multi-backend tokenizer library for LLMs
Core Metrics
| dynamo-tokenizers | llm-tokenizer | miktik | |
|---|---|---|---|
| Health Score | 63 | 65 | 57 |
| Total Downloads | 818.0K | 2.0M | 5.6K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 51 | 44 | 0 |
| Releases | 24 | 15 | 11 |
| Last Updated | 3d ago | 1d ago | 36d ago |
| Age | 5m | 9m | 7m |
Health Breakdown
dynamo-tokenizers
Maintenance
20
Quality
14
Community
12
Popularity
7
Documentation
10
llm-tokenizer
Maintenance
22
Quality
14
Community
11
Popularity
8
Documentation
10
miktik
Maintenance
17
Quality
16
Community
5
Popularity
4
Documentation
15
Technical Details
| dynamo-tokenizers | llm-tokenizer | miktik | |
|---|---|---|---|
| Version | 3.1.0 | 1.9.0 | 0.4.3 |
| Stable (≥1.0) | ✓ Yes | ✓ Yes | ✗ No |
| License | Apache-2.0 | Apache-2.0 | MIT |
| Dependencies | 16 | 24 | 4 |
| Crate Size | 76KB | 242KB | 24KB |
| Features | 0 | 0 | 6 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2021 | 2024 |
| MSRV | — | — | 1.86 |
| Owners | 4 | 1 | 1 |
Links
Quick Verdict
- •llm-tokenizer leads with a health score of 65/100, but none of the options score above 80.
- •miktik is pre-1.0 — API may change.