Decision Workspace
avila-tokenizers vs axonml-llm vs token-count
Side-by-side comparison of Rust crates
40
avila-tokenizers
experimentalv0.1.0
The most complete tokenizer library in Rust - BPE, WordPiece, Unigram, with native support for GPT, BERT, Llama, Claude
53
axonml-llm
experimentalv0.4.3
Large Language Model architectures for the Axonml ML framework
49
token-count
experimentalv0.4.0
Count tokens for LLM models using exact tokenization
Core Metrics
| avila-tokenizers | axonml-llm | token-count | |
|---|---|---|---|
| Health Score | 40 | 53 | 49 |
| Total Downloads | 50 | 275 | 72 |
| 30d Downloads | 4 | 130 | 72 |
| Dependents | 2 | 12 | 0 |
| Releases | 1 | 13 | 5 |
| Last Updated | 124d ago | 3d ago | 13d ago |
| Age | 4m | 2m | 13d |
Health Breakdown
avila-tokenizers
Maintenance
7
Quality
14
Community
7
Popularity
2
Documentation
10
axonml-llm
Maintenance
13
Quality
13
Community
9
Popularity
3
Documentation
15
token-count
Maintenance
18
Quality
11
Community
6
Popularity
2
Documentation
12
Technical Details
| avila-tokenizers | axonml-llm | token-count | |
|---|---|---|---|
| Version | 0.1.0 | 0.4.3 | 0.4.0 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT OR Apache-2.0 | MIT OR Apache-2.0 | MIT |
| Dependencies | 12 | 16 | 15 |
| Crate Size | 93KB | 92KB | 339KB |
| Features | 3 | 2 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2024 | 2021 |
| MSRV | — | 1.85 | 1.85.0 |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •axonml-llm leads with a health score of 53/100, but none of the options score above 80.