Decision Workspace
gemini-tokenizer vs shimmytok vs siumai-protocol-gemini
Side-by-side comparison of Rust crates
34
gemini-tokenizer
experimentalv0.2.0
Authoritative Gemini tokenizer for Rust, ported from the official Google Python GenAI SDK
60
shimmytok
experimentalv0.8.4
Pure Rust tokenizer for GGUF models with llama.cpp compatibility (SentencePiece + BPE + WPM + UGM + RWKV)
60
siumai-protocol-gemini
experimentalv0.11.0-beta.10
Google Gemini wire codecs for siumai
Core Metrics
| gemini-tokenizer | shimmytok | siumai-protocol-gemini | |
|---|---|---|---|
| Health Score | 34 | 60 | 60 |
| Total Downloads | 4.9K | 9.1K | 2.4K |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 2 | 25 | 18 |
| Releases | 1 | 14 | 6 |
| Last Updated | 244d ago | 19d ago | 57d ago |
| Age | 8m | 11m | 8m |
Health Breakdown
gemini-tokenizer
Maintenance
4
Quality
10
Community
7
Popularity
4
Documentation
9
shimmytok
Maintenance
18
Quality
12
Community
10
Popularity
5
Documentation
15
siumai-protocol-gemini
Maintenance
19
Quality
14
Community
8
Popularity
4
Documentation
15
Technical Details
| gemini-tokenizer | shimmytok | siumai-protocol-gemini | |
|---|---|---|---|
| Version | 0.2.0 | 0.8.4 | 0.11.0-beta.10 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | Apache-2.0 | MIT OR Apache-2.0 | MIT OR Apache-2.0 |
| Dependencies | 5 | 10 | 4 |
| Crate Size | 2.2MB | 1.2MB | 58KB |
| Features | 0 | 2 | 1 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2021 | 2021 | 2024 |
| MSRV | 1.70 | 1.70 | 1.95 |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •shimmytok leads with a health score of 60/100, but none of the options score above 80.