shimmytok

v0.7.0 Experimental

Pure Rust tokenizer for GGUF models with llama.cpp compatibility (SentencePiece + BPE + WPM + UGM + RWKV)

MIT Edition 2021

Encoding Text processing Parser implementations #tokenizer#sentencepiece#llm#llama#gguf

GitHub ↗ Docs ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 73d ago)
!Pre-1.0: API may have breaking changes
✓Permissive license (MIT)

Security

Checking security advisories...

Downloads

1.3K

Dependents

Releases

Size

178KB

Deep Insights

📈

Strong growth momentum

219 downloads in the last 30 days (7/day), up 152% from the previous period.

Health Breakdown

Maintenance 15/25

Recency, release consistency, active ratio

Quality 11/25

Yanked ratio, deps, size, maturity, features

Community 9/20

Reverse deps, ownership, ecosystem

Popularity 4/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

4/day avg+121%

Top Dependents

Most downloaded crates that depend on shimmytok

oxide-rs

129 dl 43

Version Adoption

v0.7.0

30%

v0.5.0

20%

v0.4.0

17%

v0.3.0

16%

v0.1.0

16%

Release Timeline

6 releasessince 2025

2025

2026

Less

README

Loading README...

Maintainers

Mike Kuykendall

27 crates

Dependencies

direct dependencies