shimmytok

v0.7.1 Experimental

Pure Rust tokenizer for GGUF models with llama.cpp compatibility (SentencePiece + BPE + WPM + UGM + RWKV)

MIT OR Apache-2.0 Edition 2021

Encoding Text processing Parser implementations #tokenizer#sentencepiece#llm#llama#gguf

GitHub ↗ Docs ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 0d ago)
!Pre-1.0: API may have breaking changes
✓Permissive license (MIT OR Apache-2.0)

Security

Checking security advisories...

Downloads

3.0K

Dependents

Releases

Size

1.1MB

Deep Insights

📈

Strong growth momentum

1.3K downloads in the last 30 days (44/day), up 224% from the previous period.

Health Breakdown

Maintenance 19/25

Recency, release consistency, active ratio

Quality 10/25

Yanked ratio, deps, size, maturity, features

Community 9/20

Reverse deps, ownership, ecosystem

Popularity 4/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

22/day avg+249%

Top Dependents

Most downloaded crates that depend on shimmytok

oxide-rs

139 dl 46

Version Adoption

v0.7.0

71%

v0.5.0

v0.4.0

v0.3.0

v0.1.0

Release Timeline

7 releasessince 2025

2025

2026

Less

README

Loading README...

Maintainers

Mike Kuykendall

27 crates

Dependencies

direct dependencies