rustio rustio.net
61

toktrie_hf_tokenizers

v1.9.1 Stable

HuggingFace tokenizers library support for toktrie and llguidance

MIT Edition 2021 MSRV 1.87

Quick Verdict

  • βœ“Actively maintained (updated 9d ago)
  • βœ“Stable API (1.x for 1+ years)
  • βœ“Permissive license (MIT)

Security

Checking security advisories...
Downloads
1.1M
Dependents
90
Releases
66
Size
14KB

Deep Insights

πŸ“‰
Download decline

96.4K downloads in the last 30 days, down 45% from the previous period. May indicate migration to alternatives.

βœ…
Stable API

Reached 1.0 over a year ago. The API surface is settled, making it safe for production use.

πŸ“
Compact crate

At 14KB, toktrie_hf_tokenizers is lightweight. Small crate size correlates with focused, well-scoped functionality.

🌟
Used by top crates

Notable dependents include mistralrs-core, toktrie_hf_downloader, onde-mistralrs-core, mlxrs, onnx-genai-engine. When high-quality crates choose toktrie_hf_tokenizers, it's a strong quality signal.

Health Breakdown

Maintenance 18/25

Recency, release consistency, active ratio

Quality 18/25

Yanked ratio, deps, size, maturity, features

Community 6/20

Reverse deps, ownership, ecosystem

Popularity 7/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads Β· last 90 days
722/day avg-100%
02K4K6K8K8/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v1.7.6
47%
v1.7.5
19%
v1.7.4
15%
v1.8.0
13%
v1.7.0
6%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

README

Loading README...

Maintainers

Dependencies
7
direct dependencies
Dependents
90
crates depend on toktrie_hf_tokenizers