rustio rustio.net
65

llm-tokenizer

v1.9.0 Growing

LLM tokenizer library with caching and chat template support

Apache-2.0 Edition 2021
Parsing toolsText processing #tokenizer#huggingface#llm#tiktoken#chat-template

Quick Verdict

  • โœ“Actively maintained (updated 1d ago)
  • !Heavy dependency tree (24 direct deps)
  • โœ“Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
2.0M
Dependents
44
Releases
15
Size
242KB

Deep Insights

๐Ÿ“Š
Steady growth

391.6K downloads in the last 30 days (13.1K/day), up 5% from the previous period.

๐Ÿ“ฆ
Heavy dependency tree

24 direct dependencies. Consider the impact on compile times and supply chain complexity.

๐ŸŒŸ
Used by top crates

Notable dependents include dynamo-llm, smg, engine-zmq-adapter, engine-servicer. When high-quality crates choose llm-tokenizer, it's a strong quality signal.

Health Breakdown

Maintenance 22/25

Recency, release consistency, active ratio

Quality 14/25

Yanked ratio, deps, size, maturity, features

Community 11/20

Reverse deps, ownership, ecosystem

Popularity 8/15

Downloads, momentum, growth trend

Documentation 10/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
1K/day avg-100%
05K10K15K8/229/99/2710/1511/211/19

Top Dependents

Version Adoption

v1.3.2
81%
v1.4.1
10%
v1.6.0
8%
v1.7.0
1%
v1.8.0
1%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

README

Loading README...

Maintainers

Dependencies
24
direct dependencies
Dependents
44
crates depend on llm-tokenizer

Similar Crates