rustio rustio.net
47

tok3niz3r-runtime

v0.2.0 Experimental

Encoder and decoder for byte-level BPE tokenizers, with rkyv (zero-copy) and HuggingFace tokenizer.json loading. The runtime half of tok3niz3r.

MIT Edition 2024 MSRV 1.85
EncodingParsing toolsText processing #tokenizer#nlp#tokenization#huggingface#bpe

Quick Verdict

  • !Pre-1.0: API may have breaking changes
  • ✓Permissive license (MIT)

Security

Checking security advisories...
Downloads
150
Dependents
4
Releases
2
Size
53KB

Deep Insights

📊
Stable downloads

25 downloads in the last 30 days (1/day). Volume is roughly flat compared to the previous period.

Health Breakdown

Maintenance 11/25

Recency, release consistency, active ratio

Quality 13/25

Yanked ratio, deps, size, maturity, features

Community 8/20

Reverse deps, ownership, ecosystem

Popularity 3/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days
0/day avg
08/239/109/2810/1611/311/20

Top Dependents

Most downloaded crates that depend on tok3niz3r-runtime

Version Adoption

v0.2.0
86%
v0.1.0
14%

Release Timeline

2 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
2
Less
More

Feature Flags

hf-compat

README

Loading README...

Maintainers

Dependencies
16
direct dependencies
Dependents
4
crates depend on tok3niz3r-runtime

Similar Crates