tok3niz3r-runtime

v0.2.0 Experimental

Encoder and decoder for byte-level BPE tokenizers, with rkyv (zero-copy) and HuggingFace tokenizer.json loading. The runtime half of tok3niz3r.

MIT Edition 2024 MSRV 1.85

Encoding Parsing tools Text processing #tokenizer#nlp#tokenization#huggingface#bpe

GitHub ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 37d ago)
!Pre-1.0: API may have breaking changes
✓Permissive license (MIT)

Security

Checking security advisories...

Downloads

Dependents

Releases

Size

53KB

Deep Insights

📉

Download decline

29 downloads in the last 30 days, down 24% from the previous period. May indicate migration to alternatives.

Health Breakdown

Maintenance 12/25

Recency, release consistency, active ratio

Quality 13/25

Yanked ratio, deps, size, maturity, features

Community 8/20

Reverse deps, ownership, ecosystem

Popularity 2/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

0/day avg

Top Dependents

Most downloaded crates that depend on tok3niz3r-runtime

Version Adoption

v0.2.0

72%

v0.1.0

28%

Release Timeline

2 releasessince 2026

2026

Less

Feature Flags

hf-compat

README

Loading README...

Maintainers

Andrés Correa Casablanca

26 crates

Dependencies

direct dependencies