rustio rustio.net
49

tok3niz3r-core

v0.2.0 Experimental

Core types and on-disk schema for the tok3niz3r byte-level BPE tokenizer: token ids, vocabulary, merge table, and the GPT-2 byte-level mapping.

MIT Edition 2024 MSRV 1.85
Data structuresEncodingText processing #tokenizer#nlp#tokenization#bpe

Quick Verdict

  • !Pre-1.0: API may have breaking changes
  • โœ“Permissive license (MIT)

Security

Checking security advisories...
Downloads
195
Dependents
6
Releases
2
Size
17KB

Deep Insights

๐Ÿ“Š
Stable downloads

36 downloads in the last 30 days (1/day). Volume is roughly flat compared to the previous period.

๐Ÿ“
Compact crate

At 16KB, tok3niz3r-core is lightweight. Small crate size correlates with focused, well-scoped functionality.

๐ŸŒŸ
Used by top crates

Notable dependents include tok3niz3r-runtime, tok3niz3r-trainer, tok3niz3r-cli. When high-quality crates choose tok3niz3r-core, it's a strong quality signal.

Health Breakdown

Maintenance 11/25

Recency, release consistency, active ratio

Quality 15/25

Yanked ratio, deps, size, maturity, features

Community 8/20

Reverse deps, ownership, ecosystem

Popularity 3/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
0/day avg
08/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v0.2.0
88%
v0.1.0
12%

Release Timeline

2 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
2
Less
More

Feature Flags

hf-compat

README

Loading README...

Maintainers

Dependencies
7
direct dependencies
Dependents
6
crates depend on tok3niz3r-core

Similar Crates