rustio rustio.net
56

pretokie

v0.1.3 Experimental

Fast, zero-allocation pretokenizers for BPE tokenizers

MIT OR Apache-2.0 Edition 2021
Text processing #tokenizer#nlp#bpe#transformers#pretokenizer

Quick Verdict

  • โœ“Actively maintained (updated 78d ago)
  • !Pre-1.0: API may have breaking changes
  • โœ“Tiny footprint (80KB, 3 deps)
  • โœ“Permissive license (MIT OR Apache-2.0)

Security

Checking security advisories...
Downloads
11.0K
Dependents
10
Releases
6
Size
80KB

Deep Insights

๐Ÿ“Š
Steady growth

708 downloads in the last 30 days (24/day), up 3% from the previous period.

๐Ÿชถ
Minimal dependencies

Only 3 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.

Health Breakdown

Maintenance 15/25

Recency, release consistency, active ratio

Quality 15/25

Yanked ratio, deps, size, maturity, features

Community 8/20

Reverse deps, ownership, ecosystem

Popularity 5/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
0/day avg
08/239/109/2810/1611/311/20

Top Dependents

Most downloaded crates that depend on pretokie

Version Adoption

v0.0.4
90%
v0.1.3
8%
v0.0.3
1%
v0.0.1
0%
v0.0.2
0%

Release Timeline

6 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
6
Less
More

Feature Flags

regex

README

Loading README...

Maintainers

Dependencies
3
direct dependencies
Dependents
10
crates depend on pretokie

Similar Crates