44
tpt-tokenizer-core
v0.1.0 ExperimentalPure-Rust BPE and WordPiece tokenizer — no_std + alloc compatible, zero dependencies.
MIT OR Apache-2.0 Edition 2021 MSRV 1.80.0
Quick Verdict
- ✓Actively maintained (updated 87d ago)
- !Pre-1.0: API may have breaking changes
- ✓Tiny footprint (23KB, 1 deps)
- ✓Permissive license (MIT OR Apache-2.0)
Security
Checking security advisories...
Downloads
32
Dependents
0
Releases
1
Size
23KB
Deep Insights
📈
Strong growth momentum
2 downloads in the last 30 days (0/day), up 100% from the previous period.
👤
Prolific maintainer
The primary maintainer publishes 296 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.
🪶
Minimal dependencies
Only 1 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.
📐
Compact crate
At 22KB, tpt-tokenizer-core is lightweight. Small crate size correlates with focused, well-scoped functionality.
Health Breakdown
Maintenance 8/25
Recency, release consistency, active ratio
Quality 16/25
Yanked ratio, deps, size, maturity, features
Community 6/20
Reverse deps, ownership, ecosystem
Popularity 2/15
Downloads, momentum, growth trend
Documentation 12/15
Docs, repo, license, metadata
Download Trend
Daily downloads · last 90 days
0/day avg
Version Adoption
v0.1.0
100%
Release Timeline
1 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
1
LessMore
Feature Flags
default =["std"]
std*normalization
README
Loading README...
Maintainers
Dependencies
1
direct dependencies
Dependents
0
crates depend on tpt-tokenizer-core