45
basetenkenizer
v0.2.8 ExperimentalHigh-performance tokenization for inference
MIT Edition 2024
Quick Verdict
- โActively maintained (updated 73d ago)
- !Pre-1.0: API may have breaking changes
- โPermissive license (MIT)
Security
Checking security advisories...
Downloads
622.7K
Dependents
11
Releases
1
Size
156KB
Deep Insights
๐
Strong growth momentum
340.2K downloads in the last 30 days (11.3K/day), up 61% from the previous period.
Health Breakdown
Maintenance 9/25
Recency, release consistency, active ratio
Quality 12/25
Yanked ratio, deps, size, maturity, features
Community 4/20
Reverse deps, ownership, ecosystem
Popularity 7/15
Downloads, momentum, growth trend
Documentation 13/15
Docs, repo, license, metadata
Download Trend
Daily downloads ยท last 90 days
209/day avg-100%
Top Dependents
Most downloaded crates that depend on basetenkenizer
Version Adoption
v0.2.8
100%
Release Timeline
1 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
1
LessMore
Feature Flags
default =["hf-hub"]
hf-hub*
README
Loading README...
Maintainers
Dependencies
20
direct dependencies
Dependents
11
crates depend on basetenkenizer