67
lance-tokenizer
v13.0.0 ExperimentalTokenizer abstractions and implementations for Lance
Apache-2.0 Edition 2024 MSRV 1.91.0
Development toolsData structuresDatabase implementationsScience #machine-learning#data-science#data-format#apache-arrow#data-analytics
Quick Verdict
- โActively maintained (updated 3d ago)
- โPermissive license (Apache-2.0)
Security
Checking security advisories...
Downloads
1.8M
Dependents
37
Releases
13
Size
32KB
Deep Insights
๐
Strong growth momentum
888.8K downloads in the last 30 days (29.6K/day), up 186% from the previous period.
๐
Compact crate
At 31KB, lance-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.
๐
Used by top crates
Notable dependents include lance, lance-index, pond-db. When high-quality crates choose lance-tokenizer, it's a strong quality signal.
Health Breakdown
Maintenance 20/25
Recency, release consistency, active ratio
Quality 18/25
Yanked ratio, deps, size, maturity, features
Community 10/20
Reverse deps, ownership, ecosystem
Popularity 7/15
Downloads, momentum, growth trend
Documentation 12/15
Docs, repo, license, metadata
Download Trend
Daily downloads ยท last 90 days
2K/day avg-100%
Top Dependents
Most downloaded crates that depend on lance-tokenizer
Version Adoption
v9.0.1
39%
v8.0.0
24%
v9.0.0
17%
v10.0.0
14%
v11.0.0
6%
Release Timeline
10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
LessMore
Feature Flags
linderajieba-rstokenizer-jiebatokenizer-lindera
README
Loading README...
Maintainers
Dependencies
7
direct dependencies
Dependents
37
crates depend on lance-tokenizer