rustio rustio.net
67

lance-tokenizer

v13.0.0 Experimental

Tokenizer abstractions and implementations for Lance

Apache-2.0 Edition 2024 MSRV 1.91.0
Development toolsData structuresDatabase implementationsScience #machine-learning#data-science#data-format#apache-arrow#data-analytics

Quick Verdict

  • โœ“Actively maintained (updated 3d ago)
  • โœ“Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
1.8M
Dependents
37
Releases
13
Size
32KB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

888.8K downloads in the last 30 days (29.6K/day), up 186% from the previous period.

๐Ÿ“
Compact crate

At 31KB, lance-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.

๐ŸŒŸ
Used by top crates

Notable dependents include lance, lance-index, pond-db. When high-quality crates choose lance-tokenizer, it's a strong quality signal.

Health Breakdown

Maintenance 20/25

Recency, release consistency, active ratio

Quality 18/25

Yanked ratio, deps, size, maturity, features

Community 10/20

Reverse deps, ownership, ecosystem

Popularity 7/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
2K/day avg-100%
010K20K30K40K50K8/239/109/2810/1611/311/20

Top Dependents

Most downloaded crates that depend on lance-tokenizer

Version Adoption

v9.0.1
39%
v8.0.0
24%
v9.0.0
17%
v10.0.0
14%
v11.0.0
6%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

Feature Flags

linderajieba-rstokenizer-jiebatokenizer-lindera

README

Loading README...

Maintainers

Dependencies
7
direct dependencies
Dependents
37
crates depend on lance-tokenizer

Similar Crates