62
language-tokenizer
v0.4.1 ExperimentalText tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
WTFPL Edition 2024 MSRV 1.88.0
Quick Verdict
- โActively maintained (updated 1d ago)
- !Pre-1.0: API may have breaking changes
Security
Checking security advisories...
Downloads
88.2K
Dependents
5
Releases
6
Size
24KB
Deep Insights
๐
Download decline
17.3K downloads in the last 30 days, down 22% from the previous period. May indicate migration to alternatives.
๐
Compact crate
At 23KB, language-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.
Health Breakdown
Maintenance 20/25
Recency, release consistency, active ratio
Quality 16/25
Yanked ratio, deps, size, maturity, features
Community 7/20
Reverse deps, ownership, ecosystem
Popularity 6/15
Downloads, momentum, growth trend
Documentation 13/15
Docs, repo, license, metadata
Download Trend
Daily downloads ยท last 90 days
45/day avg-100%
Top Dependents
Most downloaded crates that depend on language-tokenizer
Version Adoption
v0.1.0
100%
v0.3.0
0%
v0.4.0
0%
v0.2.0
0%
v0.3.1
0%
Release Timeline
6 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
6
LessMore
Feature Flags
cjkicufullserdelinderasnowballchinese-icujapanese-icukorean-linderachinese-linderasoutheast-asianjapanese-linderakorean-lindera-embedchinese-lindera-embedjapanese-lindera-embed-ipadicjapanese-lindera-embed-unidicjapanese-lindera-embed-ipadic-neologd
README
Loading README...
Maintainers
Dependencies
13
direct dependencies
Dependents
5
crates depend on language-tokenizer