rustio rustio.net
62

language-tokenizer

v0.4.1 Experimental

Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.

WTFPL Edition 2024 MSRV 1.88.0
Text processing #text#language#tokenizer#segmenter#linguistic

Quick Verdict

  • โœ“Actively maintained (updated 1d ago)
  • !Pre-1.0: API may have breaking changes

Security

Checking security advisories...
Downloads
88.2K
Dependents
5
Releases
6
Size
24KB

Deep Insights

๐Ÿ“‰
Download decline

17.3K downloads in the last 30 days, down 22% from the previous period. May indicate migration to alternatives.

๐Ÿ“
Compact crate

At 23KB, language-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.

Health Breakdown

Maintenance 20/25

Recency, release consistency, active ratio

Quality 16/25

Yanked ratio, deps, size, maturity, features

Community 7/20

Reverse deps, ownership, ecosystem

Popularity 6/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
45/day avg-100%
02004006008008/229/99/2710/1511/211/19

Top Dependents

Most downloaded crates that depend on language-tokenizer

Version Adoption

v0.1.0
100%
v0.3.0
0%
v0.4.0
0%
v0.2.0
0%
v0.3.1
0%

Release Timeline

6 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
6
Less
More

Feature Flags

cjkicufullserdelinderasnowballchinese-icujapanese-icukorean-linderachinese-linderasoutheast-asianjapanese-linderakorean-lindera-embedchinese-lindera-embedjapanese-lindera-embed-ipadicjapanese-lindera-embed-unidicjapanese-lindera-embed-ipadic-neologd

README

Loading README...

Maintainers

Dependencies
13
direct dependencies
Dependents
5
crates depend on language-tokenizer

Similar Crates