rustio rustio.net
51

splintr

v0.21.0 Experimental

Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings

MIT Edition 2021
EncodingText processing #tokenizer#wordpiece#bpe#sentencepiece#llm

Quick Verdict

  • โœ“Actively maintained (updated 12d ago)
  • !Pre-1.0: API may have breaking changes
  • !Heavy dependency tree (31 direct deps)
  • โœ“Permissive license (MIT)

Security

Checking security advisories...
Downloads
35.0K
Dependents
23
Releases
36
Size
612KB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

8.7K downloads in the last 30 days (289/day), up 33% from the previous period.

๐Ÿ‘ค
Prolific maintainer

The primary maintainer publishes 114 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.

๐Ÿ“ฆ
Heavy dependency tree

31 direct dependencies. Consider the impact on compile times and supply chain complexity.

โš ๏ธ
Notable yanked versions

14% of releases have been yanked. Check version history before pinning.

๐ŸŒŸ
Used by top crates

Notable dependents include boostr, w2v-bert-uk, blazr, rust-asr, voxr. When high-quality crates choose splintr, it's a strong quality signal.

Health Breakdown

Maintenance 18/25

Recency, release consistency, active ratio

Quality 8/25

Yanked ratio, deps, size, maturity, features

Community 10/20

Reverse deps, ownership, ecosystem

Popularity 5/15

Downloads, momentum, growth trend

Documentation 10/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
17/day avg-100%
01002003004008/229/99/2710/1511/211/19

Top Dependents

Version Adoption

v0.20.0
90%
v0.21.0
4%
v0.19.1
2%
v0.18.0
2%
v0.17.0
1%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

Feature Flags

default =["rayon", "regexr-jit"]

pcre2rayon*pythonvocabsvocab-glmregexr-jit*vocab-kimivocab-phi4vocab-qwenvocab-o200kvocab-olmo2vocab-cl100kvocab-gemma2vocab-gemma3vocab-gemma4vocab-llama2vocab-llama3vocab-gpt-ossvocab-mistralvocab-whispervocab-deepseekvocab-modernbert

README

Loading README...

Maintainers

Dependencies
31
direct dependencies
Dependents
23
crates depend on splintr

Similar Crates