rustio rustio.net
54

kitoken

v0.11.0 Growing

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

BSD-2-Clause Edition 2024 MSRV 1.86.0

Quick Verdict

  • !Pre-1.0: API may have breaking changes
  • !Heavy dependency tree (24 direct deps)

Security

Checking security advisories...
Downloads
46.9K
Dependents
23
Releases
3
Size
64KB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

1.6K downloads in the last 30 days (54/day), up 95% from the previous period.

๐Ÿ”ฌ
Pre-1.0 for over a year

Despite being 1+ years old, kitoken hasn't reached 1.0 yet. Expect potential API changes between versions.

๐Ÿ“ฆ
Heavy dependency tree

24 direct dependencies. Consider the impact on compile times and supply chain complexity.

๐ŸŒŸ
Used by top crates

Notable dependents include miktik, rust-genai, rlx-voxtral-tts. When high-quality crates choose kitoken, it's a strong quality signal.

Health Breakdown

Maintenance 10/25

Recency, release consistency, active ratio

Quality 16/25

Yanked ratio, deps, size, maturity, features

Community 10/20

Reverse deps, ownership, ecosystem

Popularity 6/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
1/day avg
0501008/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v0.10.1
89%
v0.11.0
10%
v0.10.0
2%

Release Timeline

3 releasessince 2024
J
F
M
A
M
J
J
A
S
O
N
D
2024
2
2025
2026
1
Less
More

Feature Flags

default =["std", "serialization", "normalization", "convert", "regex-perf", "multiversion"]

allstd*websplitconvert*unstableregex-onigregex-perf*multiversion*normalization*regex-unicodeserialization*convert-detectconvert-tekkenconvert-tiktokenconvert-tokenizerssplit-unicode-scriptconvert-sentencepiecenormalization-unicodenormalization-charsmap

README

Loading README...

Maintainers

Dependencies
24
direct dependencies
Dependents
23
crates depend on kitoken

Similar Crates