rustio rustio.net
52

tokenizer-lib

v1.6.0 Growing

Tokenization utilities for building parsers in Rust

MIT Edition 2021
Parsing tools #parsing#utils#tokenization

Quick Verdict

  • โœ•Not updated for 2+ years
  • โœ“Stable API (1.x for 5+ years)
  • โœ“Tiny footprint (7KB, 1 deps)
  • โœ“Permissive license (MIT)

Security

Checking security advisories...
Downloads
32.0K
Dependents
32
Releases
15
Size
7KB

Deep Insights

๐Ÿ“Š
Steady growth

909 downloads in the last 30 days (30/day), up 16% from the previous period.

๐Ÿ‘ค
Prolific maintainer

The primary maintainer publishes 55 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.

โœ…
Stable for 5+ years

The API has been stable (1.x) for over 5 years with 15 releases. This level of maturity means you can depend on it without worrying about breaking changes.

๐Ÿชถ
Minimal dependencies

Only 1 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.

๐Ÿ“
Compact crate

At 7KB, tokenizer-lib is lightweight. Small crate size correlates with focused, well-scoped functionality.

Health Breakdown

Maintenance 5/25

Recency, release consistency, active ratio

Quality 23/25

Yanked ratio, deps, size, maturity, features

Community 9/20

Reverse deps, ownership, ecosystem

Popularity 5/15

Downloads, momentum, growth trend

Documentation 10/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
0/day avg
08/229/99/2710/1511/211/19

Top Dependents

Most downloaded crates that depend on tokenizer-lib

Version Adoption

v1.6.0
44%
v1.5.1
21%
v1.5.0
18%
v1.2.0
10%
v1.3.0
7%

Release Timeline

10 releasessince 2021
J
F
M
A
M
J
J
A
S
O
N
D
2021
5
2022
2
2023
2
2024
1
2025
2026
Less
More

Feature Flags

default =["parallel", "buffered", "generator"]

buffered*parallel*generator*sized-tokens

README

Loading README...

Maintainers

Dependencies
1
direct dependencies
Dependents
32
crates depend on tokenizer-lib

Similar Crates