rustio rustio.net
41

robotstxt

v0.3.0 Growing

A native Rust port of Google's robots.txt parser and matcher C++ library.

Apache-2.0 Edition 2018
Parsing toolsParser implementations #parser#robots#robotstxt

Quick Verdict

  • โœ•Not updated for 5+ years
  • !Pre-1.0: API may have breaking changes
  • โœ“Trusted by 112 crates
  • โœ“Tiny footprint (18KB, 0 deps)
  • โœ“Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
612.9K
Dependents
112
Releases
3
Size
18KB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

24.2K downloads in the last 30 days (808/day), up 88% from the previous period.

๐Ÿ”—
Moderate adoption

112 crates depend on robotstxt. Reasonable ecosystem adoption, though not yet a core dependency.

๐Ÿ”ฌ
Pre-1.0 for over a year

Despite being 7+ years old, robotstxt hasn't reached 1.0 yet. Expect potential API changes between versions.

๐Ÿชถ
Zero dependencies

robotstxt has no external dependencies โ€” minimal supply chain risk and fast compile times.

๐Ÿ“
Compact crate

At 17KB, robotstxt is lightweight. Small crate size correlates with focused, well-scoped functionality.

โš ๏ธ
Notable yanked versions

33% of releases have been yanked. Check version history before pinning.

๐ŸŒŸ
Used by top crates

Notable dependents include ahref, voyager, crawly, sqlite-robotstxt, docrawl. When high-quality crates choose robotstxt, it's a strong quality signal.

Health Breakdown

Maintenance 3/25

Recency, release consistency, active ratio

Quality 10/25

Yanked ratio, deps, size, maturity, features

Community 12/20

Reverse deps, ownership, ecosystem

Popularity 7/15

Downloads, momentum, growth trend

Documentation 9/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
15/day avg-100%
02004006008/229/99/2710/1511/211/19

Top Dependents

Version Adoption

v0.3.0
99%
v0.2.0
1%

Release Timeline

3 releases1 yankedsince 2019
J
F
M
A
M
J
J
A
S
O
N
D
2019
1
2020
1
2021
1
2022
2023
2024
2025
2026
Less
More|
Yanked

README

Loading README...

Maintainers

Dependencies
0
direct dependencies
Dependents
112
crates depend on robotstxt

Similar Crates