robotstxt
v0.3.0 GrowingA native Rust port of Google's robots.txt parser and matcher C++ library.
Quick Verdict
- โNot updated for 5+ years
- !Pre-1.0: API may have breaking changes
- โTrusted by 112 crates
- โTiny footprint (18KB, 0 deps)
- โPermissive license (Apache-2.0)
Security
Deep Insights
24.2K downloads in the last 30 days (808/day), up 88% from the previous period.
112 crates depend on robotstxt. Reasonable ecosystem adoption, though not yet a core dependency.
Despite being 7+ years old, robotstxt hasn't reached 1.0 yet. Expect potential API changes between versions.
robotstxt has no external dependencies โ minimal supply chain risk and fast compile times.
At 17KB, robotstxt is lightweight. Small crate size correlates with focused, well-scoped functionality.
33% of releases have been yanked. Check version history before pinning.
Notable dependents include ahref, voyager, crawly, sqlite-robotstxt, docrawl. When high-quality crates choose robotstxt, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on robotstxt