spider

v2.51.197 Stable

A web crawler and scraper, building blocks for data curation workloads.

MIT Edition 2021

Command line utilities Web programming #spider#crawler#scraper

GitHub ↗ Docs ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 4d ago)
✓Stable API (2.x for 8+ years)
✓Massive adoption (5.7K crates depend on it)
!Heavy dependency tree (90 direct deps)
✓Permissive license (MIT)

Security

Checking security advisories...

Downloads

2.0M

Dependents

5.7K

Releases

2192

Size

717KB

Deep Insights

📊

Steady growth

20.0K downloads in the last 30 days (667/day), up 6% from the previous period.

🔗

Widely adopted

5.7K crates depend on spider. Strong ecosystem adoption means battle-tested code and long-term stability.

✅

Stable for 8+ years

The API has been stable (1.x) for over 8 years with 2192 releases. This level of maturity means you can depend on it without worrying about breaking changes.

📦

Heavy dependency tree

90 direct dependencies. Consider the impact on compile times and supply chain complexity.

🌟

Used by top crates

Notable dependents include spider_cli, spider_worker, spider_transformations, website_crawler, accessibility-rs. When high-quality crates choose spider, it's a strong quality signal.

Health Breakdown

Maintenance 19/25

Recency, release consistency, active ratio

Quality 17/25

Yanked ratio, deps, size, maturity, features

Community 17/20

Reverse deps, ownership, ecosystem

Popularity 8/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

607/day avg+21%

Top Dependents

Most downloaded crates that depend on spider

spider_transformations

swiftide-integrations

Version Adoption

v2.51.190

67%

v2.51.197

16%

v2.51.192

10%

v2.51.191

v2.51.194

Release Timeline

10 releasessince 2026

2026

Less

Feature Flags

default =["basic", "io_uring", "tcp_fastopen", "splice", "numa", "zero_copy"]

fscmdcrondiskglobnuma*simdsynctimewarcwreqagentbasic*bloomcachehedgeregexserdeservosmartsocksspoofchromecowboygeminiopenaisearchsplice*__basicadblockbalancecontrolcookiesheaderssitemaptracingagent_fsdisk_awsencodingfirewallio_uring*llm_jsonbasic_tlscache_memdns_cachewebdriverzero_copy*agent_fullchrome_cpuetag_cacherate_limitwait_guardchrome_simdflexbuffersinline-moreremote_addrsearch_bingagent_chromeagent_openaiagent_skillscache_geminicache_openaih2_multiplexreal_browserreqwest_jsonrobots_cachesearch_bravespider_cloudtcp_fastopen*tokio_io_stdua_generatorauto_throttlecache_requestchrome_headeddecentralizedsearch_serpersearch_tavilybench_jemallocbench_mimallocchrome_stealthfull_resourceswebdriver_edgeagent_skills_s3agent_webdriverdisk_native_tlsopenai_slim_fitadblock_easylistchrome_interceptrequest_coalescewebdriver_chromewebdriver_headedagent_search_bingchrome_screenshotchrome_store_pageextra_informationparallel_backendspriority_frontierreqwest_multipartwebdriver_firefoxwebdriver_stealthagent_search_bravereqwest_native_tlsreqwest_rustls_tlsrquest_hickory_dnsagent_search_serperagent_search_tavilycache_chrome_hybridchrome_headless_newchrome_remote_cachereqwest_hickory_dnsadaptive_concurrencychrome_serde_stackerwebdriver_screenshotchrome_tls_connectiondecentralized_headersparallel_backends_fullcache_chrome_hybrid_memchrome_remote_cache_memreqwest_native_tls_alpnchrome_remote_cache_diskpage_error_status_detailsreqwest_native_tls_vendoredstring_interner_bucket_backendstring_interner_buffer_backendstring_interner_string_backendreqwest_native_tls_manual_rootsreqwest_native_tls_native_rootsreqwest_native_tls_webpki_roots