CLI

Installation

cargo install

cargo install spidermedic

Build from source

target/release/spidermedic

git clone https://github.com/beekmanlabs/spidermedic
cd spidermedic
cargo build --release

Usage

spidermedic --url=<URL> [OPTIONS]

Flags

Flag	Default	Description
--url	required	Starting URL to crawl
--path	/	Restrict crawl to this URL path prefix
--port	80	Port override
--interval	300	Milliseconds between requests
--concurrency	10	Max parallel in-flight requests
--max-depth	0	Max depth (0 = unlimited)
--output	terminal	terminal / json / csv

Examples

# Crawl a site with default settings
spidermedic --url=https://example.com

# Faster crawl with more concurrency, limited to 3 levels deep
spidermedic --url=https://example.com --concurrency=20 --max-depth=3

# Only crawl the /docs section
spidermedic --url=https://example.com/docs --path=/docs

# JSON output piped to jq
spidermedic --url=https://example.com --output=json | jq '.[] | select(.status >= 400)'

# CSV output saved to a file
spidermedic --url=https://example.com --output=csv > results.csv

Exit codes

Code	Meaning
0	All crawled pages returned non-error responses
1	One or more 4xx / 5xx responses were found

Use the exit code in shell scripts to gate deployments:

spidermedic --url=https://staging.example.com && deploy_to_production.sh

Caution

Be considerate with --interval and --concurrency settings on sites you don’t own. High concurrency with a low interval can look like a DoS attack to the target server.