CLI Reference

`crawlberg`

High-performance web crawler and scraper

Options

Name	Flags	Type	Default	Description
`log_level`	`--log-level`	`Option<CliLogLevel>`		Log level: off, error, warn, info, debug, or trace (overrides -v/-q)
`log_format`	`--log-format`	`CliLogFormat`	`CliLogFormat`	Log output format
`verbose`	`-v`, `--verbose`	`u8`		Increase log verbosity (-v = info, -vv = debug, -vvv = trace)
`quiet`	`-q`, `--quiet`	`bool`		Suppress diagnostics below error level

`crawlberg batch-crawl`

Crawl multiple websites concurrently

Arguments

Name	Flags	Type	Default	Description
`urls`		`Vec<String>`		Seed URLs to crawl

Options

Name	Flags	Type	Default	Description
`depth`	`-d`, `--depth`	`usize`	`2`	Maximum crawl depth
`max_pages`	`-n`, `--max-pages`	`Option<usize>`		Maximum pages to crawl per seed
`concurrent`	`-c`, `--concurrent`	`usize`	`10`	Maximum concurrent requests
`rate_limit`	`--rate-limit`	`u64`	`200`	Rate limit delay in milliseconds
`format`	`--format`	`String`	`json`	Output format: json or markdown
`proxy`	`--proxy`	`Option<String>`		Proxy URL
`user_agent`	`--user-agent`	`Option<String>`		Custom user agent
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`respect_robots_txt`	`--respect-robots-txt`	`bool`		Respect robots.txt
`stay_on_domain`	`--stay-on-domain`	`bool`		Stay on the same domain
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg batch-scrape`

Scrape multiple URLs concurrently

Arguments

Name	Flags	Type	Default	Description
`urls`		`Vec<String>`		URLs to scrape

Options

Name	Flags	Type	Default	Description
`concurrent`	`-c`, `--concurrent`	`usize`	`10`	Maximum concurrent requests
`format`	`--format`	`String`	`json`	Output format: json or markdown
`proxy`	`--proxy`	`Option<String>`		Proxy URL
`user_agent`	`--user-agent`	`Option<String>`		Custom user agent
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`respect_robots_txt`	`--respect-robots-txt`	`bool`		Respect robots.txt
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg citations`

Convert markdown links into numbered citations

Arguments

Name	Flags	Type	Default	Description
`input`		`String`		Markdown text, or @file.md to read from a file

`crawlberg crawl`

Crawl a website following links

Arguments

Name	Flags	Type	Default	Description
`urls`		`Vec<String>`		Seed URL(s) to crawl

Options

Name	Flags	Type	Default	Description
`depth`	`-d`, `--depth`	`usize`	`2`	Maximum crawl depth
`max_pages`	`-n`, `--max-pages`	`Option<usize>`		Maximum pages to crawl
`concurrent`	`-c`, `--concurrent`	`usize`	`10`	Maximum concurrent requests
`rate_limit`	`--rate-limit`	`u64`	`200`	Rate limit delay in milliseconds
`format`	`--format`	`String`	`json`	Output format: json or markdown
`proxy`	`--proxy`	`Option<String>`		Proxy URL
`user_agent`	`--user-agent`	`Option<String>`		Custom user agent
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`respect_robots_txt`	`--respect-robots-txt`	`bool`		Respect robots.txt
`stay_on_domain`	`--stay-on-domain`	`bool`		Stay on the same domain
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg download`

Download a document from a URL and report its metadata

Arguments

Name	Flags	Type	Default	Description
`url`		`String`		URL to download

Options

Name	Flags	Type	Default	Description
`max_size`	`--max-size`	`Option<usize>`		Maximum document size in bytes
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg interact`

Execute browser actions on a single page

Arguments

Name	Flags	Type	Default	Description
`url`		`String`		URL to interact with

Options

Name	Flags	Type	Default	Description
`actions`	`--actions`	`String`		Actions as JSON array (e.g. ‘[{“type”:“click”,“selector”:“#submit”}]’)
`format`	`--format`	`String`	`json`	Output format: json or markdown
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg map`

Discover all URLs on a website via sitemaps and link extraction

Arguments

Name	Flags	Type	Default	Description
`url`		`String`		URL to map

Options

Name	Flags	Type	Default	Description
`limit`	`--limit`	`Option<usize>`		Maximum URLs to return
`search`	`--search`	`Option<String>`		Filter URLs by substring
`respect_robots_txt`	`--respect-robots-txt`	`bool`		Respect robots.txt
`format`	`--format`	`String`	`json`	Output format: json or markdown
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg mcp`

Start the MCP server (stdio transport by default).

Options

Name	Flags	Type	Default	Description
`http`	`--http`	`bool`		Serve over Streamable HTTP at `/mcp` instead of stdio (requires a build with the `mcp-http` feature)
`host`	`--host`	`String`	`127.0.0.1`	Host address to bind to in `--http` mode
`port`	`--port`	`u16`	`3001`	Port to listen on in `--http` mode

`crawlberg scrape`

Scrape a single URL and extract metadata

Arguments

Name	Flags	Type	Default	Description
`url`		`String`		URL to scrape

Options

Name	Flags	Type	Default	Description
`format`	`--format`	`String`	`json`	Output format: json or markdown
`proxy`	`--proxy`	`Option<String>`		Proxy URL
`user_agent`	`--user-agent`	`Option<String>`		Custom user agent
`timeout`	`--timeout`	`u64`	`30000`	Request timeout in milliseconds
`respect_robots_txt`	`--respect-robots-txt`	`bool`		Respect robots.txt
`browser_mode`	`--browser-mode`	`CliBrowserMode`	`CliBrowserMode`	When to use the browser: auto, always, or never
`browser_endpoint`	`--browser-endpoint`	`Option<String>`		CDP WebSocket endpoint for an external browser (must start with ws:// or wss://)
`config`	`--config`	`Option<String>`		Configuration as JSON string or @file.json

`crawlberg serve`

Start the REST API server.

Options

Name	Flags	Type	Default	Description
`host`	`--host`	`String`	`0.0.0.0`	Host address to bind to
`port`	`--port`	`u16`	`3000`	Port to listen on

`crawlberg version`

Print the crawlberg version as JSON