Ruby API Reference

Ruby API Reference v1.1.4

Functions

generate_citations()

Convert markdown links to numbered citations.

[Example](https://example.com) becomes Example[1] with [1]: <https://example.com> in the reference list. Images ![alt](url) are preserved unchanged.

Signature:

def self.generate_citations(markdown)

Example:

result = generate_citations("value")

Parameters:

Name	Type	Required	Description
markdown	String	Yes	The markdown

Returns: CitationResult

---

create_engine()

Create a new crawl engine with the given configuration.

If config is nil, uses CrawlConfig.default(). Returns an error if the configuration is invalid.

Signature:

def self.create_engine(config: nil)

Example:

result = create_engine(config: CrawlConfig.new)

Parameters:

Name	Type	Required	Description
config	CrawlConfig?	No	The configuration options

Returns: CrawlEngineHandle

Errors: Raises CrawlError.

---

scrape()

Scrape a single URL, returning extracted page data.

Signature:

def self.scrape(engine, url)

Example:

result = scrape(CrawlEngineHandle.new, "value")

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
url	String	Yes	The URL to fetch

Returns: ScrapeResult

Errors: Raises CrawlError.

---

crawl()

Crawl a website starting from url, following links up to the configured depth.

Signature:

def self.crawl(engine, url)

Example:

result = crawl(CrawlEngineHandle.new, "value")

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
url	String	Yes	The URL to fetch

Returns: CrawlResult

Errors: Raises CrawlError.

---

map_urls()

Discover all pages on a website by following links and sitemaps.

Signature:

def self.map_urls(engine, url)

Example:

result = map_urls(CrawlEngineHandle.new, "value")

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
url	String	Yes	The URL to fetch

Returns: MapResult

Errors: Raises CrawlError.

---

interact()

Execute browser actions on a single page.

Signature:

def self.interact(engine, url, actions)

Example:

result = interact(CrawlEngineHandle.new, "value", [])

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
url	String	Yes	The URL to fetch
actions	Array<PageAction>	Yes	The actions

Returns: InteractionResult

Errors: Raises CrawlError.

---

batch_scrape()

Scrape multiple URLs concurrently.

Signature:

def self.batch_scrape(engine, urls)

Example:

result = batch_scrape(CrawlEngineHandle.new, [])

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
urls	Array<String>	Yes	The urls

Returns: BatchScrapeResults

Errors: Raises CrawlError.

---

batch_crawl()

Crawl multiple seed URLs concurrently, each following links to configured depth.

Signature:

def self.batch_crawl(engine, urls)

Example:

result = batch_crawl(CrawlEngineHandle.new, [])

Parameters:

Name	Type	Required	Description
engine	CrawlEngineHandle	Yes	The crawl engine handle
urls	Array<String>	Yes	The urls

Returns: BatchCrawlResults

Errors: Raises CrawlError.

---

Types

ActionResult

Result from a single page action execution.

Field	Type	Default	Description
action_index	Integer	—	Zero-based index of the action in the sequence.
action_type	String	—	The type of action that was executed.
success	Boolean	—	Whether the action completed successfully.
data	Object?	nil	Action-specific return data (screenshot bytes, JS return value, scraped HTML).
error	String?	nil	Error message if the action failed.

---

ArticleMetadata

Article metadata extracted from article:* Open Graph tags.

Field	Type	Default	Description
published_time	String?	nil	The article publication time.
modified_time	String?	nil	The article modification time.
author	String?	nil	The article author.
section	String?	nil	The article section.
tags	Array<String>	\[\]	The article tags.

---

BatchCrawlResult

Result from a single URL in a batch crawl operation.

Field	Type	Default	Description
url	String	—	The seed URL that was crawled.
result	CrawlResult?	nil	The crawl result, if successful.
error	String?	nil	The error message, if the crawl failed.

---

BatchCrawlResults

Aggregate result of a batch crawl, exposing per-URL results plus precomputed counts.

The counts are derived once at construction so every binding language can read them as plain integer fields without re-iterating the results vector.

Field	Type	Default	Description
results	Array<BatchCrawlResult>	\[\]	Per-URL crawl results, in the order seed URLs were submitted.
total_count	Integer	—	Total number of seed URLs in the batch (equal to results.len()).
completed_count	Integer	—	Number of seed URLs whose crawl succeeded (error is nil).
failed_count	Integer	—	Number of seed URLs whose crawl failed (error is Some).

---

BatchCrawlStreamRequest

Request to begin a multi-URL streaming crawl.

Wraps a set of seed URLs for delivery through the streaming-adapter binding surface. Required as a struct because alef’s streaming adapter requires a named request type — primitives are not supported.

Field	Type	Default	Description
urls	Array<String>	\[\]	The seed URLs to crawl. Each URL is followed independently up to the engine’s configured depth.

---

BatchScrapeResult

Result from a single URL in a batch scrape operation.

Field	Type	Default	Description
url	String	—	The URL that was scraped.
result	ScrapeResult?	nil	The scrape result, if successful.
error	String?	nil	The error message, if the scrape failed.

---

BatchScrapeResults

Aggregate result of a batch scrape, exposing per-URL results plus precomputed counts.

The counts are derived once at construction so every binding language can read them as plain integer fields without re-iterating the results vector.

Field	Type	Default	Description
results	Array<BatchScrapeResult>	\[\]	Per-URL scrape results, in the order URLs were submitted.
total_count	Integer	—	Total number of URLs in the batch (equal to results.len()).
completed_count	Integer	—	Number of URLs whose scrape succeeded (error is nil).
failed_count	Integer	—	Number of URLs whose scrape failed (error is Some).

---

BrowserConfig

Browser fallback configuration.

Field	Type	Default	Description
mode	BrowserMode	:auto	When to use the headless browser fallback.
backend	BrowserBackend	:chromiumoxide	Browser backend used to render JavaScript-heavy pages.
endpoint	String?	nil	CDP WebSocket endpoint for connecting to an external browser instance.
timeout	Float	30000ms	Timeout for browser page load and rendering (in milliseconds when serialized).
wait	BrowserWait	:network_idle	Wait strategy after browser navigation.
wait_selector	String?	nil	CSS selector to wait for when wait is Selector.
extra_wait	Float?	nil	Extra time to wait after the wait condition is met.
proxy	ProxyConfig?	nil	Proxy for browser fetches. Overrides CrawlConfig.proxy when set. Native backend supports http/https only (no SOCKS5).
block_url_patterns	Array<String>	\[\]	URL patterns to block before the network request fires. Supports * wildcards. Useful for skipping ads/analytics/large images. Honored by BrowserBackend.Native; chromiumoxide ignores this field today.
eval_script	String?	nil	JavaScript snippet evaluated after navigation completes. Scraping captures the native backend result in ScrapeResult.browser.eval_result. Interactions run this script before page actions on both browser backends but do not include the script result in InteractionResult.
robots_user_agent	String?	nil	User-agent used when fetching robots.txt. Defaults to BrowserConfig.user_agent (or crawlberg’s default) if unset. Native only.
capture_network_events	Boolean	false	Capture the full network event stream into the result. Default false (only the document event is captured). Native only.
session_affinity	Boolean	true	Enable session affinity: reuse chromiumoxide Pages for same-domain requests so cookies + fingerprint + solved challenges persist. Default: true. When false, each request gets a fresh Page.

Methods

default()

Signature:

def self.default()

Example:

result = BrowserConfig.default()

Returns: BrowserConfig

---

BrowserExtras

Browser-specific extras populated when the native browser backend was used.

Available on ScrapeResult.browser when BrowserBackend.Native handled the request.

Field	Type	Default	Description
eval_result	Object?	nil	Return value of BrowserConfig.eval_script, if provided.
network_events	Array<ResponseMeta>	\[\]	Network events captured during page navigation (only populated when BrowserConfig.capture_network_events is true).
cookies	Array<CookieInfo>	\[\]	All non-expired cookies present in the browser’s cookie jar after navigation completes (includes both prior cookies and server Set-Cookie).

---

CitationReference

A single numbered reference in a citation list — produced by the citation extractor when content uses inline [N]-style markers.

Field	Type	Default	Description
index	Integer	—	1-based reference number as it appears in the source text.
url	String	—	Resolved absolute URL for this reference.
text	String	—	Human-readable anchor text or title for the reference.

---

CitationResult

Result of citation conversion.

Field	Type	Default	Description
content	String	—	Markdown with links replaced by numbered citations.
references	Array<CitationReference>	\[\]	Numbered reference list: (index, url, text).

---

ContentConfig

Content extraction and conversion configuration.

Controls how HTML is converted to the output format. Uses html-to-markdown-rs as the conversion engine for all formats (markdown, plain text, djot).

Field	Type	Default	Description
output_format	String	"markdown"	Output format: "markdown" (default), "plain", "djot".
preprocessing_preset	String	"standard"	Preprocessing aggressiveness: "minimal", "standard" (default), "aggressive". - Minimal: only scripts/styles removed. - Standard: also removes nav, nav-hinted headers/footers/asides, forms. - Aggressive: removes all footers/asides unconditionally.
remove_navigation	Boolean	true	Remove navigation elements (nav, breadcrumbs, menus). Default: true.
remove_forms	Boolean	true	Remove form elements. Default: true.
strip_tags	Array<String>	\[\]	HTML tag names to strip (render children only, remove the tag wrapper). Default: \["noscript"\].
preserve_tags	Array<String>	\[\]	HTML tag names to preserve as raw HTML in output.
exclude_selectors	Array<String>	\[\]	CSS selectors for elements to exclude entirely (element + all content). Unlike strip_tags (which removes the wrapper but keeps children), excluded elements and all descendants are dropped. Supports CSS selectors: .class, #id, \[attribute\], compound selectors. Example: \[".cookie-banner", "#ad-container", "\[role='complementary'\]"\]
skip_images	Boolean	false	Skip image elements in output. Default: false.
max_depth	Integer?	nil	Max DOM traversal depth. Prevents stack overflow on deeply nested HTML.
wrap	Boolean	false	Enable line wrapping. Default: false.
wrap_width	Integer	80	Wrap width when wrap is enabled. Default: 80.
include_document_structure	Boolean	true	Include document structure tree in output. Default: true.

Methods

default()

Signature:

def self.default()

Example:

result = ContentConfig.default()

Returns: ContentConfig

---

CookieInfo

Information about an HTTP cookie received from a response.

Field	Type	Default	Description
name	String	—	The cookie name.
value	String	—	The cookie value.
domain	String?	nil	The cookie domain, if specified.
path	String?	nil	The cookie path, if specified.

---

CrawlConfig

Configuration for crawl, scrape, and map operations.

Field	Type	Default	Description
max_depth	Integer?	nil	Maximum crawl depth (number of link hops from the start URL).
max_pages	Integer?	nil	Maximum number of pages to crawl.
max_links_per_page	Integer?	nil	Maximum links enqueued from a single page. Defaults to 10000. Bounds the work one hostile or pathological page can create; links past the cap are dropped and a warning is logged.
max_concurrent	Integer?	nil	Maximum number of concurrent requests.
respect_robots_txt	Boolean	false	Whether to respect robots.txt directives.
soft_http_errors	Boolean	false	When true, HTTP-level error responses (404 NotFound, 403 Forbidden, WAF blocks) are surfaced as ScrapeResult records with the matching status_code rather than raised as CrawlError. Default false preserves the historical throw-on-error contract for direct fetches. Independently of this flag, 404s reached at the end of a redirect chain are always surfaced softly — the user opted into redirect-following, so receiving a 404 there is part of the normal flow rather than an unexpected error.
user_agent	String?	nil	Custom user-agent string.
stay_on_domain	Boolean	false	Whether to restrict crawling to the same domain.
allow_subdomains	Boolean	false	Whether to allow subdomains when stay_on_domain is true.
include_paths	Array<String>	\[\]	Regex patterns for paths to include during crawling.
exclude_paths	Array<String>	\[\]	Regex patterns for paths to exclude during crawling.
custom_headers	Hash{String=>String}	{}	Custom HTTP headers to send with each request.
request_timeout	Float	30000ms	Timeout for individual HTTP requests (in milliseconds when serialized).
rate_limit_ms	Integer?	nil	Per-domain rate limit in milliseconds. When set, enforces a minimum delay between requests to the same domain. Defaults to 200ms when nil.
max_redirects	Integer	10	Maximum number of redirects to follow.
retry_count	Integer	0	Number of retry attempts for failed requests.
retry_codes	Array<Integer>	\[\]	HTTP status codes that should trigger a retry.
cookies_enabled	Boolean	false	Whether to enable cookie handling.
auth	AuthConfig?	nil	Authentication configuration.
max_body_size	Integer?	nil	Maximum response body size in bytes.
remove_tags	Array<String>	\[\]	CSS selectors for tags to remove from HTML before processing.
content	ContentConfig	—	Content extraction and conversion configuration.
map_limit	Integer?	nil	Maximum number of URLs to return from a map operation.
map_search	String?	nil	Search filter for map results (case-insensitive substring match on URLs).
download_assets	Boolean	false	Whether to download assets (CSS, JS, images, etc.) from the page.
asset_types	Array<AssetCategory>	\[\]	Filter for asset categories to download.
max_asset_size	Integer?	nil	Maximum size in bytes for individual asset downloads.
browser	BrowserConfig	—	Browser configuration.
proxy	ProxyConfig?	nil	Proxy configuration for HTTP requests.
user_agents	Array<String>	\[\]	List of user-agent strings for rotation. If non-empty, overrides user_agent.
capture_screenshot	Boolean	false	Whether to capture a screenshot when using the browser. Only supported by scrape() with BrowserBackend.Chromiumoxide and BrowserMode.Always or Stealth. A screenshot is 100–500 KB of PNG per page, so crawl() does not carry screenshots in CrawlPageResult/CrawlResult at all — a multi-thousand-page crawl holding one per page in memory is not a safe default. Setting this with any other configuration (a different backend, BrowserMode.Auto/Never, or during crawl()) has no effect and logs a warning rather than silently doing nothing.
follow_document_urls	Boolean	false	Re-enqueue discovered LinkType.Document URLs into the crawl frontier so the crawl follows links from document pages (PDFs, etc.) as it would from HTML pages. Default: false (documents terminate at materialisation).
document_url_depth	Integer?	nil	Maximum document-depth (from the seed URL through document links only) when follow_document_urls is true. nil means inherit max_depth. Independent of max_depth: a document URL is enqueued only if BOTH the outer max_depth and (if set) document_url_depth permit it.
download_documents	Boolean	true	Whether to download non-HTML documents (PDF, DOCX, images, code, etc.) instead of skipping them. Defaults to true — unlike download_assets and capture_screenshot, which default to false.
document_max_size	Integer?	52428800	Maximum size in bytes for document downloads. Defaults to 50 MB.
document_mime_types	Array<String>	\[\]	Allowlist of MIME types to download. If empty, uses built-in defaults.
document_output_dir	String?	nil	Directory to stream downloaded document bytes into instead of holding them in memory on DownloadedDocument.content. When set, content is left empty and DownloadedDocument.content_path is populated with <dir>/<content_hash>.<ext>. nil (default) preserves today’s in-memory-only behavior. Has no effect on wasm32, which has no filesystem — use document_content_encoding there instead.
document_content_encoding	DocumentContentEncoding?	nil	Opt-in encoding that duplicates DownloadedDocument.content into a serializable field for language bindings that need the bytes in-memory (content itself is alef(skip)ed). nil (default) means no encoding is produced. Independent of document_output_dir — set both to get a file on disk and an in-memory copy.
warc_output	String?	nil	Path to write WARC output. If nil, WARC output is disabled.
browser_profile	String?	nil	Named browser profile for persistent sessions (cookies, localStorage). Chromiumoxide backend only. The native backend runs an in-process JavaScript engine with no Chrome process and therefore no profile directory, so this is ignored there and logs a warning. It is also ignored — with a warning — when a shared browser pool is in use (the pool launches before any per-crawl config exists) or when connecting to an external CDP endpoint whose process crawlberg does not own.
save_browser_profile	Boolean	false	Whether to save changes back to the browser profile on exit.
ssrf	SsrfPolicy	—	SSRF policy for outbound network requests. Default: deny private networks, allow http/https only, max 5 redirects. deny_private, allowlist and max_redirects are exposed to all language bindings. scheme_allowlist stays Rust-only — see SsrfPolicy. wasm32 (including Node.js): deny_private does not stop hostname-based requests. There is no DNS resolution on this target, so only a literal IP host is checked against the policy — a domain name is always permitted, regardless of deny_private. Under Node, where fetch enforces no CORS, this means a service embedding the wasm binding can be driven to internal hosts by domain name even with deny_private = true. Enforce egress restrictions at the network layer for that deployment target; do not rely on this field. See crawlberg.net.validate_url.
ssrf_deny_private_explicit	Boolean?	nil	Pins SsrfPolicy.deny_private to a caller-chosen value, bypassing the CRAWLBERG_ALLOW_PRIVATE_NETWORK operator override entirely for this config. ssrf.deny_private is a plain, always-serialized bool: several alef-generated bindings construct SsrfPolicy.default() (hardcoding deny_private: true) whenever their caller never touches SSRF settings at all, so true on that field alone cannot distinguish “the caller wants private networks denied” from “the binding’s own structural default landed on true”. The environment variable exists precisely to resolve that ambiguity in the common case by treating any true as inconclusive and deferring to the operator. Set this field when that default-deferral is wrong for your call — e.g. a test that must prove deny_private: true still denies even while the operator has set CRAWLBERG_ALLOW_PRIVATE_NETWORK suite-wide for every other call. nil (default) preserves today’s behavior: the environment variable may still flip ssrf.deny_private to false. Some(value) pins ssrf.deny_private to value and the environment variable is not consulted for this config.

Methods

default()

Signature:

def self.default()

Example:

result = CrawlConfig.default()

Returns: CrawlConfig

validate()

Validate the configuration, returning an error if any values are invalid.

Signature:

def validate()

Example:

instance.validate()

Returns: No return value.

Errors: Raises CrawlError.

---

CrawlEngineHandle

Opaque handle to a configured crawl engine.

Constructed via create_engine with an optional CrawlConfig. Default implementations for all pluggable components are used internally.

---

CrawlPageResult

The result of crawling a single page during a crawl operation.

Field	Type	Default	Description
url	String	—	The original URL of the page.
normalized_url	String	—	The normalized URL of the page.
status_code	Integer	—	The HTTP status code of the response.
content_type	String	—	The Content-Type header value.
html	String	—	The HTML body of the response.
body_size	Integer	—	The size of the response body in bytes.
metadata	PageMetadata	—	Extracted metadata from the page.
links	Array<LinkInfo>	\[\]	Links found on the page.
images	Array<ImageInfo>	\[\]	Images found on the page.
feeds	Array<FeedInfo>	\[\]	Feed links found on the page.
json_ld	Array<JsonLdEntry>	\[\]	JSON-LD entries found on the page.
depth	Integer	—	The depth of this page from the start URL.
stayed_on_domain	Boolean	—	Whether this page is on the same domain as the start URL.
was_skipped	Boolean	—	Whether this page was skipped (binary or PDF content).
is_pdf	Boolean	—	Whether the content is a PDF.
detected_charset	String?	nil	The detected character set encoding.
markdown	MarkdownResult?	nil	Markdown conversion of the page content.
extracted_data	Object?	nil	Structured data extracted by LLM. Populated when extraction is configured.
extraction_meta	ExtractionMeta?	nil	Metadata about the LLM extraction pass (cost, tokens, model).
downloaded_document	DownloadedDocument?	nil	Downloaded non-HTML document (PDF, DOCX, image, code, etc.).
browser_used	Boolean	—	Whether the browser fallback was used to fetch this page.

---

CrawlResult

The result of a multi-page crawl operation.

Field	Type	Default	Description
pages	Array<CrawlPageResult>	\[\]	The list of crawled pages.
final_url	String	—	The final URL after following redirects.
redirect_count	Integer	—	The number of redirects followed.
was_skipped	Boolean	—	Whether any page was skipped during crawling.
error	String?	nil	An error message, if the crawl encountered an issue.
cookies	Array<CookieInfo>	\[\]	Cookies collected during the crawl.
stayed_on_domain	Boolean	—	Whether all crawled pages stayed on the same domain as the start URL.
browser_used	Boolean	—	Whether the browser fallback was used for any page in this crawl.

Methods

unique_normalized_urls()

Returns the count of unique normalized URLs encountered during crawling.

Computed from pages (not the deprecated normalized_urls field) so it is correct across every binding that reconstructs CrawlResult from pages alone. In streaming mode pages is empty, so this returns 0 on the opaque-handle (C/Go/C#/Zig/Dart) path where it previously counted streamed pages — a known, accepted cost of making the other ten binding families correct.

Signature:

def unique_normalized_urls()

Example:

result = instance.unique_normalized_urls()

Returns: Integer

---

CrawlStreamRequest

Request to begin a single-URL streaming crawl.

Wraps a single seed URL for delivery through the streaming-adapter binding surface. Required as a struct because alef’s streaming adapter requires a named request type — primitives are not supported.

Field	Type	Default	Description
url	String	—	The seed URL to crawl.

---

DownloadedAsset

A downloaded asset from a page.

Field	Type	Default	Description
url	String	—	The original URL of the asset.
content_hash	String	—	The SHA-256 content hash of the asset.
mime_type	String?	nil	The MIME type from the Content-Type header.
size	Integer	—	The size of the asset in bytes.
asset_category	AssetCategory	:image	The category of the asset.
html_tag	String?	nil	The HTML tag that referenced this asset (e.g., “link”, “script”, “img”).

---

DownloadedDocument

A downloaded non-HTML document (PDF, DOCX, image, code file, etc.).

When the crawler encounters non-HTML content and download_documents is enabled, it downloads the raw bytes and populates this struct instead of skipping the resource.

Field	Type	Default	Description
url	String	—	The URL the document was fetched from.
mime_type	String	—	The MIME type from the Content-Type header.
size	Integer	—	Size of the document in bytes.
filename	String?	nil	Filename extracted from Content-Disposition or URL path.
content_hash	String	—	SHA-256 hex digest of the content.
headers	Hash{String=>String}	{}	Selected response headers.
truncated	Boolean	—	True when content (or the file at content_path) was truncated to document_max_size; size still reports the original, untruncated length.
content_path	String?	nil	Filesystem path the document was streamed to when document_output_dir was set. content is empty in memory when this is populated.
content_base64	String?	nil	Base64-encoded copy of content, populated only when document_content_encoding was set to Base64.

---

ExtractionMeta

Metadata about an LLM extraction pass.

Field	Type	Default	Description
cost	Float?	nil	Estimated cost of the LLM call in USD.
prompt_tokens	Integer?	nil	Number of prompt (input) tokens consumed.
completion_tokens	Integer?	nil	Number of completion (output) tokens generated.
model	String?	nil	The model identifier used for extraction.
chunks_processed	Integer	—	Number of content chunks sent to the LLM.

---

FaviconInfo

Information about a favicon or icon link.

Field	Type	Default	Description
url	String	—	The icon URL.
rel	String	—	The rel attribute (e.g., “icon”, “apple-touch-icon”).
sizes	String?	nil	The sizes attribute, if present.
mime_type	String?	nil	The MIME type, if present.

---

FeedInfo

Information about a feed link found on a page.

Field	Type	Default	Description
url	String	—	The feed URL.
title	String?	nil	The feed title, if present.
feed_type	FeedType	:rss	The type of feed.

---

HeadingInfo

A heading element extracted from the page.

Field	Type	Default	Description
level	Integer	—	The heading level (1-6).
text	String	—	The heading text content.

---

HreflangEntry

An hreflang alternate link entry.

Field	Type	Default	Description
lang	String	—	The language code (e.g., “en”, “fr”, “x-default”).
url	String	—	The URL for this language variant.

---

ImageInfo

Information about an image found on a page.

Field	Type	Default	Description
url	String	—	The image URL.
alt	String?	nil	The alt text, if present.
width	Integer?	nil	The width attribute, if present and parseable.
height	Integer?	nil	The height attribute, if present and parseable.
source	ImageSource	:img	The source of the image reference.

---

InteractionResult

Result of executing a sequence of page interaction actions.

Field	Type	Default	Description
action_results	Array<ActionResult>	\[\]	Results from each executed action.
final_html	String	—	Final page HTML after all actions completed.
final_url	String	—	Final page URL (may have changed due to navigation).
screenshot_base64	String?	nil	Base64-encoded PNG screenshot taken after all actions. Populated only when a PageAction.Screenshot action actually ran, so callers that never request a screenshot do not pay the encoding cost.

---

JsonLdEntry

A JSON-LD structured data entry found on a page.

Field	Type	Default	Description
schema_type	String	—	The @type value from the JSON-LD object.
name	String?	nil	The name value, if present.
raw	String	—	The raw JSON-LD string.

---

LinkInfo

Information about a link found on a page.

Field	Type	Default	Description
url	String	—	The resolved URL of the link.
text	String	—	The visible text of the link.
link_type	LinkType	:internal	The classification of the link.
rel	String?	nil	The rel attribute value, if present.
nofollow	Boolean	—	Whether the link has rel="nofollow".

---

MapResult

The result of a map operation, containing discovered URLs.

Field	Type	Default	Description
urls	Array<SitemapUrl>	\[\]	The list of discovered URLs.

---

MarkdownResult

Rich markdown conversion result from HTML processing.

Field	Type	Default	Description
content	String	—	Converted markdown text.
document_structure	Object?	nil	Structured document tree with semantic nodes.
tables	Array<Object>	\[\]	Extracted tables with structured cell data.
warnings	Array<String>	\[\]	Non-fatal processing warnings.
citations	Boolean	—	Whether citation conversion was applied and produced at least one reference. true when the markdown contained inline links that were converted to numbered citation references. The converted content (with \[N\] markers) is available in content; the full reference list is accessible via generate_citations if needed separately.
fit_content	String?	nil	Content-filtered markdown optimized for LLM consumption.

---

PageMetadata

Metadata extracted from an HTML page’s <meta> tags and <title> element.

Field	Type	Default	Description
title	String?	nil	The page title from the <title> element.
description	String?	nil	The meta description.
canonical_url	String?	nil	The canonical URL from <link rel="canonical">.
keywords	String?	nil	Keywords from <meta name="keywords">.
author	String?	nil	Author from <meta name="author">.
viewport	String?	nil	Viewport content from <meta name="viewport">.
theme_color	String?	nil	Theme color from <meta name="theme-color">.
generator	String?	nil	Generator from <meta name="generator">.
robots	String?	nil	Robots content from <meta name="robots">.
html_lang	String?	nil	The lang attribute from the <html> element.
html_dir	String?	nil	The dir attribute from the <html> element.
og_title	String?	nil	Open Graph title.
og_type	String?	nil	Open Graph type.
og_image	String?	nil	Open Graph image URL.
og_description	String?	nil	Open Graph description.
og_url	String?	nil	Open Graph URL.
og_site_name	String?	nil	Open Graph site name.
og_locale	String?	nil	Open Graph locale.
og_video	String?	nil	Open Graph video URL.
og_audio	String?	nil	Open Graph audio URL.
og_locale_alternates	Array<String>?	\[\]	Open Graph locale alternates.
twitter_card	String?	nil	Twitter card type.
twitter_title	String?	nil	Twitter title.
twitter_description	String?	nil	Twitter description.
twitter_image	String?	nil	Twitter image URL.
twitter_site	String?	nil	Twitter site handle.
twitter_creator	String?	nil	Twitter creator handle.
dc_title	String?	nil	Dublin Core title.
dc_creator	String?	nil	Dublin Core creator.
dc_subject	String?	nil	Dublin Core subject.
dc_description	String?	nil	Dublin Core description.
dc_publisher	String?	nil	Dublin Core publisher.
dc_date	String?	nil	Dublin Core date.
dc_type	String?	nil	Dublin Core type.
dc_format	String?	nil	Dublin Core format.
dc_identifier	String?	nil	Dublin Core identifier.
dc_language	String?	nil	Dublin Core language.
dc_rights	String?	nil	Dublin Core rights.
article	ArticleMetadata?	nil	Article metadata from article:* Open Graph tags.
hreflangs	Array<HreflangEntry>?	\[\]	Hreflang alternate links.
favicons	Array<FaviconInfo>?	\[\]	Favicon and icon links.
headings	Array<HeadingInfo>?	\[\]	Heading elements (h1-h6).
word_count	Integer?	nil	Computed word count of the page body text.

---

ProxyConfig

Proxy configuration for HTTP requests.

Field	Type	Default	Description
url	String	—	Proxy URL (e.g. “http://proxy:8080", “socks5://proxy:1080”).
username	String?	nil	Optional username for proxy authentication.
password	String?	nil	Optional password for proxy authentication.

---

ResponseMeta

Response metadata extracted from HTTP headers.

Field	Type	Default	Description
etag	String?	nil	The ETag header value.
last_modified	String?	nil	The Last-Modified header value.
cache_control	String?	nil	The Cache-Control header value.
server	String?	nil	The Server header value.
x_powered_by	String?	nil	The X-Powered-By header value.
content_language	String?	nil	The Content-Language header value.
content_encoding	String?	nil	The Content-Encoding header value.

---

ScrapeResult

The result of a single-page scrape operation.

Field	Type	Default	Description
status_code	Integer	—	The HTTP status code of the response.
final_url	String	—	The final URL after following all redirects.
content_type	String	—	The Content-Type header value.
html	String	—	The HTML body of the response.
body_size	Integer	—	The size of the response body in bytes.
metadata	PageMetadata	—	Extracted metadata from the page.
links	Array<LinkInfo>	\[\]	Links found on the page.
images	Array<ImageInfo>	\[\]	Images found on the page.
feeds	Array<FeedInfo>	\[\]	Feed links found on the page.
json_ld	Array<JsonLdEntry>	\[\]	JSON-LD entries found on the page.
is_allowed	Boolean	—	Whether the URL is allowed by robots.txt.
crawl_delay	Integer?	nil	The crawl delay from robots.txt, in seconds.
noindex_detected	Boolean	—	Whether a noindex directive was detected.
nofollow_detected	Boolean	—	Whether a nofollow directive was detected.
x_robots_tag	String?	nil	The X-Robots-Tag header value, if present.
is_pdf	Boolean	—	Whether the content is a PDF.
was_skipped	Boolean	—	Whether the page was skipped (binary or PDF content).
detected_charset	String?	nil	The detected character set encoding.
auth_header_sent	Boolean	—	Whether an authentication header was sent with the request.
response_meta	ResponseMeta?	nil	Response metadata extracted from HTTP headers.
assets	Array<DownloadedAsset>	\[\]	Downloaded assets from the page.
js_render_hint	Boolean	—	Whether the page content suggests JavaScript rendering is needed.
browser_used	Boolean	—	Whether the browser fallback was used to fetch this page.
markdown	MarkdownResult?	nil	Markdown conversion of the page content.
extracted_data	Object?	nil	Structured data extracted by LLM. Populated when extraction is configured.
extraction_meta	ExtractionMeta?	nil	Metadata about the LLM extraction pass (cost, tokens, model).
screenshot_base64	String?	nil	Base64-encoded PNG screenshot of the page. Populated only when CrawlConfig.capture_screenshot was enabled for this request, so callers that never requested a screenshot do not pay the encoding cost.
downloaded_document	DownloadedDocument?	nil	Downloaded non-HTML document (PDF, DOCX, image, code, etc.).
browser	BrowserExtras?	nil	Browser-specific extras (eval result, network events, cookies). Only populated when BrowserBackend.Native was used for this request.

---

SitemapUrl

A URL entry from a sitemap.

Field	Type	Default	Description
url	String	—	The URL.
lastmod	String?	nil	The last modification date, if present.
changefreq	String?	nil	The change frequency, if present.
priority	String?	nil	The priority, if present.

---

SsrfPolicy

SSRF policy configuration.

Field	Type	Default	Description
deny_private	Boolean	true	If true, reject URLs that resolve to private/metadata IP ranges.
allowlist	Array<HostMatcher>	\[\]	Hostnames and IP ranges permitted regardless of deny_private. The allowlist is an override of deny_private, not an intersection with it. Precedence, in order: 1. deny_private == false permits everything; the allowlist is not consulted. 2. A hostname matching an Exact or Suffix entry is permitted immediately, before DNS resolution — so the deny-list is never applied to it. This trusts the host string: a name that resolves into private space is still permitted. 3. A literal or resolved IP inside a Cidr entry is permitted even though it is in the default deny-list. 4. Otherwise the default deny-list decides. An empty allowlist therefore denies nothing by itself — it simply leaves deny_private and the deny-list in sole control.
max_redirects	Integer	5	Maximum number of HTTP redirects to follow during validation.

Methods

default()

Signature:

def self.default()

Example:

result = SsrfPolicy.default()

Returns: SsrfPolicy

from_env()

Create a policy from environment variables.

On native platforms, reads CRAWLBERG_ALLOW_PRIVATE_NETWORK — if set to “1” or “true” (case-insensitive), sets deny_private = false. Otherwise, defaults to deny_private = true.

On wasm32 targets (browser/Node.js), environment variables are not accessible to the compiled module. Defaults to deny_private = false because:

• Outbound requests in a browser go through the fetch API, which enforces its own network policies.
• Rust-side SSRF checking is unenforceable and redundant in a wasm32 context.
• For testing and localhost access, the host’s network sandbox is the enforcing boundary.

Node.js caveat: deny_private (whatever its value) has no effect on hostname-based requests under wasm32. There is no DNS resolution on this target, so validate_url only ever checks a literal IP host; a domain name falls straight through to Ok(()). In a browser this is covered by same-origin/CORS. Node’s fetch enforces no CORS, so a Node service embedding this wasm module can be driven to internal hosts by domain name even though deny_private = true. Do not rely on this policy to stop that in Node — enforce egress restrictions (network policy, firewall, proxy allowlist) outside the process.

Signature:

def self.from_env()

Example:

result = SsrfPolicy.from_env()

Returns: SsrfPolicy

---

Enums

BrowserMode

When to use the headless browser fallback.

Value	Description
auto	Automatically detect when JS rendering is needed and fall back to browser.
always	Always use the browser for every request.
never	Never use the browser fallback.
stealth	Always use the browser with all stealth surfaces enabled. Behaves like Always for escalation purposes (every request is routed through the browser tier), but additionally enables: - browser JavaScript stealth patches - native-backend TLS fingerprint spoofing - stealth-aware default user-agent when no explicit UA is set - 1920×1080 viewport override Use this instead of setting the now-removed BrowserConfig.stealth boolean field.

---

BrowserWait

Wait strategy for browser page rendering.

Value	Description
network_idle	Wait until network activity is idle.
selector	Wait for a specific CSS selector to appear in the DOM.
fixed	Wait for a fixed duration after navigation.

---

BrowserBackend

Browser backend used for JavaScript rendering.

Value	Description
chromiumoxide	Existing Chromium/CDP backend powered by chromiumoxide.
native	Crawlberg-owned native browser backend derived from Obscura.

---

DocumentContentEncoding

Opt-in encoding applied to a downloaded document’s bytes for callers who need the content available in a serializable field rather than reading it from disk.

nil (the CrawlConfig.document_content_encoding default) produces neither — unlike screenshots, base64-encoding a document by default would duplicate an already up-to-document_max_size buffer (50 MB default) in memory per document.

Value	Description
base64	Populate DownloadedDocument.content_base64 with a base64-encoded copy.

---

AuthConfig

Authentication configuration.

Value	Description
basic	HTTP Basic authentication. — Fields: username: String, password: String
bearer	Bearer token authentication. — Fields: token: String
header	Custom authentication header. — Fields: name: String, value: String

---

LinkType

The classification of a link.

Value	Description
internal	A link to the same domain.
external	A link to a different domain.
anchor	A fragment-only link (e.g., #section).
document	A link to a downloadable document (PDF, DOC, etc.).

---

ImageSource

The source of an image reference.

Value	Description
img	An <img> tag.
picture_source	A <source> tag inside <picture>.
og_image	An og:image meta tag.
twitter_image	A twitter:image meta tag.

---

FeedType

The type of a feed (RSS, Atom, or JSON Feed).

Value	Description
rss	RSS feed.
atom	Atom feed.
json_feed	JSON Feed.

---

AssetCategory

The category of a downloaded asset.

Value	Description
document	A document file (PDF, DOC, etc.).
image	An image file.
audio	An audio file.
video	A video file.
font	A font file.
stylesheet	A CSS stylesheet.
script	A JavaScript file.
archive	An archive file (ZIP, TAR, etc.).
data	A data file (JSON, XML, CSV, etc.).
other	An unrecognized asset type.

---

CrawlEvent

An event emitted during a streaming crawl operation.

Not available on wasm32 targets — streaming requires native concurrency primitives (tokio channels, JoinSet) that are not supported on wasm32.

Delivered to bindings through each target’s native streaming idiom.

Value	Description
page	A single page has been crawled. — Fields: result: CrawlPageResult
error	An error occurred while crawling a URL. — Fields: url: String, error: String
complete	The crawl has completed. — Fields: pages_crawled: Integer

---

PageAction

A single page interaction action.

Actions are serialized with a type tag using camelCase naming, except ExecuteJs which is explicitly renamed to "executeJs".

Value	Description
click	Click on an element matching the given CSS selector. — Fields: selector: String
type_text	Type text into an element matching the given CSS selector. — Fields: selector: String, text: String
press	Press a keyboard key (e.g. “Enter”, “Tab”, “Escape”). — Fields: key: String
scroll	Scroll the page or a specific element. — Fields: direction: ScrollDirection, selector: String, amount: Integer
wait	Wait for a duration or for an element to appear. — Fields: milliseconds: Integer, selector: String
screenshot	Take a screenshot of the current page. — Fields: full_page: Boolean
execute_js	Execute arbitrary JavaScript in the page context. Safety: The script runs with full page privileges in the browser context. Only execute scripts from trusted sources. — Fields: script: String
scrape	Scrape the current page HTML.

---

ScrollDirection

Direction for a scroll action.

Value	Description
up	Scroll upward.
down	Scroll downward.

---

HostMatcher

Hostname/IP allowlist matcher for SSRF policy.

Serializes as an internally-tagged object so each variant is distinguishable on the wire and round-trips losslessly:

{"type": "exact",  "value": "api.example.com"}
{"type": "suffix", "value": ".example.com"}
{"type": "cidr",   "value": "10.0.0.0/8"}

A bare JSON string is still accepted on deserialization and resolves to Exact, preserving configs written against the previous untagged representation.

Exact: HostMatcher.Exact

Value	Description
exact	Exact hostname match (case-insensitive). — Fields: value: String
suffix	Suffix match: “.xberg.io” matches “api.xberg.io” and “xberg.io”. — Fields: value: String
cidr	CIDR match: “10.0.0.0/8” matches IP addresses in that range. — Fields: value: String

---

Errors

CrawlError

Errors that can occur during crawling, scraping, or mapping operations.

Variant	Description
not_found	The requested page was not found (HTTP 404).
unauthorized	The request was unauthorized (HTTP 401).
forbidden	The request was forbidden (HTTP 403).
waf_blocked	The request was blocked by a WAF or bot protection (HTTP 403 with WAF indicators). vendor is the lowercase identifier of the detected WAF (e.g. “cloudflare”, “datadome”). When the engine cannot identify the vendor, it uses “unknown”. message is the freeform description for logs and human readers. The stable error tag remains forbidden: waf/blocked: MESSAGE so existing log-grep patterns and cross-language bindings continue to work; vendor is surfaced separately for structured consumers.
timeout	The request timed out.
rate_limited	The request was rate-limited (HTTP 429).
server_error	A server error occurred (HTTP 5xx).
bad_gateway	A bad gateway error occurred (HTTP 502).
gone	The resource is permanently gone (HTTP 410).
connection	A connection error occurred.
dns	A DNS resolution error occurred.
ssl	An SSL/TLS error occurred.
data_loss	Data was lost or truncated during transfer.
browser_error	The browser failed to launch, connect, or navigate.
browser_timeout	The browser page load or rendering timed out.
invalid_config	The provided configuration is invalid.
unsupported	The requested capability is not supported by the active backend or build.
ssrf_policy_violation	A URL was rejected by SSRF policy (private IP, metadata, disallowed scheme, etc).
other	An unclassified error occurred.

---

SsrfError

SSRF validation error.

Variant	Description
denied_by_policy	URL denied by SSRF policy: private IP, metadata IP, etc.
not_on_allowlist	Host not on allowlist when an allowlist is configured.
invalid_cidr	Allowlist entry is not a parseable CIDR block.
dns_resolution_failed	DNS resolution failed for hostname.
invalid_url	Invalid URL format.
disallowed_scheme	URL scheme not in allowlist (e.g., ftp:// when only http/https allowed).
too_many_redirects	Too many HTTP redirects encountered during validation.

---