C API Reference
C API Reference v1.8.0
Section titled “C API Reference v1.8.0”Functions
Section titled “Functions”cberg_generate_citations()
Section titled “cberg_generate_citations()”Convert markdown links to numbered citations.
[Example](https://example.com) becomes Example[1]
with [1]: <https://example.com> in the reference list.
Images !alt are preserved unchanged.
Signature:
CBERGAlefHandle cberg_generate_citations(const char* markdown);Example:
CBERGAlefHandle result = cberg_generate_citations("value");Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
markdown |
const char* |
Yes | The markdown |
Returns: CBERGAlefHandle
cberg_create_engine()
Section titled “cberg_create_engine()”Create a new crawl engine with the given configuration.
If config is NULL, uses CrawlConfig.default().
Returns an error if the configuration is invalid.
Signature:
CBERGAlefHandle cberg_create_engine(CBERGAlefHandle config);Example:
CBERGAlefHandle result = cberg_create_engine(0);Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
config |
CBERGAlefHandle |
No | The configuration options |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_scrape()
Section titled “cberg_scrape()”Scrape a single URL, returning extracted page data.
Signature:
CBERGAlefHandle cberg_scrape(CBERGAlefHandle engine, const char* url);Example:
CBERGAlefHandle result = cberg_scrape(0, "value");Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
url |
const char* |
Yes | The URL to fetch |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_crawl()
Section titled “cberg_crawl()”Crawl a website starting from url, following links up to the configured depth.
Signature:
CBERGAlefHandle cberg_crawl(CBERGAlefHandle engine, const char* url);Example:
CBERGAlefHandle result = cberg_crawl(0, "value");Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
url |
const char* |
Yes | The URL to fetch |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_map_urls()
Section titled “cberg_map_urls()”Discover all pages on a website by following links and sitemaps.
Signature:
CBERGAlefHandle cberg_map_urls(CBERGAlefHandle engine, const char* url);Example:
CBERGAlefHandle result = cberg_map_urls(0, "value");Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
url |
const char* |
Yes | The URL to fetch |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_interact()
Section titled “cberg_interact()”Execute browser actions on a single page.
Signature:
CBERGAlefHandle cberg_interact(CBERGAlefHandle engine, const char* url, const char* actions);Example:
CBERGAlefHandle result = cberg_interact(0, "value", NULL);Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
url |
const char* |
Yes | The URL to fetch |
actions |
const char* |
Yes | The actions |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_batch_scrape()
Section titled “cberg_batch_scrape()”Scrape multiple URLs concurrently.
Signature:
CBERGAlefHandle cberg_batch_scrape(CBERGAlefHandle engine, const char* urls);Example:
CBERGAlefHandle result = cberg_batch_scrape(0, NULL);Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
urls |
const char* |
Yes | The urls |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
cberg_batch_crawl()
Section titled “cberg_batch_crawl()”Crawl multiple seed URLs concurrently, each following links to configured depth.
Signature:
CBERGAlefHandle cberg_batch_crawl(CBERGAlefHandle engine, const char* urls);Example:
CBERGAlefHandle result = cberg_batch_crawl(0, NULL);Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
engine |
CBERGAlefHandle |
Yes | The crawl engine handle |
urls |
const char* |
Yes | The urls |
Returns: CBERGAlefHandle
Errors: Returns the sentinel handle 0 on error.
CBERGActionResult
Section titled “CBERGActionResult”C representation: CBERGActionResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGActionResult does not appear anywhere in the generated header.
Result from a single page action execution.
| Field | Type | Default | Description |
|---|---|---|---|
action_index |
uintptr_t |
— | Zero-based index of the action in the sequence. |
action_type |
const char* |
— | The type of action that was executed. |
success |
int32_t |
— | Whether the action completed successfully. |
data |
const char* |
NULL |
Action-specific return data (screenshot bytes, JS return value, scraped HTML). |
error |
const char* |
NULL |
Error message if the action failed. |
CBERGArticleMetadata
Section titled “CBERGArticleMetadata”C representation: CBERGArticleMetadata is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGArticleMetadata does not appear anywhere in the generated header.
Article metadata extracted from article:* Open Graph tags.
| Field | Type | Default | Description |
|---|---|---|---|
published_time |
const char* |
NULL |
The article publication time. |
modified_time |
const char* |
NULL |
The article modification time. |
author |
const char* |
NULL |
The article author. |
section |
const char* |
NULL |
The article section. |
tags |
const char* |
NULL |
The article tags. |
CBERGBatchCrawlResult
Section titled “CBERGBatchCrawlResult”C representation: CBERGBatchCrawlResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBatchCrawlResult does not appear anywhere in the generated header.
Result from a single URL in a batch crawl operation.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The seed URL that was crawled. |
result |
CBERGAlefHandle |
NULL |
The crawl result, if successful. |
error |
const char* |
NULL |
The error message, if the crawl failed. |
CBERGBatchCrawlResults
Section titled “CBERGBatchCrawlResults”C representation: CBERGBatchCrawlResults is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBatchCrawlResults does not appear anywhere in the generated header.
Aggregate result of a batch crawl, exposing per-URL results plus precomputed counts.
The counts are derived once at construction so every binding language can read them
as plain integer fields without re-iterating the results vector.
| Field | Type | Default | Description |
|---|---|---|---|
results |
const char* |
NULL |
Per-URL crawl results, in the order seed URLs were submitted. |
total_count |
uintptr_t |
— | Total number of seed URLs in the batch (equal to results.len()). |
completed_count |
uintptr_t |
— | Number of seed URLs whose crawl succeeded (error is NULL). |
failed_count |
uintptr_t |
— | Number of seed URLs whose crawl failed (error is Some). |
CBERGBatchCrawlStreamRequest
Section titled “CBERGBatchCrawlStreamRequest”C representation: CBERGBatchCrawlStreamRequest is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBatchCrawlStreamRequest does not appear anywhere in the generated header.
Request to begin a multi-URL streaming crawl.
Wraps a set of seed URLs for delivery through the streaming-adapter binding surface. Required as a struct because alef’s streaming adapter requires a named request type — primitives are not supported.
| Field | Type | Default | Description |
|---|---|---|---|
urls |
const char* |
NULL |
The seed URLs to crawl. Each URL is followed independently up to the engine’s configured depth. |
CBERGBatchScrapeResult
Section titled “CBERGBatchScrapeResult”C representation: CBERGBatchScrapeResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBatchScrapeResult does not appear anywhere in the generated header.
Result from a single URL in a batch scrape operation.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The URL that was scraped. |
result |
CBERGAlefHandle |
NULL |
The scrape result, if successful. |
error |
const char* |
NULL |
The error message, if the scrape failed. |
CBERGBatchScrapeResults
Section titled “CBERGBatchScrapeResults”C representation: CBERGBatchScrapeResults is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBatchScrapeResults does not appear anywhere in the generated header.
Aggregate result of a batch scrape, exposing per-URL results plus precomputed counts.
The counts are derived once at construction so every binding language can read them
as plain integer fields without re-iterating the results vector.
| Field | Type | Default | Description |
|---|---|---|---|
results |
const char* |
NULL |
Per-URL scrape results, in the order URLs were submitted. |
total_count |
uintptr_t |
— | Total number of URLs in the batch (equal to results.len()). |
completed_count |
uintptr_t |
— | Number of URLs whose scrape succeeded (error is NULL). |
failed_count |
uintptr_t |
— | Number of URLs whose scrape failed (error is Some). |
CBERGBrowserConfig
Section titled “CBERGBrowserConfig”C representation: CBERGBrowserConfig is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBrowserConfig does not appear anywhere in the generated header.
Browser fallback configuration.
| Field | Type | Default | Description |
|---|---|---|---|
mode |
CBERGAlefHandle |
CBERG_AUTO |
When to use the headless browser fallback. |
backend |
CBERGAlefHandle |
CBERG_CHROMIUMOXIDE |
Browser backend used to render JavaScript-heavy pages. |
endpoint |
const char* |
NULL |
CDP WebSocket endpoint for connecting to an external browser instance. |
timeout |
uint64_t |
30000ms |
Timeout for browser page load and rendering (in milliseconds when serialized). |
overall_timeout |
uint64_t |
60000ms |
Overall deadline for a single browser fetch, covering browser launch (or page acquisition from a shared pool), page setup, navigation, rendering, and screenshot capture. Must exceed timeout to leave room for launch and setup overhead; a fetch that has not returned within this deadline fails with a timeout error (in milliseconds when serialized). Shutdown/teardown is governed separately by shutdown_timeout and is not counted against this deadline: an already-computed result is delivered to the caller without waiting for the browser process to exit. |
shutdown_timeout |
uint64_t |
5000ms |
How long to wait for the browser process to close and exit cleanly during teardown before the process is forcibly killed (in milliseconds when serialized). |
wait |
CBERGAlefHandle |
CBERG_NETWORK_IDLE |
Wait strategy after browser navigation. |
wait_selector |
const char* |
NULL |
CSS selector to wait for when wait is Selector. |
extra_wait |
uint64_t* |
NULL |
Extra time to wait after the wait condition is met. |
proxy |
CBERGAlefHandle |
NULL |
Proxy for browser fetches. Overrides CrawlConfig.proxy when set. Native backend supports http/https only (no SOCKS5). |
block_url_patterns |
const char* |
NULL |
URL patterns to block before the network request fires. Supports * wildcards. Useful for skipping ads/analytics/large images. Honored by BrowserBackend.Native; chromiumoxide ignores this field today. |
eval_script |
const char* |
NULL |
JavaScript snippet evaluated after navigation completes. Scraping captures the native backend result in ScrapeResult.browser.eval_result. Interactions run this script before page actions on both browser backends but do not include the script result in InteractionResult. |
robots_user_agent |
const char* |
NULL |
User-agent used when fetching robots.txt. Defaults to BrowserConfig.user_agent (or crawlberg’s default) if unset. Native only. |
capture_network_events |
int32_t |
false |
Capture the full network event stream into the result. Default false (only the document event is captured). Native only. |
session_affinity |
int32_t |
true |
Enable session affinity: reuse chromiumoxide Pages for same-domain requests so cookies + fingerprint + solved challenges persist. Default: true. When false, each request gets a fresh Page. |
CBERGBrowserExtras
Section titled “CBERGBrowserExtras”C representation: CBERGBrowserExtras is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGBrowserExtras does not appear anywhere in the generated header.
Browser-specific extras populated when the native browser backend was used.
Available on ScrapeResult.browser when BrowserBackend.Native handled the request.
| Field | Type | Default | Description |
|---|---|---|---|
eval_result |
const char* |
NULL |
Return value of BrowserConfig.eval_script, if provided. |
network_events |
const char* |
NULL |
Network events captured during page navigation (only populated when BrowserConfig.capture_network_events is true). |
cookies |
const char* |
NULL |
All non-expired cookies present in the browser’s cookie jar after navigation completes (includes both prior cookies and server Set-Cookie). |
CBERGCitationReference
Section titled “CBERGCitationReference”C representation: CBERGCitationReference is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCitationReference does not appear anywhere in the generated header.
A single numbered reference in a citation list — produced by the citation
extractor when content uses inline [N]-style markers.
| Field | Type | Default | Description |
|---|---|---|---|
index |
uintptr_t |
— | 1-based reference number as it appears in the source text. |
url |
const char* |
— | Resolved absolute URL for this reference. |
text |
const char* |
— | Human-readable anchor text or title for the reference. |
CBERGCitationResult
Section titled “CBERGCitationResult”C representation: CBERGCitationResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCitationResult does not appear anywhere in the generated header.
Result of citation conversion.
| Field | Type | Default | Description |
|---|---|---|---|
content |
const char* |
— | Markdown with links replaced by numbered citations. |
references |
const char* |
NULL |
Numbered reference list: (index, url, text). |
CBERGContentConfig
Section titled “CBERGContentConfig”C representation: CBERGContentConfig is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGContentConfig does not appear anywhere in the generated header.
Content extraction and conversion configuration.
Controls how HTML is converted to the output format. Uses html-to-markdown-rs as the conversion engine for all formats (markdown, plain text, djot).
| Field | Type | Default | Description |
|---|---|---|---|
output_format |
const char* |
"markdown" |
Output format: "markdown" (default), "plain", "djot". |
preprocessing_preset |
const char* |
"standard" |
Preprocessing aggressiveness: "minimal", "standard" (default), "aggressive". - Minimal: only scripts/styles removed. - Standard: also removes nav, nav-hinted headers/footers/asides, forms. - Aggressive: removes all footers/asides unconditionally. |
remove_navigation |
int32_t |
true |
Remove navigation elements (nav, breadcrumbs, menus). Default: true. |
remove_forms |
int32_t |
true |
Remove form elements. Default: true. |
strip_tags |
const char* |
NULL |
HTML tag names to strip (render children only, remove the tag wrapper). Default: []. |
preserve_tags |
const char* |
NULL |
HTML tag names to preserve as raw HTML in output. |
exclude_selectors |
const char* |
["noscript"] |
CSS selectors for elements to exclude entirely (element + all content). Unlike strip_tags (which removes the wrapper but keeps children), excluded elements and all descendants are dropped. Supports CSS selectors: .class, #id, [attribute], compound selectors. Default: ["noscript"]. <noscript> fallback content (no-JS notices, tracking pixels, GTM iframes) is meant for browsers with JavaScript disabled, not for a markdown reader, and strip_tags cannot drop it — on preprocessing_preset: "standard" (crawlberg’s only path) it only removes the wrapper and still renders the children. Example: [".cookie-banner", "#ad-container", "[role='complementary']"] |
skip_images |
int32_t |
false |
Skip image elements in output. Default: false. |
max_depth |
uintptr_t* |
NULL |
Max DOM traversal depth. Prevents stack overflow on deeply nested HTML. |
wrap |
int32_t |
false |
Enable line wrapping. Default: false. |
wrap_width |
uintptr_t |
80 |
Wrap width when wrap is enabled. Default: 80. |
include_document_structure |
int32_t |
true |
Include document structure tree in output. Default: true. |
extract_metadata |
int32_t |
true |
Prepend a YAML frontmatter block (title, description, etc., extracted from <head>) to the markdown output. Default: true. This only controls the frontmatter text inside markdown.content. Crawlberg never reads <head> metadata back out of the converter’s result – PageMetadata is populated independently by crate.html.metadata.extract_metadata from the parsed DOM, so turning this off does not lose any metadata field. |
CBERGCookieInfo
Section titled “CBERGCookieInfo”C representation: CBERGCookieInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCookieInfo does not appear anywhere in the generated header.
Information about an HTTP cookie received from a response.
| Field | Type | Default | Description |
|---|---|---|---|
name |
const char* |
— | The cookie name. |
value |
const char* |
— | The cookie value. |
domain |
const char* |
NULL |
The cookie domain, if specified. |
path |
const char* |
NULL |
The cookie path, if specified. |
CBERGCrawlConfig
Section titled “CBERGCrawlConfig”C representation: CBERGCrawlConfig is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCrawlConfig does not appear anywhere in the generated header.
Configuration for crawl, scrape, and map operations.
| Field | Type | Default | Description |
|---|---|---|---|
max_depth |
uintptr_t* |
NULL |
Maximum crawl depth (number of link hops from the start URL). |
max_pages |
uintptr_t* |
NULL |
Maximum number of pages to crawl. |
max_links_per_page |
uintptr_t* |
NULL |
Maximum links enqueued from a single page. Defaults to 10000. Bounds the work one hostile or pathological page can create; links past the cap are dropped and a warning is logged. |
max_concurrent |
uintptr_t* |
NULL |
Maximum number of concurrent requests. |
crawl_strategy |
CBERGAlefHandle |
CBERG_BFS |
Traversal order. Defaults to breadth-first. A frontier or strategy set explicitly on CrawlEngineBuilder takes precedence over this field. |
content_filter |
CBERGAlefHandle |
NULL |
Content filter applied to each page. NULL keeps every page. A content filter set explicitly on CrawlEngineBuilder takes precedence. |
bm25_query |
const char* |
NULL |
Query the BM25 content filter scores pages against. Required by ContentFilterKind.Bm25. |
bm25_threshold |
double* |
NULL |
Minimum BM25 score a page must reach to be kept. Defaults to 0.0. |
respect_robots_txt |
int32_t |
false |
Whether to respect robots.txt directives. |
soft_http_errors |
int32_t |
false |
When true, HTTP-level error responses (404 NotFound, 403 Forbidden, WAF blocks) are surfaced as ScrapeResult records with the matching status_code rather than raised as CrawlError. Default false preserves the historical throw-on-error contract for direct fetches. Independently of this flag, 404s reached at the end of a redirect chain are always surfaced softly — the user opted into redirect-following, so receiving a 404 there is part of the normal flow rather than an unexpected error. |
user_agent |
const char* |
NULL |
Custom user-agent string. |
stay_on_domain |
int32_t |
false |
Whether to confine document links (.pdf, .docx, .zip, …) to the seed domain. Page links are always confined to the seed host, widened to its subdomains by Self.allow_subdomains; this flag does not loosen that. It applies only to document links, which are classified by file extension before their host is considered and so are followed cross-host by default – the usual case being documents served from a CDN or object store. Set this to true to require documents to live on the seed domain too. |
allow_subdomains |
int32_t |
false |
Whether subdomains of the seed host are in scope. Applies to page links unconditionally, and to document links when Self.stay_on_domain is set. |
include_paths |
const char* |
NULL |
Regex patterns for paths to include during crawling. |
exclude_paths |
const char* |
NULL |
Regex patterns for paths to exclude during crawling. |
path_patterns_match_query |
int32_t |
false |
Whether include_paths/exclude_paths match against path?query instead of just path. Defaults to false, matching path only: a pattern anchored with $ (e.g. /feed/?$) changes meaning once the query joins the matched text, so this must stay opt-in rather than silently changing what an existing config matches. |
dedup_include_query |
int32_t |
false |
Whether the crawl-dedup key includes the (sorted) query string. Defaults to false, matching historical behavior: /item?id=1 and /item?id=2 are treated as one page and only the first is fetched. true keeps the query, sorted, in the key, so each distinct query is fetched once. |
strip_tracking_params |
int32_t |
false |
Whether to strip tracking_params from a discovered URL before it is deduplicated, fetched, and reported. Defaults to false, so no tracking parameters are stripped unless explicitly enabled. |
tracking_params |
const char* |
["utm_*", "fbclid", "gclid", "ref"] |
Query parameter name patterns to strip when strip_tracking_params is true. A pattern ending in * matches by prefix (utm_* matches utm_source, utm_campaign, …); any other pattern matches the parameter name exactly. Defaults to ["utm_*", "fbclid", "gclid", "ref"], applied only once strip_tracking_params is enabled. |
custom_headers |
const char* |
NULL |
Custom HTTP headers to send with each request. |
request_timeout |
uint64_t |
30000ms |
Timeout for individual HTTP requests (in milliseconds when serialized). |
rate_limit_ms |
uint64_t* |
NULL |
Per-domain rate limit in milliseconds. When set, enforces a minimum delay between requests to the same domain. Defaults to 200ms when NULL. |
max_redirects |
uintptr_t |
10 |
Maximum number of redirects to follow. |
retry_count |
uintptr_t |
0 |
Number of retry attempts for failed requests. Bounded by MAX_RETRY_COUNT. |
retry_codes |
const char* |
NULL |
HTTP status codes that should trigger a retry. |
retry_initial_delay_ms |
uint64_t |
100 |
Initial delay, in milliseconds, before the first retry. Doubled on each subsequent attempt (capped at retry_max_delay_ms). Defaults to 100ms. |
retry_max_delay_ms |
uint64_t |
60000 |
Upper bound, in milliseconds, on the exponential retry backoff. Defaults to 60s. |
rate_limit_jitter_ratio |
double |
0 |
Fraction of the per-domain rate-limit delay to randomly jitter by, in [0.0, 1.0]. 0.0 (the default) applies no jitter and preserves the previous fixed-interval behaviour; 0.1 jitters the delay by up to ±10%. |
cookies_enabled |
int32_t |
false |
Whether to enable cookie handling. |
auth |
CBERGAlefHandle |
NULL |
Authentication configuration. |
max_body_size |
uintptr_t* |
NULL |
Maximum response body size in bytes. NULL does not mean unbounded: an unset cap falls back to a 100 MiB safety ceiling, because HTTP responses are decompressed while being read and a few hundred compressed bytes can otherwise expand to gigabytes in memory. To read bodies larger than that, set this explicitly. |
remove_tags |
const char* |
NULL |
CSS selectors for tags to remove from HTML before processing. |
content |
CBERGAlefHandle |
— | Content extraction and conversion configuration. |
map_limit |
uintptr_t* |
NULL |
Maximum number of URLs to return from a map operation. |
map_search |
const char* |
NULL |
Search filter for map results (case-insensitive substring match on URLs). |
download_assets |
int32_t |
false |
Whether to download assets (CSS, JS, images, etc.) from the page. |
asset_types |
const char* |
NULL |
Filter for asset categories to download. |
max_asset_size |
uintptr_t* |
NULL |
Maximum size in bytes for individual asset downloads. |
browser |
CBERGAlefHandle |
— | Browser configuration. |
proxy |
CBERGAlefHandle |
NULL |
Proxy configuration for HTTP requests. |
user_agents |
const char* |
NULL |
List of user-agent strings for rotation. If non-empty, overrides user_agent. |
capture_screenshot |
int32_t |
false |
Whether to capture a screenshot when using the browser. Only supported by scrape() with BrowserBackend.Chromiumoxide and BrowserMode.Always or Stealth. A screenshot is 100–500 KB of PNG per page, so crawl() does not carry screenshots in CrawlPageResult/CrawlResult at all — a multi-thousand-page crawl holding one per page in memory is not a safe default. Setting this with any other configuration (a different backend, BrowserMode.Auto/Never, or during crawl()) has no effect and logs a warning rather than silently doing nothing. |
follow_document_urls |
int32_t |
false |
Re-enqueue discovered LinkType.Document URLs into the crawl frontier so the crawl follows links from document pages (PDFs, etc.) as it would from HTML pages. Default: false (documents terminate at materialisation). |
document_url_depth |
uint32_t* |
NULL |
Maximum document-depth (from the seed URL through document links only) when follow_document_urls is true. NULL means inherit max_depth. Independent of max_depth: a document URL is enqueued only if BOTH the outer max_depth and (if set) document_url_depth permit it. |
download_documents |
int32_t |
true |
Whether to download non-HTML documents (PDF, DOCX, images, code, etc.) instead of skipping them. Defaults to true — unlike download_assets and capture_screenshot, which default to false. |
document_max_size |
uintptr_t* |
52428800 |
Maximum size in bytes for document downloads. Defaults to 50 MB. |
document_mime_types |
const char* |
NULL |
Allowlist of MIME types to download. If empty, uses built-in defaults. |
document_output_dir |
const char* |
NULL |
Directory to stream downloaded document bytes into instead of holding them in memory on DownloadedDocument.content. When set, content is left empty and DownloadedDocument.content_path is populated with <dir>/<content_hash>.<ext>. NULL (default) preserves today’s in-memory-only behavior. Has no effect on wasm32, which has no filesystem — use document_content_encoding there instead. |
document_content_encoding |
CBERGAlefHandle |
NULL |
Opt-in encoding that duplicates DownloadedDocument.content into a serializable field for language bindings that need the bytes in-memory (content itself is alef(skip)ed). NULL (default) means no encoding is produced. Independent of document_output_dir — set both to get a file on disk and an in-memory copy. |
warc_output |
const char* |
NULL |
Path to write WARC output. If NULL, WARC output is disabled. |
browser_profile |
const char* |
NULL |
Named browser profile for persistent sessions (cookies, localStorage). Chromiumoxide backend only. The native backend runs an in-process JavaScript engine with no Chrome process and therefore no profile directory, so this is ignored there and logs a warning. It is also ignored — with a warning — when a shared browser pool is in use (the pool launches before any per-crawl config exists) or when connecting to an external CDP endpoint whose process crawlberg does not own. |
save_browser_profile |
int32_t |
false |
Whether to save changes back to the browser profile on exit. |
ssrf |
CBERGAlefHandle |
crawlberg::SsrfPolicy::from_env() |
SSRF policy for outbound network requests. Default: deny private networks, allow http/https only, max 5 redirects. All policy fields are exposed to language bindings. wasm32 (including Node.js): deny_private does not stop hostname-based requests. There is no DNS resolution on this target, so only a literal IP host is checked against the policy — a domain name is always permitted, regardless of deny_private. Under Node, where fetch enforces no CORS, this means a service embedding the wasm binding can be driven to internal hosts by domain name even with deny_private = true. Enforce egress restrictions at the network layer for that deployment target; do not rely on this field. See crawlberg.net.validate_url. |
ssrf_deny_private_explicit |
int32_t* |
NULL |
Pins SsrfPolicy.deny_private to a caller-chosen value, bypassing the CRAWLBERG_ALLOW_PRIVATE_NETWORK operator override entirely for this config. ssrf.deny_private is a plain, always-serialized bool: several alef-generated bindings construct SsrfPolicy.default() (hardcoding deny_private: true) whenever their caller never touches SSRF settings at all, so true on that field alone cannot distinguish “the caller wants private networks denied” from “the binding’s own structural default landed on true”. The environment variable exists precisely to resolve that ambiguity in the common case by treating any true as inconclusive and deferring to the operator. Set this field when that default-deferral is wrong for your call — e.g. a test that must prove deny_private: true still denies even while the operator has set CRAWLBERG_ALLOW_PRIVATE_NETWORK suite-wide for every other call. NULL (default) preserves today’s behavior: the environment variable may still flip ssrf.deny_private to false. Some(value) pins ssrf.deny_private to value and the environment variable is not consulted for this config. |
CBERGCrawlEngineHandle
Section titled “CBERGCrawlEngineHandle”C representation: CBERGCrawlEngineHandle is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCrawlEngineHandle does not appear anywhere in the generated header.
Opaque handle to a configured crawl engine.
Constructed via create_engine with an optional CrawlConfig.
Default implementations for all pluggable components are used internally.
CBERGCrawlPageResult
Section titled “CBERGCrawlPageResult”C representation: CBERGCrawlPageResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCrawlPageResult does not appear anywhere in the generated header.
The result of crawling a single page during a crawl operation.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The original URL of the page. |
normalized_url |
const char* |
— | The normalized URL of the page. |
status_code |
uint16_t |
— | The HTTP status code of the response. |
content_type |
const char* |
— | The Content-Type header value. |
html |
const char* |
— | The HTML body of the response. |
body_size |
uintptr_t |
— | The size of the response body in bytes. |
metadata |
CBERGAlefHandle |
— | Extracted metadata from the page. |
links |
const char* |
NULL |
Links found on the page. |
images |
const char* |
NULL |
Images found on the page. |
feeds |
const char* |
NULL |
Feed links found on the page. |
json_ld |
const char* |
NULL |
JSON-LD entries found on the page. |
depth |
uintptr_t |
— | The depth of this page from the start URL. |
stayed_on_domain |
int32_t |
— | Whether this page is on the same domain as the start URL. |
was_skipped |
int32_t |
— | Whether this page was skipped (binary or PDF content). |
is_pdf |
int32_t |
— | Whether the content is a PDF. |
detected_charset |
const char* |
NULL |
The detected character set encoding. |
markdown |
CBERGAlefHandle |
NULL |
Markdown conversion of the page content. |
extracted_data |
const char* |
NULL |
Structured data extracted by LLM. Populated when extraction is configured. |
extraction_meta |
CBERGAlefHandle |
NULL |
Metadata about the LLM extraction pass (cost, tokens, model). |
downloaded_document |
CBERGAlefHandle |
NULL |
Downloaded non-HTML document (PDF, DOCX, image, code, etc.). |
browser_used |
int32_t |
— | Whether the browser fallback was used to fetch this page. |
final_url |
const char* |
— | The URL this page’s content was actually fetched from, after following any HTTP, Refresh header, or <meta http-equiv="refresh"> redirect url pointed at. Equal to url when the fetch did not redirect. |
redirect_count |
uintptr_t |
— | Redirect hops taken to reach final_url from url. |
CBERGCrawlResult
Section titled “CBERGCrawlResult”C representation: CBERGCrawlResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCrawlResult does not appear anywhere in the generated header.
The result of a multi-page crawl operation.
| Field | Type | Default | Description |
|---|---|---|---|
pages |
const char* |
NULL |
The list of crawled pages. |
final_url |
const char* |
— | The final URL after following redirects. |
redirect_count |
uintptr_t |
— | The number of redirects followed. |
was_skipped |
int32_t |
— | Whether any page was skipped during crawling. |
error |
const char* |
NULL |
An error message, if the crawl encountered an issue. |
cookies |
const char* |
NULL |
Cookies collected during the crawl. |
stayed_on_domain |
int32_t |
— | Whether all crawled pages stayed on the same domain as the start URL. |
browser_used |
int32_t |
— | Whether the browser fallback was used for any page in this crawl. |
CBERGCrawlStreamRequest
Section titled “CBERGCrawlStreamRequest”C representation: CBERGCrawlStreamRequest is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGCrawlStreamRequest does not appear anywhere in the generated header.
Request to begin a single-URL streaming crawl.
Wraps a single seed URL for delivery through the streaming-adapter binding surface. Required as a struct because alef’s streaming adapter requires a named request type — primitives are not supported.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The seed URL to crawl. |
CBERGDownloadedAsset
Section titled “CBERGDownloadedAsset”C representation: CBERGDownloadedAsset is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGDownloadedAsset does not appear anywhere in the generated header.
A downloaded asset from a page.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The original URL of the asset. |
content_hash |
const char* |
— | The SHA-256 content hash of the asset. |
mime_type |
const char* |
NULL |
The MIME type from the Content-Type header. |
size |
uintptr_t |
— | The size of the asset in bytes. |
asset_category |
CBERGAlefHandle |
CBERG_IMAGE |
The category of the asset. |
html_tag |
const char* |
NULL |
The HTML tag that referenced this asset (e.g., “link”, “script”, “img”). |
CBERGDownloadedDocument
Section titled “CBERGDownloadedDocument”C representation: CBERGDownloadedDocument is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGDownloadedDocument does not appear anywhere in the generated header.
A downloaded non-HTML document (PDF, DOCX, image, code file, etc.).
When the crawler encounters non-HTML content and download_documents is
enabled, it downloads the raw bytes and populates this struct instead of
skipping the resource.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The URL the document was fetched from. |
mime_type |
const char* |
— | The MIME type from the Content-Type header. |
size |
uintptr_t |
— | Size of the document in bytes. |
filename |
const char* |
NULL |
Filename extracted from Content-Disposition or URL path. |
content_hash |
const char* |
— | SHA-256 hex digest of the content. |
headers |
const char* |
NULL |
Selected response headers. |
truncated |
int32_t |
— | True when content (or the file at content_path) was truncated to document_max_size; size still reports the original, untruncated length. |
content_path |
const char* |
NULL |
Filesystem path the document was streamed to when document_output_dir was set. content is empty in memory when this is populated. |
content_base64 |
const char* |
NULL |
Base64-encoded copy of content, populated only when document_content_encoding was set to Base64. |
CBERGExtractionMeta
Section titled “CBERGExtractionMeta”C representation: CBERGExtractionMeta is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGExtractionMeta does not appear anywhere in the generated header.
Metadata about an LLM extraction pass.
| Field | Type | Default | Description |
|---|---|---|---|
cost |
double* |
NULL |
Estimated cost of the LLM call in USD. |
prompt_tokens |
uint64_t* |
NULL |
Number of prompt (input) tokens consumed. |
completion_tokens |
uint64_t* |
NULL |
Number of completion (output) tokens generated. |
model |
const char* |
NULL |
The model identifier used for extraction. |
chunks_processed |
uintptr_t |
— | Number of content chunks sent to the LLM. |
CBERGFaviconInfo
Section titled “CBERGFaviconInfo”C representation: CBERGFaviconInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGFaviconInfo does not appear anywhere in the generated header.
Information about a favicon or icon link.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The icon URL. |
rel |
const char* |
— | The rel attribute (e.g., “icon”, “apple-touch-icon”). |
sizes |
const char* |
NULL |
The sizes attribute, if present. |
mime_type |
const char* |
NULL |
The MIME type, if present. |
CBERGFeedInfo
Section titled “CBERGFeedInfo”C representation: CBERGFeedInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGFeedInfo does not appear anywhere in the generated header.
Information about a feed link found on a page.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The feed URL. |
title |
const char* |
NULL |
The feed title, if present. |
feed_type |
CBERGAlefHandle |
CBERG_RSS |
The type of feed. |
CBERGHeadingInfo
Section titled “CBERGHeadingInfo”C representation: CBERGHeadingInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGHeadingInfo does not appear anywhere in the generated header.
A heading element extracted from the page.
| Field | Type | Default | Description |
|---|---|---|---|
level |
uint8_t |
— | The heading level (1-6). |
text |
const char* |
— | The heading text content. |
CBERGHreflangEntry
Section titled “CBERGHreflangEntry”C representation: CBERGHreflangEntry is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGHreflangEntry does not appear anywhere in the generated header.
An hreflang alternate link entry.
| Field | Type | Default | Description |
|---|---|---|---|
lang |
const char* |
— | The language code (e.g., “en”, “fr”, “x-default”). |
url |
const char* |
— | The URL for this language variant. |
CBERGImageInfo
Section titled “CBERGImageInfo”C representation: CBERGImageInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGImageInfo does not appear anywhere in the generated header.
Information about an image found on a page.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The image URL. |
alt |
const char* |
NULL |
The alt text, if present. |
width |
uint32_t* |
NULL |
The width attribute, if present and parseable. |
height |
uint32_t* |
NULL |
The height attribute, if present and parseable. |
source |
CBERGAlefHandle |
CBERG_IMG |
The source of the image reference. |
CBERGInteractionResult
Section titled “CBERGInteractionResult”C representation: CBERGInteractionResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGInteractionResult does not appear anywhere in the generated header.
Result of executing a sequence of page interaction actions.
| Field | Type | Default | Description |
|---|---|---|---|
action_results |
const char* |
NULL |
Results from each executed action. |
final_html |
const char* |
— | Final page HTML after all actions completed. |
final_url |
const char* |
— | Final page URL (may have changed due to navigation). |
screenshot_base64 |
const char* |
NULL |
Base64-encoded PNG screenshot taken after all actions. Populated only when a PageAction.Screenshot action actually ran, so callers that never request a screenshot do not pay the encoding cost. |
CBERGJsonLdEntry
Section titled “CBERGJsonLdEntry”C representation: CBERGJsonLdEntry is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGJsonLdEntry does not appear anywhere in the generated header.
A JSON-LD structured data entry found on a page.
| Field | Type | Default | Description |
|---|---|---|---|
schema_type |
const char* |
— | The @type value from the JSON-LD object. |
name |
const char* |
NULL |
The name value, if present. |
raw |
const char* |
— | The raw JSON-LD string. |
CBERGLinkInfo
Section titled “CBERGLinkInfo”C representation: CBERGLinkInfo is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGLinkInfo does not appear anywhere in the generated header.
Information about a link found on a page.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The resolved URL of the link. |
text |
const char* |
— | The visible text of the link. |
link_type |
CBERGAlefHandle |
CBERG_INTERNAL |
The classification of the link. |
rel |
const char* |
NULL |
The rel attribute value, if present. |
nofollow |
int32_t |
— | Whether the link has rel="nofollow". |
CBERGMapResult
Section titled “CBERGMapResult”C representation: CBERGMapResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGMapResult does not appear anywhere in the generated header.
The result of a map operation, containing discovered URLs.
| Field | Type | Default | Description |
|---|---|---|---|
urls |
const char* |
NULL |
The list of discovered URLs. |
CBERGMarkdownResult
Section titled “CBERGMarkdownResult”C representation: CBERGMarkdownResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGMarkdownResult does not appear anywhere in the generated header.
Rich markdown conversion result from HTML processing.
| Field | Type | Default | Description |
|---|---|---|---|
content |
const char* |
— | Converted markdown text. |
document_structure |
const char* |
NULL |
Structured document tree with semantic nodes. |
tables |
const char* |
NULL |
Extracted tables with structured cell data. |
warnings |
const char* |
NULL |
Non-fatal processing warnings. |
citations |
int32_t |
— | Whether citation conversion was applied and produced at least one reference. true when the markdown contained inline links that were converted to numbered citation references. The converted content (with [N] markers) is available in content; the full reference list is accessible via generate_citations if needed separately. |
fit_content |
const char* |
NULL |
Content-filtered markdown optimized for LLM consumption. |
CBERGPageMetadata
Section titled “CBERGPageMetadata”C representation: CBERGPageMetadata is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGPageMetadata does not appear anywhere in the generated header.
Metadata extracted from an HTML page’s <meta> tags and <title> element.
| Field | Type | Default | Description |
|---|---|---|---|
title |
const char* |
NULL |
The page title from the <title> element. |
description |
const char* |
NULL |
The meta description. |
canonical_url |
const char* |
NULL |
The canonical URL from <link rel="canonical">. |
keywords |
const char* |
NULL |
Keywords from <meta name="keywords">. |
author |
const char* |
NULL |
Author from <meta name="author">. |
viewport |
const char* |
NULL |
Viewport content from <meta name="viewport">. |
theme_color |
const char* |
NULL |
Theme color from <meta name="theme-color">. |
generator |
const char* |
NULL |
Generator from <meta name="generator">. |
robots |
const char* |
NULL |
Robots content from <meta name="robots">. |
html_lang |
const char* |
NULL |
The lang attribute from the <html> element. |
html_dir |
const char* |
NULL |
The dir attribute from the <html> element. |
og_title |
const char* |
NULL |
Open Graph title. |
og_type |
const char* |
NULL |
Open Graph type. |
og_image |
const char* |
NULL |
Open Graph image URL. |
og_description |
const char* |
NULL |
Open Graph description. |
og_url |
const char* |
NULL |
Open Graph URL. |
og_site_name |
const char* |
NULL |
Open Graph site name. |
og_locale |
const char* |
NULL |
Open Graph locale. |
og_video |
const char* |
NULL |
Open Graph video URL. |
og_audio |
const char* |
NULL |
Open Graph audio URL. |
og_locale_alternates |
const char* |
NULL |
Open Graph locale alternates. |
twitter_card |
const char* |
NULL |
Twitter card type. |
twitter_title |
const char* |
NULL |
Twitter title. |
twitter_description |
const char* |
NULL |
Twitter description. |
twitter_image |
const char* |
NULL |
Twitter image URL. |
twitter_site |
const char* |
NULL |
Twitter site handle. |
twitter_creator |
const char* |
NULL |
Twitter creator handle. |
dc_title |
const char* |
NULL |
Dublin Core title. |
dc_creator |
const char* |
NULL |
Dublin Core creator. |
dc_subject |
const char* |
NULL |
Dublin Core subject. |
dc_description |
const char* |
NULL |
Dublin Core description. |
dc_publisher |
const char* |
NULL |
Dublin Core publisher. |
dc_date |
const char* |
NULL |
Dublin Core date. |
dc_type |
const char* |
NULL |
Dublin Core type. |
dc_format |
const char* |
NULL |
Dublin Core format. |
dc_identifier |
const char* |
NULL |
Dublin Core identifier. |
dc_language |
const char* |
NULL |
Dublin Core language. |
dc_rights |
const char* |
NULL |
Dublin Core rights. |
article |
CBERGAlefHandle |
NULL |
Article metadata from article:* Open Graph tags. |
hreflangs |
const char* |
NULL |
Hreflang alternate links. |
favicons |
const char* |
NULL |
Favicon and icon links. |
headings |
const char* |
NULL |
Heading elements (h1-h6). |
word_count |
uintptr_t* |
NULL |
Computed word count of the page body text. |
CBERGProxyConfig
Section titled “CBERGProxyConfig”C representation: CBERGProxyConfig is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGProxyConfig does not appear anywhere in the generated header.
Proxy configuration for HTTP requests.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | Proxy URL (e.g. “http://proxy:8080”, “socks5://proxy:1080”). |
username |
const char* |
NULL |
Optional username for proxy authentication. |
password |
const char* |
NULL |
Optional password for proxy authentication. |
CBERGResponseMeta
Section titled “CBERGResponseMeta”C representation: CBERGResponseMeta is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGResponseMeta does not appear anywhere in the generated header.
Response metadata extracted from HTTP headers.
| Field | Type | Default | Description |
|---|---|---|---|
etag |
const char* |
NULL |
The ETag header value. |
last_modified |
const char* |
NULL |
The Last-Modified header value. |
cache_control |
const char* |
NULL |
The Cache-Control header value. |
server |
const char* |
NULL |
The Server header value. |
x_powered_by |
const char* |
NULL |
The X-Powered-By header value. |
content_language |
const char* |
NULL |
The Content-Language header value. |
content_encoding |
const char* |
NULL |
The Content-Encoding header value. |
CBERGScrapeResult
Section titled “CBERGScrapeResult”C representation: CBERGScrapeResult is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGScrapeResult does not appear anywhere in the generated header.
The result of a single-page scrape operation.
| Field | Type | Default | Description |
|---|---|---|---|
status_code |
uint16_t |
— | The HTTP status code of the response. |
final_url |
const char* |
— | The final URL after following all redirects. |
content_type |
const char* |
— | The Content-Type header value. |
html |
const char* |
— | The HTML body of the response. |
body_size |
uintptr_t |
— | The size of the response body in bytes. |
metadata |
CBERGAlefHandle |
— | Extracted metadata from the page. |
links |
const char* |
NULL |
Links found on the page. |
images |
const char* |
NULL |
Images found on the page. |
feeds |
const char* |
NULL |
Feed links found on the page. |
json_ld |
const char* |
NULL |
JSON-LD entries found on the page. |
is_allowed |
int32_t |
— | Whether the URL is allowed by robots.txt. |
crawl_delay |
uint64_t* |
NULL |
The crawl delay from robots.txt, in seconds. |
noindex_detected |
int32_t |
— | Whether a noindex directive was detected. |
nofollow_detected |
int32_t |
— | Whether a nofollow directive was detected. |
x_robots_tag |
const char* |
NULL |
The X-Robots-Tag header value, if present. |
is_pdf |
int32_t |
— | Whether the content is a PDF. |
was_skipped |
int32_t |
— | Whether the page was skipped (binary or PDF content). |
detected_charset |
const char* |
NULL |
The detected character set encoding. |
auth_header_sent |
int32_t |
— | Whether an authentication header was sent with the request. |
response_meta |
CBERGAlefHandle |
NULL |
Response metadata extracted from HTTP headers. |
assets |
const char* |
NULL |
Downloaded assets from the page. |
js_render_hint |
int32_t |
— | Whether the page content suggests JavaScript rendering is needed. |
browser_used |
int32_t |
— | Whether the browser fallback was used to fetch this page. |
markdown |
CBERGAlefHandle |
NULL |
Markdown conversion of the page content. |
extracted_data |
const char* |
NULL |
Structured data extracted by LLM. Populated when extraction is configured. |
extraction_meta |
CBERGAlefHandle |
NULL |
Metadata about the LLM extraction pass (cost, tokens, model). |
screenshot_base64 |
const char* |
NULL |
Base64-encoded PNG screenshot of the page. Populated only when CrawlConfig.capture_screenshot was enabled for this request, so callers that never requested a screenshot do not pay the encoding cost. |
downloaded_document |
CBERGAlefHandle |
NULL |
Downloaded non-HTML document (PDF, DOCX, image, code, etc.). |
browser |
CBERGAlefHandle |
NULL |
Browser-specific extras (eval result, network events, cookies). Only populated when BrowserBackend.Native was used for this request. |
CBERGSitemapUrl
Section titled “CBERGSitemapUrl”C representation: CBERGSitemapUrl is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGSitemapUrl does not appear anywhere in the generated header.
A URL entry from a sitemap.
| Field | Type | Default | Description |
|---|---|---|---|
url |
const char* |
— | The URL. |
lastmod |
const char* |
NULL |
The last modification date, if present. |
changefreq |
const char* |
NULL |
The change frequency, if present. |
priority |
const char* |
NULL |
The priority, if present. |
CBERGSsrfPolicy
Section titled “CBERGSsrfPolicy”C representation: CBERGSsrfPolicy is a documentation-only name for this type. The C ABI hands you a scalar CBERGAlefHandle handle – the literal string CBERGSsrfPolicy does not appear anywhere in the generated header.
SSRF policy configuration.
| Field | Type | Default | Description |
|---|---|---|---|
deny_private |
int32_t |
true |
If true, reject URLs that resolve to private/metadata IP ranges. |
allowlist |
const char* |
NULL |
Hostnames and IP ranges permitted regardless of deny_private. The allowlist is an override of deny_private, not an intersection with it. Precedence, in order: 1. deny_private == false permits everything; the allowlist is not consulted. 2. A hostname matching an Exact or Suffix entry is permitted immediately, before DNS resolution — so the deny-list is never applied to it. This trusts the host string: a name that resolves into private space is still permitted. 3. A literal or resolved IP inside a Cidr entry is permitted even though it is in the default deny-list. 4. Otherwise the default deny-list decides. An empty allowlist therefore denies nothing by itself — it simply leaves deny_private and the deny-list in sole control. |
max_redirects |
uint8_t |
5 |
Maximum number of HTTP redirects to follow during validation. |
scheme_allowlist |
const char* |
["http", "https"] |
Allowed URI schemes. Default: ["http", "https"]. Only http and https are supported. An empty list denies every URL. |
CBERGBrowserMode
Section titled “CBERGBrowserMode”When to use the headless browser fallback.
| Value | Description |
|---|---|
CBERG_AUTO |
Automatically detect when JS rendering is needed and fall back to browser. |
CBERG_ALWAYS |
Always use the browser for every request. |
CBERG_NEVER |
Never use the browser fallback. |
CBERG_STEALTH |
Always use the browser with all stealth surfaces enabled. Behaves like Always for escalation purposes (every request is routed through the browser tier), but additionally enables: - browser JavaScript stealth patches - native-backend TLS fingerprint spoofing - stealth-aware default user-agent when no explicit UA is set - 1920×1080 viewport override Use this instead of setting the now-removed BrowserConfig.stealth boolean field. |
CBERGBrowserWait
Section titled “CBERGBrowserWait”Wait strategy for browser page rendering.
| Value | Description |
|---|---|
CBERG_NETWORK_IDLE |
Wait until network activity is idle. |
CBERG_SELECTOR |
Wait for a specific CSS selector to appear in the DOM. |
CBERG_FIXED |
Wait for a fixed duration after navigation. |
CBERGBrowserBackend
Section titled “CBERGBrowserBackend”Browser backend used for JavaScript rendering.
| Value | Description |
|---|---|
CBERG_CHROMIUMOXIDE |
Existing Chromium/CDP backend powered by chromiumoxide. |
CBERG_NATIVE |
Crawlberg-owned native browser backend derived from Obscura. |
CBERGDocumentContentEncoding
Section titled “CBERGDocumentContentEncoding”Opt-in encoding applied to a downloaded document’s bytes for callers who need the content available in a serializable field rather than reading it from disk.
NULL (the CrawlConfig.document_content_encoding default) produces neither — unlike
screenshots, base64-encoding a document by default would duplicate an already
up-to-document_max_size buffer (50 MB default) in memory per document.
| Value | Description |
|---|---|
CBERG_BASE64 |
Populate DownloadedDocument.content_base64 with a base64-encoded copy. |
CBERGCrawlStrategyKind
Section titled “CBERGCrawlStrategyKind”Traversal order for a crawl.
Selects both the queue discipline and the selection strategy, because global order is a property of the frontier: the engine hands its bounded selection window to the strategy, so a strategy alone can only reorder URLs that have already been dequeued.
| Value | Description |
|---|---|
CBERG_BFS |
Breadth-first: a FIFO frontier visits every URL at one depth before the next. |
CBERG_DFS |
Depth-first: a LIFO frontier descends into a page’s children before its siblings. |
CBERG_BEST_FIRST |
Highest-priority-first within the selection window, scored by CrawlStrategy.score_url. |
CBERG_ADAPTIVE |
Like BestFirst, but stops once newly crawled pages stop contributing new terms. |
CBERGContentFilterKind
Section titled “CBERGContentFilterKind”Content filter applied to each crawled page before it reaches the result.
| Value | Description |
|---|---|
CBERG_BM25 |
Keep only pages scoring at or above bm25_threshold for bm25_query. |
CBERGAuthConfig
Section titled “CBERGAuthConfig”Authentication configuration.
| Value | Description |
|---|---|
CBERG_BASIC |
HTTP Basic authentication. — Fields: username: const char*, password: const char* |
CBERG_BEARER |
Bearer token authentication. — Fields: token: const char* |
CBERG_HEADER |
Custom authentication header. — Fields: name: const char*, value: const char* |
CBERGLinkType
Section titled “CBERGLinkType”The classification of a link.
| Value | Description |
|---|---|
CBERG_INTERNAL |
A link to the same domain. |
CBERG_EXTERNAL |
A link to a different domain. |
CBERG_ANCHOR |
A fragment-only link (e.g., #section). |
CBERG_DOCUMENT |
A link to a downloadable document (PDF, DOC, etc.). |
CBERGImageSource
Section titled “CBERGImageSource”The source of an image reference.
| Value | Description |
|---|---|
CBERG_IMG |
An <img> tag. |
CBERG_PICTURE_SOURCE |
A <source> tag inside <picture>. |
CBERG_OG_IMAGE |
An og:image meta tag. |
CBERG_TWITTER_IMAGE |
A twitter:image meta tag. |
CBERGFeedType
Section titled “CBERGFeedType”The type of a feed (RSS, Atom, or JSON Feed).
| Value | Description |
|---|---|
CBERG_RSS |
RSS feed. |
CBERG_ATOM |
Atom feed. |
CBERG_JSON_FEED |
JSON Feed. |
CBERGAssetCategory
Section titled “CBERGAssetCategory”The category of a downloaded asset.
| Value | Description |
|---|---|
CBERG_DOCUMENT |
A document file (PDF, DOC, etc.). |
CBERG_IMAGE |
An image file. |
CBERG_AUDIO |
An audio file. |
CBERG_VIDEO |
A video file. |
CBERG_FONT |
A font file. |
CBERG_STYLESHEET |
A CSS stylesheet. |
CBERG_SCRIPT |
A JavaScript file. |
CBERG_ARCHIVE |
An archive file (ZIP, TAR, etc.). |
CBERG_DATA |
A data file (JSON, XML, CSV, etc.). |
CBERG_OTHER |
An unrecognized asset type. |
CBERGCrawlEvent
Section titled “CBERGCrawlEvent”An event emitted during a streaming crawl operation.
Not available on wasm32 targets — streaming requires native concurrency
primitives (tokio channels, JoinSet) that are not supported on wasm32.
Delivered to bindings through each target’s native streaming idiom.
| Value | Description |
|---|---|
CBERG_PAGE |
A single page has been crawled. — Fields: result: CBERGAlefHandle |
CBERG_ERROR |
An error occurred while crawling a URL. — Fields: url: const char*, error: const char* |
CBERG_COMPLETE |
The crawl has completed. — Fields: pages_crawled: uintptr_t |
CBERGPageAction
Section titled “CBERGPageAction”A single page interaction action.
Actions are serialized with a type tag using camelCase naming,
except ExecuteJs which is explicitly renamed to "executeJs".
| Value | Description |
|---|---|
CBERG_CLICK |
Click on an element matching the given CSS selector. — Fields: selector: const char* |
CBERG_TYPE_TEXT |
Type text into an element matching the given CSS selector. — Fields: selector: const char*, text: const char* |
CBERG_PRESS |
Press a keyboard key (e.g. “Enter”, “Tab”, “Escape”). — Fields: key: const char* |
CBERG_SCROLL |
Scroll the page or a specific element. — Fields: direction: CBERGAlefHandle, selector: const char*, amount: int64_t |
CBERG_WAIT |
Wait for a duration or for an element to appear. — Fields: milliseconds: int64_t, selector: const char* |
CBERG_SCREENSHOT |
Take a screenshot of the current page. — Fields: full_page: int32_t |
CBERG_EXECUTE_JS |
Execute arbitrary JavaScript in the page context. Safety: The script runs with full page privileges in the browser context. Only execute scripts from trusted sources. — Fields: script: const char* |
CBERG_SCRAPE |
Scrape the current page HTML. |
CBERGScrollDirection
Section titled “CBERGScrollDirection”Direction for a scroll action.
| Value | Description |
|---|---|
CBERG_UP |
Scroll upward. |
CBERG_DOWN |
Scroll downward. |
CBERGHostMatcher
Section titled “CBERGHostMatcher”Hostname/IP allowlist matcher for SSRF policy.
Serializes as an internally-tagged object so each variant is distinguishable on the wire and round-trips losslessly:
{"type": "exact", "value": "api.example.com"}{"type": "suffix", "value": ".example.com"}{"type": "cidr", "value": "10.0.0.0/8"}A bare JSON string is still accepted on deserialization and resolves to Exact,
preserving configs written against the previous untagged representation.
Exact: HostMatcher.Exact
| Value | Description |
|---|---|
CBERG_EXACT |
Exact hostname match (case-insensitive). — Fields: value: const char* |
CBERG_SUFFIX |
Suffix match: “.xberg.io” matches “api.xberg.io” and “xberg.io”. — Fields: value: const char* |
CBERG_CIDR |
CIDR match: “10.0.0.0/8” matches IP addresses in that range. — Fields: value: const char* |
Errors
Section titled “Errors”CBERGCrawlError
Section titled “CBERGCrawlError”Errors that can occur during crawling, scraping, or mapping operations.
| Variant | Description |
|---|---|
CBERG_NOT_FOUND |
The requested page was not found (HTTP 404). |
CBERG_UNAUTHORIZED |
The request was unauthorized (HTTP 401). |
CBERG_FORBIDDEN |
The request was forbidden (HTTP 403). |
CBERG_WAF_BLOCKED |
The request was blocked by a WAF or bot protection (HTTP 403 with WAF indicators). vendor is the lowercase identifier of the detected WAF (e.g. “cloudflare”, “datadome”). When the engine cannot identify the vendor, it uses “unknown”. message is the freeform description for logs and human readers. The stable error tag remains forbidden: waf/blocked: MESSAGE so existing log-grep patterns and cross-language bindings continue to work; vendor is surfaced separately for structured consumers. |
CBERG_TIMEOUT |
The request timed out. |
CBERG_RATE_LIMITED |
The request was rate-limited (HTTP 429). |
CBERG_SERVER_ERROR |
A server error occurred (HTTP 5xx). |
CBERG_BAD_GATEWAY |
A bad gateway error occurred (HTTP 502). |
CBERG_GONE |
The resource is permanently gone (HTTP 410). |
CBERG_CONNECTION |
A connection error occurred. |
CBERG_DNS |
A DNS resolution error occurred. |
CBERG_SSL |
An SSL/TLS error occurred. |
CBERG_DATA_LOSS |
Data was lost or truncated during transfer. |
CBERG_BROWSER_ERROR |
The browser failed to launch, connect, or navigate. |
CBERG_BROWSER_TIMEOUT |
The browser page load or rendering timed out. |
CBERG_INVALID_CONFIG |
The provided configuration is invalid. |
CBERG_UNSUPPORTED |
The requested capability is not supported by the active backend or build. |
CBERG_SSRF_POLICY_VIOLATION |
A URL was rejected by SSRF policy (private IP, metadata, disallowed scheme, etc). |
CBERG_OTHER |
An unclassified error occurred. |
CBERGSsrfError
Section titled “CBERGSsrfError”SSRF validation error.
| Variant | Description |
|---|---|
CBERG_DENIED_BY_POLICY |
URL denied by SSRF policy: private IP, metadata IP, etc. |
CBERG_NOT_ON_ALLOWLIST |
Host not on allowlist when an allowlist is configured. |
CBERG_INVALID_CIDR |
Allowlist entry is not a parseable CIDR block. |
CBERG_DNS_RESOLUTION_FAILED |
DNS resolution failed for hostname. |
CBERG_INVALID_URL |
Invalid URL format. |
CBERG_DISALLOWED_SCHEME |
URL scheme not in allowlist (e.g., ftp:// when only http/https allowed). |
CBERG_TOO_MANY_REDIRECTS |
Too many HTTP redirects encountered during validation. |