crawlberg

Why crawlberg

Crawl a whole site or a single page

Follow links across an entire site or fetch one URL, with concurrent requests and live progress events. Tune how it walks the site with breadth-first, depth-first, best-first, or adaptive strategies.

Clean Markdown, every time

Every page comes back as ready-to-use Markdown — feed it straight to an LLM or your own pipeline. Generate a citation list when you need one.

Know what's on every page

Titles, links, images, feeds, and social-card data arrive already extracted and structured, so you skip the HTML parsing. Covers Open Graph, Twitter Card, Dublin Core, JSON-LD, and hreflang.

Handles JS-heavy and bot-protected sites

When a page needs JavaScript or trips a bot filter, crawlberg renders it in a real browser for you. It escalates to a Chromium (CDP) or native browser backend only when a page needs it.

Use it from your agent, API, or CLI

Drive crawlberg from an AI agent over MCP, a REST service, or the command line — no glue code required. The MCP and REST servers are gated behind cargo features.

Works in the language you already use

One Rust engine, identical results across 14 languages: Python, TypeScript, Go, Java, Kotlin Android, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, WebAssembly, and C FFI.

See all features →

Language support

Language	Package	Docs
Rust	cargo add crawlberg	API Reference
Python	pip install crawlberg	API Reference
TypeScript / Node	npm install @xberg-io/crawlberg	API Reference
WebAssembly	npm install @xberg-io/crawlberg-wasm	API Reference
Go	go get github.com/xberg-io/crawlberg/packages/go	API Reference
Java	Maven Central io.xberg.crawlberg:crawlberg	API Reference
Kotlin (Android)	Maven Central io.xberg.crawlberg:crawlberg-android	API Reference
C#	dotnet add package Crawlberg	API Reference
Ruby	gem install crawlberg	API Reference
PHP	composer require xberg-io/crawlberg	API Reference
Elixir	{:crawlberg, "~> 0.3.0"}	API Reference
Dart / Flutter	dart pub add crawlberg	API Reference
Swift	Swift Package Manager	API Reference
Zig	zig fetch --save from GitHub	API Reference
C (FFI)	Shared library + header	API Reference
CLI	cargo install crawlberg-cli	CLI Guide
Docker	ghcr.io/xberg-io/crawlberg	Docker Guide

Choosing between TypeScript packages

@xberg-io/crawlberg — Native NAPI-RS bindings. Use for Node.js servers and CLI tools. Full feature set including the browser fallback.

@xberg-io/crawlberg-wasm — Pure WebAssembly. Use for browsers, Cloudflare Workers, Deno, Bun, and serverless. No native browser backend, REST server, MCP server, or native streaming crawl wrappers.

Quick example

Rust

src/main.rs

use crawlberg::{CrawlConfig, ContentConfig, create_engine, crawl};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = CrawlConfig {
        max_depth: Some(2),
        max_pages: Some(50),
        content: ContentConfig::default(),
        ..Default::default()
    };
    let engine = create_engine(Some(config))?;

    let result = crawl(&engine, "https://example.com").await?;
    for page in &result.pages {
        let title = page.metadata.title.as_deref().unwrap_or("(no title)");
        println!("{} — {}", page.url, title);
    }
    Ok(())
}

Python

main.py

import asyncio
from crawlberg import CrawlConfig, create_engine, crawl

async def main():
    engine = create_engine(CrawlConfig(max_depth=2, max_pages=50))
    result = await crawl(engine, "https://example.com")
    for page in result.pages:
        print(f"{page.url} — {page.metadata.title or '(no title)'}")

asyncio.run(main())

TypeScript

index.ts

import { createEngine, crawl } from "@xberg-io/crawlberg";

const engine = createEngine({ maxDepth: 2, maxPages: 50 });
const result = await crawl(engine, "https://example.com");

for (const page of result.pages) {
  console.log(`${page.url} — ${page.metadata.title ?? "(no title)"}`);
}

Part of Xberg.io

XbergThe open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence.

Xberg ProA complete self-hosted content-intelligence backend in a single container. Commercial.

Xberg EnterpriseThe distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.

crawlbergWeb crawling and scraping with HTML→Markdown and headless-Chrome fallback.

html-to-markdownFast, lossless HTML→Markdown engine.

liter-llmUniversal LLM API client with native bindings for 14 languages and 165 providers.

tree-sitter-language-packTree-sitter grammars and code-intelligence primitives.

alefThe polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

Explore the docs

Get StartedInstall crawlberg and run your first crawl in under five minutes.

GuidesCrawling, scraping, URL discovery, browser automation, WARC output, and deployment.

ConceptsPublic surface, data flow, the binding matrix, feature gates, and the extraction pipeline.

ReferencePer-language API docs, the configuration schema, type catalogue, and error matrix.

CLI & ServersThe crawlberg CLI, REST API server, and MCP server for AI agents.

FeaturesCrawl strategies, metadata extraction, browser backends, WARC, MCP, REST.

Getting help

• Bugs & feature requests — Open an issue on GitHub
• Community chat — Join the Discord
• Contributing — Read the contributor guide