Installation

Installation

CLI

The fastest way to get started. No Rust toolchain needed if you use Homebrew.

Homebrew (macOS / Linux)

brew trust xberg-io/tap
brew install xberg-io/tap/crawlberg

Scoop (Windows)

scoop bucket add xberg https://github.com/xberg-io/scoop-bucket
scoop install crawlberg

Cargo

cargo install crawlberg-cli

Or grab the prebuilt binary from GitHub Releases via cargo-binstall:

cargo binstall crawlberg-cli

With optional features:

cargo install crawlberg-cli --features "api,mcp"

npm

npm install -g @xberg-io/crawlberg-cli

Installs a thin proxy that downloads and runs the native crawlberg binary from GitHub releases. Requires Node.js 18+; no Rust toolchain needed.

Verify:

crawlberg --version

---

Language Bindings

Rust

Add to your Cargo.toml:

Cargo.toml

[dependencies]
crawlberg = "0.3"
tokio = { version = "1", features = ["full"] }

Optional feature flags:

Feature	Description
native-runtime	Native OS runtime marker; enabled by default outside wasm32
browser	Chromiumoxide browser backend
browser-native	In-process native browser backend
ai	LLM extraction via liter-llm
otel	One-call OpenTelemetry/OTLP setup
interact	Compatibility alias for browser-backed page interaction
mcp	Model Context Protocol server
api	REST API server via Axum
mcp-http	MCP over HTTP transport
warc	WARC archive output

Enable features as needed:

Cargo.toml

[dependencies]
crawlberg = { version = "0.3", features = ["browser", "ai", "mcp"] }

Rust version

Crawlberg requires Rust 1.91+ (edition 2024).

Python

Requires Python 3.10+. Install from PyPI:

pip install crawlberg

Or with uv:

uv add crawlberg

The package ships pre-built wheels with the Rust core compiled via maturin. No Rust toolchain needed.

No musl wheels

Wheels are built for glibc Linux (manylinux), macOS, and Windows. There is no musllinux wheel, so on Alpine pip falls back to a source build. See platform support.

TypeScript

Install the @xberg-io/crawlberg package:

npm install @xberg-io/crawlberg

Or with pnpm:

pnpm add @xberg-io/crawlberg

The package includes pre-built native binaries via NAPI-RS and ships with TypeScript type definitions (.d.ts).

Not available on Alpine

Native binaries are built for glibc Linux, macOS, and Windows — not musl. On Alpine the install still exits 0, because npm silently skips the musl optional dependency, but no native binary is installed and require() fails at runtime. See platform support.

Ruby

Requires Ruby 3.2+. Add to your Gemfile:

Gemfile

gem "crawlberg", "~> 0.3"

Then:

bundle install

Or install directly:

gem install crawlberg

The gem includes a native extension built with Magnus and rb_sys.

Go

Requires Go 1.21+. The Go bindings use cgo with the C FFI layer:

go get github.com/xberg-io/crawlberg/packages/go

Build requirement

A C compiler is required. The crawlberg shared library must be available at link time.

Java

Requires Java 25+ (Panama FFM). Add the Maven dependency:

pom.xml

<dependency>
    <groupId>io.xberg.crawlberg</groupId>
    <artifactId>crawlberg</artifactId>
    <version>0.3.0</version>
</dependency>

The Java bindings use the Panama Foreign Function & Memory API to call the C FFI layer.

Kotlin (Android)

Add the Android AAR dependency:

build.gradle.kts

dependencies {
    implementation("io.xberg.crawlberg:crawlberg-android:0.3.0")
}

The Kotlin package targets Android and bundles JNI shared libraries. Kotlin/JVM server applications should use the Java package.

C#

Requires .NET 10+. Add the NuGet package:

dotnet add package Crawlberg

Or in your .csproj:

Crawlberg.csproj

<PackageReference Include="Crawlberg" Version="0.3.0" />

The C# bindings use P/Invoke to call the C FFI layer.

PHP

Requires PHP 8.2+. Install the extension via Composer:

composer require xberg-io/crawlberg

The PHP bindings are built with ext-php-rs and load as a native PHP extension.

Verify the extension is loaded

php -m | grep crawlberg

Elixir

Requires Elixir 1.14+ with OTP 25+. Add to your mix.exs:

mix.exs

defp deps do
  [
    {:crawlberg, "~> 0.3"}
  ]
end

Then:

mix deps.get
mix compile

The Elixir bindings use Rustler NIFs.

WebAssembly

For browser or Node.js environments via wasm-bindgen:

npm install @xberg-io/crawlberg-wasm

import init from "@xberg-io/crawlberg-wasm";
await init();

Limitations

The WASM build runs without native browser backends, REST/MCP server starters, WARC file output, native streaming crawl wrappers, and other native-only I/O surfaces.

Dart

Requires Dart or Flutter with native FFI support:

dart pub add crawlberg

Swift

Add the Swift package from GitHub:

.package(url: "https://github.com/xberg-io/crawlberg", exact: "0.3.0")

Zig

Use the generated Zig package over the C FFI layer:

zig fetch --save https://github.com/xberg-io/crawlberg/archive/refs/tags/v0.3.0.tar.gz

C FFI

For languages not listed above, crawlberg provides a C-compatible FFI layer. Build the shared library:

cargo build --release -p crawlberg-ffi

The output is a shared library (libcrawlberg.so / libcrawlberg.dylib / crawlberg.dll) with C headers generated by cbindgen.

---

Docker

Pull the official image:

docker pull ghcr.io/xberg-io/crawlberg:latest

Run the CLI:

docker run --rm ghcr.io/xberg-io/crawlberg:latest scrape https://example.com

Run with a volume for WARC output:

docker run --rm -v $(pwd)/output:/output \
  ghcr.io/xberg-io/crawlberg:latest \
  crawl https://example.com --depth 2 --warc-output /output/archive.warc

---

Platform support

Every binding ships prebuilt artifacts for glibc Linux (x86_64 and aarch64), macOS, and Windows. On musl Linux — Alpine and anything else built on musl — coverage is deliberately partial, and the table below is the authoritative statement of it.

Distribution	glibc Linux	musl Linux
CLI (GitHub release, Homebrew)	✅	✅ x86_64, aarch64
Docker (ghcr.io/xberg-io/…)	✅	✅ Alpine image, static binary
Rust (crates.io)	✅	✅ builds from source
Ruby (RubyGems)	✅	✅ compiles at install
Java (Maven Central)	✅	✅ bundled in the jar
C# (NuGet)	✅	✅ bundled in the package
Elixir (Hex)	✅	✅ precompiled NIF
Node (npm)	✅	❌ not published
Python (PyPI)	✅	❌ not published
Go	✅	❌ not published
PHP	✅	❌ not published

Why musl is omitted for Node, Python, Go, and PHP

These four bindings load a native library that must be built specifically for musl. Producing and testing that second Linux artifact doubles the release matrix for each of them, and the platforms where Alpine is actually used — containers and CI images — are already served by the statically linked CLI and the Alpine Docker image. So the omission is a deliberate scope decision, not an oversight or a broken build, and it is not scheduled to change. If you need Crawlberg on Alpine, use the CLI or the Docker image, or run your application on a glibc base image such as node:22-bookworm-slim, python:3.13-slim, or debian:bookworm-slim.

Two consequences are worth knowing about, because neither announces itself:

• npm installs succeed on Alpine and then fail at runtime. @xberg-io/crawlberg declares @xberg-io/crawlberg-linux-x64-musl and @xberg-io/crawlberg-linux-arm64-musl as optional dependencies. Those package names are registered but hold only a 0.0.1 placeholder, so npm skips them without an error and installs no native binary at all. The failure surfaces later, when require() cannot find a native module. Do not read a clean npm install on Alpine as evidence that the package works there.
• pip finds no musl wheel and falls back to the sdist. PyPI carries manylinux wheels only, so pip install crawlberg on Alpine attempts a source build and needs a full Rust toolchain to have any chance of succeeding.

---

Where to go next

• Quick start — Scrape a page, run a crawl, and map a site in under five minutes. Covers the CLI and the Rust API side by side.
• Configuration guide — Every CrawlConfig field with its default and validation rules. Start here if you need to tune depth, concurrency, or content filtering.
• Features overview — What the engine can do: browser rendering, LLM extraction, REST API, MCP, WARC output, and more. Useful for figuring out which feature flags you need.