Crawl4AI Python tutorial: from pip install to LLM extraction

1. Install Crawl4AI

python -m venv .venv && source .venv/bin/activate
pip install -U crawl4ai
crawl4ai-setup      # downloads the Playwright browser
crawl4ai-doctor     # optional health check

Crawl4AI needs a recent Python 3 version. crawl4ai-setup installs the Chromium browser that Playwright uses; without it the first crawl fails with a browser error.

2. First crawl with AsyncWebCrawler

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://docs.crawl4ai.com/")
        print(result.markdown[:500])

asyncio.run(main())

arun() returns a result object. The most useful fields are success, markdown, html, links and extracted_content.

3. Clean “fit” Markdown with a content filter

Raw Markdown contains menus and footers. A content filter keeps the main text – exactly what you want for an LLM:

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilter

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed")
    ),
)
# inside the crawler context:
result = await crawler.arun(url=URL, config=config)
print(result.markdown.fit_markdown)

The online tool above uses the same idea with “Main content only” and generates this script for your URL.

4. Crawl many URLs in parallel

urls = ["https://example.com/a", "https://example.com/b"]
results = await crawler.arun_many(urls=urls, config=config)
for r in results:
    print(r.url, r.success, len(r.markdown or ""))

5. Extract structured JSON with an LLM

import os
from pydantic import BaseModel
from crawl4ai import CrawlerRunConfig, LLMConfig, LLMExtractionStrategy

class Item(BaseModel):
    name: str
    price: str

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
    schema=Item.model_json_schema(),
    extraction_type="schema",
    instruction="Extract every product with its name and price.",
)
config = CrawlerRunConfig(extraction_strategy=strategy)
result = await crawler.arun(url=URL, config=config)
print(result.extracted_content)  # JSON string

For repeated pages with the same layout, JsonCssExtractionStrategy is cheaper: you define CSS selectors once (or let an LLM generate the schema one time) and extract without per-page LLM costs.

Official reference: docs.crawl4ai.com. Next: run Crawl4AI as an API with Docker.