DeepCrawl API for turning websites into knowledge bases
Start a background crawl, track task status, and collect site content for RAG systems and agent memory workflows.
Endpoint
Collect content across a larger website in the background so a product can build a knowledge base without blocking the user.
Crawl docs, blogs, and help centers.
Track long jobs in the background.
Pages ready for RAG and search.
Useful for
Docs KBs / Search indexes / AI workflows
Asynchronous crawling for larger sites
DeepCrawl starts a task and returns a task ID, so your application can poll status while the crawl runs. Use sitemap mode for controlled crawling or all mode for broader link discovery.
Quickly Turn Websites into Documents
Simply enter a link to quickly crawl all linked content within a site and save it locally.
Markdown Output
Output in clean Markdown file format, directly usable for LLM knowledge base retrieval.
Asynchronous Processing
Tasks run in the background, and you can check the status at any time.
Recursive Link Following
Freely choose between crawling based on a sitemap or crawling all linked content within the site.
What is DeepCrawl?
DeepCrawl is an asynchronous full-site crawling API that collects content from every page on a website in the background. Unlike a single-page crawl API that processes one URL at a time, DeepCrawl discovers all pages on a site, fetches their content, and returns it as a structured dataset. This makes it ideal for building RAG knowledge bases from documentation sites, help centers, or blogs, where you need the entire site corpus rather than individual pages. Deep crawling is the process of recursively following links to map and extract content from a whole website, not just the entry page.
Typical workflow
Start a background crawl, track task status, and collect site content for RAG systems and agent memory workflows.
Start a DeepCrawl task with a URL and discovery mode.
Store the returned task ID and poll the status endpoint.
Process completed results into a knowledge base or document store.
Best for
Quickly turn websites into documents with our powerful deep crawling capabilities
Building RAG knowledge bases from docs, help centers, and blogs.
Refreshing internal search indexes from a full website.
Packaging large site content without blocking a user request.
Building your own crawler and hitting anti-bot blocks?
If you run your own scraper and keep getting rate-limited or blocked by bot detection, route requests through our rotating residential proxies for fresh IPs and geo-targeting.
FAQ
Why is DeepCrawl asynchronous?
Full-site crawling can take longer than a normal request. DeepCrawl returns a task ID so the job can run in the background while your app checks status.
How many credits does DeepCrawl API use?
Starting a DeepCrawl task costs 20 credits.
What is DeepCrawl?
DeepCrawl is an asynchronous full-site crawling API that discovers all pages on a website, fetches their content, and returns it as a structured dataset. It is designed for building RAG knowledge bases, refreshing search indexes, and packaging large site content without blocking user requests.
What is deep crawling?
Deep crawling is the process of recursively following links to map and extract content from an entire website, not just the entry page. DeepCrawl automates this by starting a background task that discovers pages via sitemap or link traversal, fetches each page, and returns clean content for your knowledge base or RAG pipeline.
How is DeepCrawl different from Crawl API?
Crawl API processes one URL or a small batch of URLs synchronously and returns clean content immediately. DeepCrawl crawls an entire website asynchronously in the background, discovering all pages and returning a full site corpus. Use Crawl for individual pages; use DeepCrawl when you need the whole site.