DeepCrawl API for turning websites into knowledge bases
Start a background crawl, track task status, and collect site content for RAG systems and agent memory workflows.
Collect content across a larger website in the background so a product can build a knowledge base without blocking the user.
Crawl docs, blogs, and help centers.
Track long jobs in the background.
Pages ready for RAG and search.
Useful for
Docs KBs / Search indexes / AI workflows
Asynchronous crawling for larger sites
DeepCrawl starts a task and returns a task ID, so your application can poll status while the crawl runs. Use sitemap mode for controlled crawling or all mode for broader link discovery.
Quickly Turn Websites into Documents
Simply enter a link to quickly crawl all linked content within a site and save it locally.
Markdown Output
Output in clean Markdown file format, directly usable for LLM knowledge base retrieval.
Asynchronous Processing
Tasks run in the background, and you can check the status at any time.
Recursive Link Following
Freely choose between crawling based on a sitemap or crawling all linked content within the site.
Asynchronous crawling for larger sites
Start a background crawl, track task status, and collect site content for RAG systems and agent memory workflows.
Start a DeepCrawl task with a URL and discovery mode.
Store the returned task ID and poll the status endpoint.
Process completed results into a knowledge base or document store.
Best for
Quickly turn websites into documents with our powerful deep crawling capabilities
Building RAG knowledge bases from docs, help centers, and blogs.
Refreshing internal search indexes from a full website.
Packaging large site content without blocking a user request.
Deep Crawl API
Quickly turn websites into documents with our powerful deep crawling capabilities
FAQ
Why is DeepCrawl asynchronous?
Full-site crawling can take longer than a normal request. DeepCrawl returns a task ID so the job can run in the background while your app checks status.
How many credits does DeepCrawl API use?
Starting a DeepCrawl task costs 20 credits.