Choosing an endpoint

Start with two questions:

1. Do you have a topic, a URL, or an entire site?
2. Do you need links, readable text, structured fields, or a list of pages?

Endpoint map

Topic or URL: search vs. crawl

Use /search when you need to discover relevant pages. Use /crawl when you already know which page to read.

When you begin with a topic but also need source text, set crawl_results on /search. Search1API will crawl the top results in the same request. See Improving search results for the request pattern.

Text or fields: crawl vs. extract

Use /crawl when a model or person will read the page as text: summarization, question answering, or RAG ingestion.

Use /extract when your application needs typed fields in a predictable shape. You provide the URL, an extraction prompt, and a JSON Schema; the endpoint returns data that follows that schema.

If you plan to crawl a page and immediately ask a model to convert it into fields, /extract combines those steps.

URL list or site content: sitemap vs. deepcrawl

Use /sitemap to discover which URLs a site contains. It returns links synchronously and is useful when your application will decide what to fetch next.

Use /deepcrawl when you need content from many pages. It runs asynchronously: the start request returns 202 Accepted with a taskId, and you poll GET /deepcrawl/status/{taskId} until the task finishes.

Web coverage or recent reporting: search vs. news

Use /search for general web discovery. Use /news when recency and news-specific sources are central to the task. Both endpoints support time_range, but they search different source sets.

A common agent workflow

For research and answer-generation agents:

1. Start with /search and a small crawl_results value to collect ranked sources and readable text.
2. Use /crawl only when you need to revisit a specific URL.
3. Use /extract when the output must follow a stable data schema.

Review Credits and limits before choosing crawl depth or running the workflow in a batch.