Mode Summary
| Mode | API Value | Best For |
|---|---|---|
| Single Page | single-page | Extract data from a single page |
| List | paginated-page | Navigate through lists with pagination |
| List + Details | page-and-detail | Navigate lists then open each item for details |
| All Pages | all-pages | Crawl all pages and extract matching entities |
| Custom AI Navigation | agentic-navigation | AI-driven navigation using natural language |
Single Page
- Stay on this page
- Extract data
List (Paginated Page)
- Navigate lists with pagination or scroll
- Extract each item
List + Details
- Navigate lists with pagination or scroll
- Click items
- Extract details
All Pages (Crawler)
The crawler mode is currently available only via API. Support for UI and SDK is coming soon.
- Crawls all pages or up to
maxPages(if specified) - Discovers and follows links automatically
- Extracts matching entities from crawled pages
Crawler Parameters
| Parameter | Description | Default |
|---|---|---|
maxPages | Maximum pages to crawl (1-100,000). Crawling stops when reached. | 10,000 |
maxDepth | Maximum crawl depth from starting URL (1-200) | 50 |
pathsFilterIn | Regex patterns to include specific paths (e.g., ["/products/.*"]) | None |
pathsFilterOut | Regex patterns to exclude specific paths (e.g., ["/admin/.*"]) | None |
The starting URL must display the entity you want to extract.
All URLs must share the exact same hostname. For example,
https://example.com and https://example.com/products are valid, but mixing https://example.com with https://www.example.com or https://shop.example.com fails.Raw Data Mode
You can also crawl without a schema to retrieve raw page artifacts (HTML, Markdown, screenshots). This is useful for LLM ingestion, site archival, or content analysis.| Field | Type | Default | Description |
|---|---|---|---|
includeHtml | boolean | true | Include raw HTML content |
includeMarkdown | boolean | false | Include Markdown-converted content |
includeScreenshots | boolean | false | Include Base64-encoded screenshots |
includeJson | boolean | false | Include JSON extraction results |
Custom AI Navigation
Available for Enterprise customers.
- Write prompts that the AI will follow
- Automate browser interactions
- Extract data from files (PDFs, CSVs, etc.)
Slower and uses more resources. Only use when simpler modes won’t work.
Variables
Variables let you define reusable values for your prompts using@variableName syntax. This is useful for parameterized extractions like search terms, locations, or dates.
Learn more about Variables →
Secret Variables
For sensitive data like passwords and API keys, use the Secret variable type. Secrets are encrypted at rest and never exposed to LLMs. Learn more about Secrets → Learn more about Custom AI Navigation →Using Navigation Modes
- In UI
- In Code
When creating a workflow in the dashboard, select your navigation mode from the visual selector:
- Go to Create Workflow
- Enter your source URL
- Choose navigation mode from the options
- Configure additional settings