Crawling
Start a new crawling session
Initiates a crawling session with the specified URL and optional filters for paths.
POST
/
v4
/
crawl
Authorizations
x-api-key
string
headerrequiredHeaders
x-api-key
string
requiredAPI key for authentication
Body
application/json
url
string
requiredURL to start crawling
pathsFilterIn
string[]
List of path patterns to include in the crawl
pathsFilterOut
string[]
List of path patterns to exclude from the crawl
timeout
integer
Maximum time for crawling in milliseconds
maxDepth
integer
Maximum depth to traverse during the crawl
maxPages
integer
Maximum number of pages to crawl
strictDomain
boolean
Set this flag to true when you want the crawler to stay on the same exact domain, not going to subdomains
proxyCountry
string
country flag code to assign to proxy configuration to navigate behind the proxy
Response
200 - application/json
message
string
sessionId
string
Unique ID for the started crawl session
error
string | null
Null if no error occurred