Skip to main content

​
Prerequisites

  • A Kadoa account with API key
  • SDK installed: npm install @kadoa/node-sdk or uv add kadoa-sdk
  • An existing workflow (create one first)

​
Scheduling Options

Configure when your workflow runs: 
const workflow = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/ecommerce/pagination"],
    name: "Scheduled Extraction",
    extraction: (builder) =>
      builder
        .entity("Product")
        .field("title", "Product name", "STRING", { example: "Sample" }),
  })
  .setInterval({
    schedules: ["0 9 * * MON-FRI", "0 18 * * MON-FRI"],
  })
  .create();

// Workflow runs automatically on schedule
console.log("Scheduled workflow:", workflow.workflowId);

workflow = (
    client.extract(
        ExtractOptions(
            urls=["https://sandbox.kadoa.com/ecommerce/pagination"],
            name="Scheduled Extraction",
            extraction=lambda builder: builder.entity("Product").field(
                "title", "Product name", "STRING", FieldOptions(example="Sample")
            ),
        )
    )
    .set_interval({"schedules": ["0 9 * * MON-FRI", "0 18 * * MON-FRI"]})
    .create()
)

# Workflow runs automatically on schedule
print("Scheduled workflow:", workflow.workflow_id)

curl -X POST https://api.kadoa.com/v4/workflows \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://sandbox.kadoa.com/ecommerce/pagination"],
    "name": "Scheduled Extraction",
    "interval": "CUSTOM",
    "schedules": ["0 9 * * MON-FRI", "0 18 * * MON-FRI"],
    "entity": "Product",
    "fields": [{"name": "title", "dataType": "STRING", "description": "Product name"}]
  }'

> "Create a workflow to extract products from sandbox.kadoa.com/ecommerce/pagination, scheduled at 9am and 6pm on weekdays"

​
Available Intervals

IntervalDescription
ONLY_ONCERun once
EVERY_10_MINUTESEvery 10 minutes
HALF_HOURLYEvery 30 minutes
HOURLYEvery hour
THREE_HOURLYEvery 3 hours
SIX_HOURLYEvery 6 hours
TWELVE_HOURLYEvery 12 hours
DAILYOnce per day
WEEKLYOnce per week
MONTHLYOnce per month
REAL_TIMEContinuous monitoring
CUSTOMUse cron expressions

​
Manual Execution

Run workflows on demand: 
const workflow = await client.workflow.get(workflowId);
console.log(`Current workflow state: ${workflow.displayState}`);

const result = await client.workflow.runWorkflow(workflowId, {
  limit: 10,
});
console.log(`Workflow scheduled with runId: ${result.jobId}`);

workflow = client.workflow.get(workflow_id)
print(f"Current workflow state: {workflow.display_state}")

result = client.workflow.run_workflow(
    workflow_id,
    input=RunWorkflowOptions(limit=10),
)
print(f"Workflow scheduled with runId: {result.job_id}")

curl -X POST https://api.kadoa.com/v4/workflows/YOUR_WORKFLOW_ID/run \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json"

> "Run my 'Scheduled Extraction' workflow with a limit of 10 records"

{
  "success": true,
  "jobId": "job_abc123"
}

​
Checking Workflow Status

Poll the workflow status to know when extraction is complete: 
const extraction = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/ecommerce/pagination"],
    name: "Paginated Extraction",
    userPrompt: "Extract all products, paginating through all pages",
    extraction: (builder) =>
      builder
        .entity("Product")
        .field("title", "Product name", "STRING", {
          example: "Sennheiser HD 6XX",
        })
        .field("price", "Product price", "MONEY"),
  })
  .create();

const result = await extraction.run({ limit: 10 });

// Fetch a single page with pagination info
const page = await result.fetchData({ page: 1, limit: 5 });
console.log("Page data:", page.data);
console.log("Pagination:", page.pagination);

// Or get all data at once
const allData = await result.fetchAllData({});
console.log("All data:", allData);

extraction = (
    client.extract(
        ExtractOptions(
            urls=["https://sandbox.kadoa.com/ecommerce/pagination"],
            name="Paginated Extraction",
            user_prompt="Extract all products, paginating through all pages",
            extraction=lambda builder: builder.entity("Product")
            .field(
                "title",
                "Product name",
                "STRING",
                FieldOptions(example="Sennheiser HD 6XX"),
            )
            .field("price", "Product price", "MONEY"),
        )
    )
    .create()
)

result = extraction.run(RunWorkflowOptions(limit=10))

# Fetch a single page with pagination info
page = result.fetch_data({"page": 1, "limit": 5})
print("Page data:", page.data)
print("Pagination:", page.pagination)

# Or get all data at once
all_data = result.fetch_all_data({})
print("All data:", all_data)

# 1. Run workflow
curl -X POST https://api.kadoa.com/v4/workflows/YOUR_WORKFLOW_ID/run \
  -H "x-api-key: YOUR_API_KEY"

# 2. Poll status until complete
curl -X GET https://api.kadoa.com/v4/workflows/YOUR_WORKFLOW_ID \
  -H "x-api-key: YOUR_API_KEY"

# 3. Once state is "COMPLETED", fetch data
curl -X GET https://api.kadoa.com/v4/workflows/YOUR_WORKFLOW_ID/data \
  -H "x-api-key: YOUR_API_KEY"

{
  "id": "507f1f77bcf86cd799439011",
  "name": "My Workflow",
  "state": "ACTIVE",
  "lastRun": {
    "id": "run-123",
    "state": "IN_PROGRESS",
    "startedAt": "2024-01-15T10:00:00Z",
    "completedAt": null
  }
}
Workflow States:
  • IN_PROGRESS - Extraction is running
  • COMPLETED - Data is ready to retrieve
  • FAILED - Extraction failed (check errors field)

​
Proxy Locations

Specify geographic location for extraction: 
const workflow = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/magic"],
    name: "Geo-located Extraction",
    extraction: (builder) =>
      builder
        .entity("Product")
        .field("title", "Title", "STRING", { example: "example" }),
  })
  .setLocation({
    type: "manual",
    isoCode: "US",
  })
  .create();

workflow = (
    client.extract(
        ExtractOptions(
            urls=["https://sandbox.kadoa.com/magic"],
            name="Geo-located Extraction",
            extraction=lambda builder: builder.entity("Product").field(
                "title", "Title", "STRING", FieldOptions(example="example")
            ),
        )
    )
    .set_location({"type": "manual", "isoCode": "US"})
    .create()
)

curl -X POST https://api.kadoa.com/v4/workflows \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://sandbox.kadoa.com/magic"],
    "location": {
      "type": "manual",
      "isoCode": "US"
    }
  }'
Available locations:
  • US - United States
  • GB - United Kingdom
  • DE - Germany
  • NL - Netherlands
  • CA - Canada
  • auto - Automatic selection

​
Bypass Preview Mode

Skip manual review and activate workflows immediately: 
const workflow = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/magic"],
    name: "Direct Activation",
    extraction: (builder) =>
      builder
        .entity("Product")
        .field("title", "Title", "STRING", { example: "example" }),
  })
  .bypassPreview() // Skip review step
  .create();

// Workflow is immediately active

workflow = (
    client.extract(
        ExtractOptions(
            urls=["https://sandbox.kadoa.com/magic"],
            name="Direct Activation",
            extraction=lambda builder: builder.entity("Product").field(
                "title", "Title", "STRING", FieldOptions(example="example")
            ),
        )
    )
    .bypass_preview()  # Skip review step
    .create()
)

# Workflow is immediately active

curl -X POST https://api.kadoa.com/v4/workflows \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "bypassPreview": true,
    "urls": ["https://sandbox.kadoa.com/magic"],
    "entity": "Product",
    "fields": [{"name": "title", "dataType": "STRING", "description": "Title"}]
  }'

​
Next Steps