> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kadoa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a dataset

> Build a dataset across many different sources with a single shared template

A dataset collects the same kind of record, in the same structure, from many websites. This guide builds a jobs dataset from company career pages in five stages: research sources, create a template, test it on a few sources, scale to all of them, and deliver the data.

You can also [let Kadoa build it](#let-kadoa-build-it).

## 1. Research sources

Ask Kadoa's agent to find sources in Workspace chat. It searches the web and lists candidate URLs, with what each page shows.

1. Select **Agent** in the bottom-right corner of any page, then select **Workspace**.
2. Describe the data, for example *"Find career pages of European fintech companies that list open roles."*

Pick the sources you want. A workflow run confirms what each one holds.

![A Workspace chat answering a request for European fintech career pages: ten candidate URLs, each marked as company-run or job board, with what each page shows](https://mintcdn.com/kadoa/v7EEuXHZHS8J6dQ7/images/assistant/workspace-research.png?fit=max\&auto=format\&n=v7EEuXHZHS8J6dQ7\&q=85\&s=4be09a346e941ea1cbecdb3f0f65c5ac)

## 2. Create the template

A [template](https://docs.kadoa.com/docs/templates.md) holds what every source shares: the setup prompt, schema, key field, data quality rules, and schedule. Create it in one of two ways:

* **From scratch**: select **Templates** in the sidebar, then **Create template**. Turn on the parts the template should manage and fill them in. See [Create a template](https://docs.kadoa.com/docs/templates.md#create-a-template).
* **From one source**: build a workflow for one career page with **Add workflow**, and fix the sample with the agent until it's right. Then open the workflow menu (**...**), select **Save as template**, choose the parts every source should share, and select **Create template**. Use this when you want to see real data first.

Before you test, check the template against [Template best practices](https://docs.kadoa.com/docs/templates/best-practices.md):

* The setup prompt says what a record is, such as "Each row is one job posting." It has no URLs.
* Each field has a short description and the narrowest type, such as `DATE` for the posted date.
* The [key field](https://docs.kadoa.com/docs/workflows/key-fields.md) is stable, such as the posting URL.
* Data quality rules are ones every source can pass.

![The template editor with a name, the Setup prompt and Schema parts turned on](https://mintcdn.com/kadoa/J2BP2lmQhWDWhIB8/images/templates/template-editor.png?fit=max\&auto=format\&n=J2BP2lmQhWDWhIB8\&q=85\&s=9bda29674dfc9c2b1c5a7323289ff10d)

## 3. Test on a few sources

A clean ten-row preview does not prove a full run holds up. Test the template on a few sources that look different from each other, such as one company site and one job board.

**1. Create the test workflows**

Select **Add workflow**, select **Template** below the prompt, and pick your template. Under **Workflow-specific instructions**, add the source URL. Repeat for each test source.

**2. Approve and run**

Approve each preview and let each workflow finish one full run.

**3. Compare with the live sites**

Check that the row count matches the site, required fields are filled, and the agent reached the detail pages the schema needs.

**4. Fix the template**

Change the template, publish the new version, and apply it to the test workflows. See [Publish and apply a new version](https://docs.kadoa.com/docs/templates.md#publish-and-apply-a-new-version).

> [!NOTE]
>
> A setup prompt change only reaches workflows built after it. Settle the prompt during testing, before you scale.

## 4. Scale to all sources

When the test sources look right, create one workflow per remaining website. Each one links to the template, so a new template version reaches every source when you apply it.

**Dashboard**

In Workspace chat, name the template and the websites in one message, for example *"Create workflows from the Job Postings template for acme.com/careers, globex.com/jobs, and initech.com/open-roles."* The agent creates one workflow per website and reports any failures.

**SDK & API**

Loop over your list and pass `templateId` with each URL. Space the requests out and keep a list of failures, so a retry picks up only those sources.

```typescript Node SDK theme={null}
const careerPages = [
  "https://acme.com/careers",
  "https://globex.com/jobs",
  // one URL per company
];
const failed: string[] = [];

for (const url of careerPages) {
  try {
    await client.workflow.create({ urls: [url], templateId: "TEMPLATE_ID" });
  } catch {
    failed.push(url);
  }
  // Space out the requests
  await new Promise((resolve) => setTimeout(resolve, 2000));
}

console.log("Retry these:", failed);
```

The REST API takes the same fields on `POST /v4/workflows`. See [Create from a template](https://docs.kadoa.com/docs/workflows/create/sdk.md#create-from-a-template).

**MCP**

Ask your AI tool, for example:

```text theme={null}
> Create a workflow from the Job Postings template for each of these career pages: acme.com/careers, globex.com/jobs, initech.com/open-roles
```

Your AI tool calls `create_workflow` with the `templateId` for each website.

Each new workflow stops at a preview for you to approve. The template's **Linked workflows** tab lists every source and the version it runs.

![A Job Postings Template page listing two linked workflows on v2, each with an Unlink button](https://mintcdn.com/kadoa/zP0ciqAKcbuKDagz/images/templates/linked-workflows.png?fit=max\&auto=format\&n=zP0ciqAKcbuKDagz\&q=85\&s=c6608ce80b445e687dacc221c67c8fc0)

## 5. Deliver to your tools

Send the dataset to the tools you already use. This example uses Snowflake. [Integrations](https://docs.kadoa.com/docs/destinations/overview.md) lists the other destinations.

A destination belongs to your team and can deliver all of its workflows or a selected set. Set it up once for the whole dataset.

**1. Request the destination**

Open any workflow and select **Pipeline** in the side rail. Choose **Deliver**, select **Add** under **Destinations**, then select **Set up** next to Snowflake. Fill in the prefilled Support Center request and submit it.

**2. Import the share**

Import the share in Snowsight and send Kadoa the verification code. [Snowflake](https://docs.kadoa.com/docs/destinations/snowflake.md) has the steps and the role you need.

**3. Query the views**

Each run arrives about 5 minutes after it completes. Each workflow has its own read-only views, named in `WORKFLOW_SCHEMA_VERSIONS`. Optional views include `TPL_<GENERATED_ID>__LATEST` for template data.

![The Deliver tab of the Pipeline panel with a connected Snowflake destination that delivers about 5 minutes after each run](https://mintcdn.com/kadoa/OUUNs4rDwUaNuNby/images/workflows/pipeline-deliver.png?fit=max\&auto=format\&n=OUUNs4rDwUaNuNby\&q=85\&s=ae20a3732f42573d4aa8359b57835351)

## Keep the dataset healthy

* Data quality rules in the template guard every linked workflow at once. See [Data quality](https://docs.kadoa.com/docs/data-quality.md).
* Check each new source's first full run against the live site.
* Subscribe a channel to **Workflow health** to hear about failed runs. Set it as a team default in **Settings > Notifications**, or add it to the template. See [Notifications](https://docs.kadoa.com/docs/notifications.md).

## Let Kadoa build it

Kadoa can build and maintain the dataset for you as a managed service, starting with a free sample. [Contact sales](https://www.kadoa.com/contact/sales) to begin.

To hand off only the setup, select **Let support set it up** when you name several websites in **Add workflow**.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.