How data integrations work

A data integration delivers data from completed runs to storage you own or a data platform you use. Kadoa configures every integration for you through the Support Center.

When Kadoa delivers

Kadoa delivers data to Amazon S3, Google Cloud Storage, and SFTP immediately after a workflow run completes. Data is available in Snowflake and Databricks about 5 minutes after completion.

Formats

Amazon S3, Google Cloud Storage, and SFTP support these file formats.

Format	Use case
Parquet	Compressed columnar data for analytics and warehouse loads
JSONL	One JSON record per line for streaming and line-based processing
JSON	A JSON document with a metadata envelope for APIs
CSV	Comma-separated values for spreadsheets and systems that require flat files

File layout and path template

The default path template is:

{teamId}/{workflowId}/{runDatetimeSafe}-{runId}/data

If you set a prefix, Kadoa adds it before this path. For each run, Kadoa writes one file for each selected format. With the default template, each run has a unique path. This prevents overwrites of files from other runs.

Token	Description	Example
`{teamId}`	Your team UUID	`a1b2c3d4-5e6f-7a8b-9c0d-e1f2a3b4c5d6`
`{workflowId}`	Workflow identifier	`b2c3d4e5-6f7a-8b9c-0d1e-f2a3b4c5d6e7`
`{runId}`	Run identifier	`c3d4e5f6-7a8b-9c0d-1e2f-a3b4c5d6e7f8`
`{runDatetimeSafe}`	Run timestamp without colons, safe for filenames	`2025-01-15_10-30-00Z`
`{runDatetime}`	Run timestamp in ISO 8601 format	`2025-01-15T10:30:00.000Z`
`{runDate}`	Run date	`2025-01-15`

Additional fields

Additional fields add source labels or workflow details to every exported record. They are available in CSV, JSONL, and JSON files, but not Parquet. Each field has a name you choose and a fixed value or a variable. Kadoa replaces variables with the details of the workflow or run.

Field name	Configured value	Example value in the file
`source`	`Kadoa`	`Kadoa`
`workflowName`	`{workflowName}`	`Product Monitor`
`runTimestamp`	`{runDatetime}`	`2025-01-15T10:30:00.000Z`

Supported variables:

{teamId}
{workflowId}
{workflowName}
{runId}
{runDate}
{runDatetime}

Workflow health files

Health files let you monitor workflow status from your Amazon S3, Google Cloud Storage, or SFTP destination. They are enabled by default for Amazon S3 and Google Cloud Storage, and disabled for SFTP. Kadoa can enable or disable health files for your integration. When enabled, Kadoa writes one JSON file for each of these events:

workflow_finished: a scheduled workflow finishes.
workflow_failed: a scheduled or realtime workflow fails.
workflow_recovered: a realtime workflow recovers.

File path:

<PREFIX>/workflow-health/date=2025-01-15/<EVENT_ID>.json

{
  "schemaVersion": 1,
  "eventId": "<EVENT_ID>",
  "eventType": "workflow_failed",
  "status": "failed",
  "eligibleAt": "2025-01-15T10:30:00.000Z",
  "workflow": { "id": "<WORKFLOW_ID>", "name": "Product Monitor" },
  "runId": "<RUN_ID>"
}

runId is present only when the event belongs to a run.

Request, change, or disable an integration

Submit integration requests through the Support Center. For a change, include the integration, what you want to change, and the new value.

Glossary

Data integration: The configuration that sends completed-run data to your storage or data platform. Delivery: One transfer or publication for one run. Run: One execution of a workflow. Workflow: The configured process that collects and structures your data. Team: The Kadoa workspace that owns workflows and data integrations.

Foundations

Security & Compliance

Build with UI

Build with Code

Integrations

When Kadoa delivers

Formats

File layout and path template

Additional fields

Workflow health files

Request, change, or disable an integration

Glossary

​When Kadoa delivers

​Formats

​File layout and path template

​Additional fields

​Workflow health files

​Request, change, or disable an integration

​Glossary

When Kadoa delivers

Formats

File layout and path template

Additional fields

Workflow health files

Request, change, or disable an integration

Glossary