Skip to main content

​
Working with Schemas

Define the structure of data you want to extract using the builder API: 
const extraction = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/ecommerce"],
    name: "Product Extraction",
    extraction: (builder) =>
      builder
        .entity("Product")
        .field("title", "Product name", "STRING", { example: "Laptop" })
        .field("price", "Product price", "MONEY")
        .field("inStock", "Availability", "BOOLEAN")
        .field("rating", "Star rating 1-5", "NUMBER"),
  })
  .create();

​
Reusable Schemas

For consistent data extraction across multiple workflows, you can create and manage schemas separately using the Schema Management API.

​
Schema Management API

The Schema Management API allows you to create, retrieve, and delete schemas programmatically. Saved schemas can be reused across multiple extractions, ensuring consistent data structure.

​
When to Use Saved Schemas

Use saved schemas when you:
  • Extract the same data structure from multiple websites
  • Want to maintain consistent field definitions across workflows
  • Need to programmatically manage schema lifecycle
  • Share schemas across different parts of your application
For one-off extractions, inline schema definitions (shown above) are simpler and don’t require separate schema management.

​
Create a Schema

const schema = await client.schema.createSchema({
  name: schemaName,
  entity: "Product",
  fields: [
    {
      name: "title",
      description: "Product name",
      fieldType: "SCHEMA",
      dataType: "STRING",
      example: "iPhone 15 Pro",
    },
    {
      name: "price",
      description: "Product price",
      fieldType: "SCHEMA",
      dataType: "MONEY",
    },
    {
      name: "inStock",
      description: "Availability",
      fieldType: "SCHEMA",
      dataType: "BOOLEAN",
    },
    {
      name: "rating",
      description: "Star rating",
      fieldType: "SCHEMA",
      dataType: "NUMBER",
    },
  ],
});

console.log("Schema created:", schema.id);

​
Get a Schema

Retrieve an existing schema by ID: 
const schema = await client.schema.getSchema(schemaId);

console.log(schema.name); // 'Product Schema'
console.log(schema.entity); // 'Product'
console.log(schema.schema); // Array of field definitions

​
Delete a Schema

Remove a schema when it’s no longer needed: 
await client.schema.deleteSchema(schemaId);
Deleting a schema does not affect existing workflows or extractions that were created using it.

​
Update a Schema

Modify an existing schema’s name, entity, or fields: 
const updated = await client.schema.updateSchema(schemaId, {
  name: updatedName,
  entity: "Product",
  fields: [
    {
      name: "title",
      description: "Product name",
      fieldType: "SCHEMA",
      dataType: "STRING",
      example: "MacBook Pro",
    },
    {
      name: "price",
      description: "Product price in USD",
      fieldType: "SCHEMA",
      dataType: "MONEY",
    },
    {
      name: "sku",
      description: "Stock keeping unit",
      fieldType: "SCHEMA",
      dataType: "STRING",
      example: "Stock keeping unit",
    },
  ],
});

console.log("Schema updated:", updated.id);

​
Use a Saved Schema

Reference a saved schema in your extraction: 
const extraction = await client
  .extract({
    urls: ["https://sandbox.kadoa.com/ecommerce"],
    name: "Product Extraction",
    extraction: () => ({ schemaId }),
  })
  .create();

const result = await extraction.run();

​
Field Types

Schemas support three types of fields:
  1. Regular fields - Structured data extraction (shown above)
  2. Classification fields - Categorize content into predefined labels
  3. Metadata fields - Extract raw page content (HTML, Markdown, URLs)

​
Available Data Types

For regular fields, specify the dataType: STRING • NUMBER • BOOLEAN • DATE • DATETIME • MONEY • IMAGE • LINK • OBJECT • ARRAY See data type details and examples →

​
Classification Fields

Categorize extracted content into predefined labels: 
const schema = await client.schema.createSchema({
  name: schemaName,
  entity: "Article",
  fields: [
    {
      name: "title",
      description: "Article headline",
      fieldType: "SCHEMA",
      dataType: "STRING",
      example: "Breaking News",
    },
    {
      name: "category",
      description: "Article category",
      fieldType: "CLASSIFICATION",
      categories: [
        { title: "Technology", definition: "Tech news and updates" },
        { title: "Business", definition: "Business and finance" },
        { title: "Sports", definition: "Sports coverage" },
      ],
    },
  ],
});

​
Metadata Fields (Raw Content)

Extract raw page content alongside structured data: 
const schema = await client.schema.createSchema({
  name: schemaName,
  entity: "Article",
  fields: [
    {
      name: "title",
      description: "Article headline",
      fieldType: "SCHEMA",
      dataType: "STRING",
      example: "Latest News",
    },
    {
      name: "rawMarkdown",
      description: "Page content as Markdown",
      fieldType: "METADATA",
      metadataKey: "MARKDOWN",
    },
    {
      name: "rawHtml",
      description: "Page HTML source",
      fieldType: "METADATA",
      metadataKey: "HTML",
    },
    {
      name: "pageUrl",
      description: "Page URL",
      fieldType: "METADATA",
      metadataKey: "PAGE_URL",
    },
  ],
});
Available options: HTML • MARKDOWN • PAGE_URL