DatasetPipelineConfig

Complete pipeline configuration for a dataset.

Defines a sequence of processing steps to be executed when entries are uploaded or manually triggered.

enabledEnabled (boolean)

Whether the pipeline is active for this dataset

Default value: true

triggerTrigger (string)

When to trigger the pipeline: automatically on upload or manually

Possible values: [on_upload, manual]

Default value: on_upload

steps object[]required

Ordered list of pipeline steps to execute

Possible values: >= 1

Array [

nameName (string)required

Unique name for this step within the pipeline (e.g., 'ocr', 'chunk', 'embed')

Examples:

Example 1

Example: ocr

Example 2

Example: chunk

Example 3

Example: embed

Example 4

Example: store-qdrant

componentComponent (string)required

Component identifier (WorkflowTemplate name in Argo)

Examples:

Example 1

Example: mistral-ocr-processor-v1

Example 2

Example: markdown-chunker-v1

Example 3

Example: data-cluster-embedding-v1

templateTemplate (string)required

Template name to reference (usually same as component without version suffix)

Examples:

Example 1

Example: mistral-ocr-processor

Example 2

Example: markdown-chunker

Example 3

Example: data-cluster-embedding

parameters object

Component-specific parameters

property name*any

Component-specific parameters

Examples:

Example 1

Example: {"overlap":128,"size":1024}

Example 2

Example: {"batchsize":32,"concurrency":10}

dependsstring[]

List of step names this step depends on (for ordering)

Examples:

Example 1

Example: ["ocr"]

Example 2

Example: ["chunk","embed"]

workflow_parametersstring[]

Workflow-level parameters to inject (e.g., entry_id, dataset_id)

Examples:

Example 1

Example: ["entry_id"]

Example 2

Example: ["entry_id","dataset_id"]

Example 3

Example: []

inputs object[]

Explicit artifact inputs with source step and artifact name

Array [

nameName (string)required

Input artifact name expected by this step's template

Examples:

Example 1

Example: input-text

Example 2

Example: input-pdf

Example 3

Example: chunks

from_stepFrom Step (string)required

Name of the step that produces this artifact

Examples:

Example 1

Example: ocr

Example 2

Example: fetch-entry

Example 3

Example: chunk

from_artifactFrom Artifact (string)required

Output artifact name from the source step

Examples:

Example 1

Example: output-markdown

Example 2

Example: entry-file

Example 3

Example: chunks

]

outputsstring[]

Declared output artifact names (for documentation only, not validated)

Examples:

Example 1

Example: ["output-markdown","output-figures"]

Example 2

Example: ["chunks"]

Example 3

Example: ["embeddings"]

]

timeoutTimeout (string)

Maximum execution time for the entire pipeline (e.g., '30m', '2h')

Possible values: Value must match regular expression ^\d+[smh]$

Default value: 30m

DatasetPipelineConfig

{
  "enabled": true,
  "steps": [
    {
      "component": "mistral-ocr-processor-v1",
      "depends": [],
      "inputs": [],
      "name": "ocr",
      "outputs": [
        "output-markdown",
        "output-figures"
      ],
      "parameters": {},
      "template": "mistral-ocr-processor",
      "workflow_parameters": []
    },
    {
      "component": "markdown-chunker-v1",
      "depends": [
        "ocr"
      ],
      "inputs": [
        {
          "from_artifact": "output-markdown",
          "from_step": "ocr",
          "name": "input-text"
        }
      ],
      "name": "chunk",
      "outputs": [
        "chunks"
      ],
      "parameters": {
        "overlap": 128,
        "size": 1024
      },
      "template": "markdown-chunker",
      "workflow_parameters": []
    }
  ],
  "timeout": "30m",
  "trigger": "on_upload"
}