Skip to main content

Pipeline Presets & the General Purpose Pipeline

A pipeline is the chain of processing steps that turns a raw uploaded document into searchable, AI-queryable content: it ingests the file, processes it (OCR / parsing, figure linking, chunking, embedding), and indexes the result for keyword and semantic search. A preset is a ready-made pipeline — a named, pre-wired configuration that you apply to a dataset in a single call instead of authoring the full step graph by hand.

This guide is the API-and-presets companion to Configure a Pipeline. Where that guide walks through the web wizard and the trade-offs between presets, this one shows how to discover, apply, and trigger presets programmatically — and gives a complete create → upload → poll → query walkthrough you can copy.

For the architecture behind pipelines (stages, components, the execution model), see Pipelines (Concept).

What a Preset Gives You​

When you apply a preset to a dataset, the platform writes a full DatasetPipelineConfig onto that dataset — the ordered list of steps, their dependencies, parameters, and the trigger mode. From then on, every document uploaded to the dataset is processed by that pipeline.

Concretely, a preset bundles three things:

  • An ordered step graph (DAG) — which components run, in what order, and which step's output feeds which step's input.
  • Component parameters — chunk size, overlap, the embedding configuration, and so on.
  • A trigger mode — on_upload (process automatically) or manual (wait for an explicit trigger). See The on_upload Trigger.

You never have to hand-write any of this for a common format — that is the point of a preset.

The general_purpose Preset​

general_purpose is the default, production-ready preset and the right choice for most document collections. It processes PDF, DOCX, and image files through a complete OCR-based pipeline:

Upload -> OCR -> Link Figures -> Chunk -> Embed -> Store
StageWhat it does
OCRExtracts text as structured markdown and identifies images (Mistral OCR). Handles scanned documents, multi-column layouts, figures, and tables.
Link FiguresResolves cross-references between text and figures and converts extracted images to PNG.
ChunkSplits the markdown into semantic, heading-aware chunks (token-bounded, with overlap).
EmbedGenerates a vector embedding for each chunk using the cluster's configured embedding provider.
Store / RegisterSaves processed content, indexes it for keyword search, and upserts chunk vectors for semantic search.

The preset's own one-line description, as returned by the API, is:

Process PDF/DOCX files: OCR -> Link Figures -> Chunk -> Embed -> Store

tip

Use general_purpose unless you have a specific reason not to. Its OCR-based extraction handles the widest range of document complexity, and its output is immediately searchable through both keyword and semantic search.

Applying general_purpose to a Dataset​

Apply a preset by name with a single POST. The preset name is passed as a query parameter; the request body is an optional JSON object of parameter overrides (pass null, or omit it, to use the preset's defaults).

Endpoint: POST /api/v1/pipelines/datasets/{dataset_id}/apply-preset?preset_name=general_purpose (reference: Apply Preset)

Python​

from data_api_client import ApiClient, Configuration, PipelinesApi

config = Configuration(
    host="https://api.alien.club/clusters/YOUR_CLUSTER_ID/proxy"
)
client = ApiClient(
    config,
    header_name="Authorization",
    header_value="Bearer oat_YOUR_API_TOKEN",
)
pipelines_api = PipelinesApi(client)

# Apply the general_purpose preset to the dataset.
dataset = pipelines_api.apply_preset_api_v1_pipelines_datasets_dataset_id_apply_preset_post(
    dataset_id=YOUR_DATASET_ID,
    preset_name="general_purpose",
)
print(f"Preset applied to dataset {dataset.id}")

cURL​

curl -X POST \
  "https://api.alien.club/clusters/YOUR_CLUSTER_ID/proxy/api/v1/pipelines/datasets/YOUR_DATASET_ID/apply-preset?preset_name=general_purpose" \
  -H "Authorization: Bearer oat_YOUR_API_TOKEN"

The response is the updated Dataset object — the preset's full step graph is now stored on the dataset, with trigger defaulting to on_upload.

info

Applying a preset is a separate step from creating the dataset. Create the dataset first (see Create a Dataset), then apply a preset to it. A typical integration creates the dataset, immediately applies general_purpose, and then begins uploading.

The on_upload Trigger​

A pipeline's trigger mode decides when processing starts. It lives on the dataset's pipeline config and has two values:

TriggerBehaviour
on_upload (default)The pipeline runs automatically the moment a file finishes uploading. Upload a document and it becomes searchable with no further calls.
manualUploaded files sit in the uploaded status until you explicitly trigger processing per entry. Useful for batch-then-process workflows or pre-processing review.

general_purpose ships with trigger: "on_upload", so a fresh dataset configured with it needs nothing beyond uploading. The stored config looks like:

{
  "enabled": true,
  "trigger": "on_upload",
  "timeout": "30m",
  "steps": [ /* OCR -> Link Figures -> Chunk -> Embed -> Store */ ]
}

Triggering a Pipeline Manually​

If a dataset uses manual (or you uploaded files before a pipeline was configured), trigger each entry explicitly:

Endpoint: POST /api/v1/entries/{entry_id}/trigger-pipeline (reference: Trigger Pipeline)

from data_api_client import EntriesApi

entries_api = EntriesApi(client)
entries_api.trigger_pipeline_api_v1_entries_entry_id_trigger_pipeline_post(
    entry_id=ENTRY_ID,
)

To inspect or change the trigger mode for an existing dataset, read or patch its pipeline config — see Configure a Pipeline.

The Entry Status Lifecycle​

When auto-trigger is on, each uploaded file (an entry) advances through a fixed set of statuses. A consumer that wants to know when a document is ready polls the entry's status until it reaches a terminal state. The status values are:

StatusMeaning
pendingEntry record created; the file has not been stored yet.
uploadingThe file is being transferred and stored in the data cluster.
uploadedThe file is stored. With manual trigger it stays here until you trigger processing; with on_upload it advances to processing.
processingThe pipeline is running (OCR, figure linking, chunking, embedding, registration).
processedTerminal (success). All stages completed. The document is indexed for keyword and semantic search.
errorTerminal (failure). A stage failed after retries. Inspect the entry for the failing step.
note

These are the authoritative EntryStatus values from the Data API. The two terminal states are processed and error — poll until every entry you care about has reached one of them. A reasonable poll interval is every 10 seconds.

Discovering Other Presets and Components​

general_purpose is not the only preset. Others may be available on your cluster — for example a scientific-articles preset for JATS XML / MECA academic archives, or presets enabled on demand for your organization. Rather than hard-coding names, discover what is available at runtime.

List Available Presets​

Endpoint: GET /api/v1/pipelines/presets (reference: List Pipeline Presets)

curl "https://api.alien.club/clusters/YOUR_CLUSTER_ID/proxy/api/v1/pipelines/presets" \
  -H "Authorization: Bearer oat_YOUR_API_TOKEN"
{
  "presets": [
    {
      "name": "general_purpose",
      "description": "Process PDF/DOCX files: OCR -> Link Figures -> Chunk -> Embed -> Store"
    }
  ],
  "total": 1
}

Each entry carries the preset's name (what you pass to apply-preset) and a human-readable description.

Get Full Preset Details​

To inspect the actual step graph a preset would apply — its components, dependencies, and parameters — call:

Endpoint: GET /api/v1/pipelines/presets/details (reference: Get Preset Details)

This returns the detailed configuration for each preset, keyed by name — useful when you want to apply a preset and then patch a parameter, or build a custom pipeline starting from a known-good preset.

List Available Components​

Presets are assembled from individual components (OCR, chunker, embedder, …). To see every component the cluster can run — names, versions, expected inputs and outputs — call:

Endpoint: GET /api/v1/pipelines/components (reference: List Pipeline Components)

curl "https://api.alien.club/clusters/YOUR_CLUSTER_ID/proxy/api/v1/pipelines/components" \
  -H "Authorization: Bearer oat_YOUR_API_TOKEN"

Use this when composing a custom pipeline — it tells you which components exist and how to wire their artifacts together.

tip

If a preset you need (for example, the scientific-articles pipeline) is not in the list-presets response, it may be available on demand for your organization. Contact us to have it enabled.

End-to-End Walkthrough​

This is the complete flow: create a dataset, apply general_purpose, upload a document, poll until it is processed, then query it.

import time
from data_api_client import (
    ApiClient, Configuration,
    DatasetsApi, EntriesApi, PipelinesApi, SearchApi,
)
from data_api_client.models.dataset_create_request import DatasetCreateRequest
from data_api_client.models.entry_create_request import EntryCreateRequest
from data_api_client.models.vector_search_request import VectorSearchRequest

config = Configuration(host="https://api.alien.club/clusters/YOUR_CLUSTER_ID/proxy")
client = ApiClient(
    config,
    header_name="Authorization",
    header_value="Bearer oat_YOUR_API_TOKEN",
)
datasets_api = DatasetsApi(client)
entries_api = EntriesApi(client)
pipelines_api = PipelinesApi(client)
search_api = SearchApi(client)

# 1. Create the dataset.
dataset = datasets_api.create_dataset_api_v1_datasets_post(
    DatasetCreateRequest(
        name="My Document Collection",
        slug="my-document-collection",
        description="Created via the API",
        dataset_type="text",
    )
)
dataset_id = dataset.id

# 2. Apply the general_purpose preset (defaults to trigger: on_upload).
pipelines_api.apply_preset_api_v1_pipelines_datasets_dataset_id_apply_preset_post(
    dataset_id=dataset_id,
    preset_name="general_purpose",
)

# 3. Create an entry, then upload the file to it (two separate calls).
entry = entries_api.create_entry_api_v1_entries_post(
    EntryCreateRequest(
        dataset_id=dataset_id,
        name="My Research Paper",
        slug="my-research-paper",
        description="Uploaded via the API",
        metadata={},
    )
)
entry_id = entry.entry.id

with open("paper.pdf", "rb") as f:
    entries_api.upload_file_to_entry_api_v1_entries_entry_id_upload_post(
        entry_id=entry_id,
        file=("paper.pdf", f.read()),
        file_type="original",
    )
# Because the dataset uses on_upload, processing now starts automatically.

# 4. Poll the entry status until it reaches a terminal state.
while True:
    current = entries_api.get_entry_api_v1_entries_entry_id_get(entry_id)
    status = str(current.status).split(".")[-1].lower()
    print(f"status: {status}")
    if status in ("processed", "error"):
        break
    time.sleep(10)

# 5. Query the now-indexed document with semantic search.
results = search_api.vector_search_chunks_api_v1_vector_chunks_post(
    VectorSearchRequest(
        query="the topic you are looking for",
        dataset_ids=[dataset_id],
        limit=5,
    )
)
for chunk in results.results:
    print(f"{chunk.score:.2f}  {chunk.chunk_text[:120]}...")

The same flow is available in TypeScript via @alien/data-api-client — see Upload Documents for the SDK setup and the entry create/upload pattern.

What's Next​