> ## Documentation Index
> Fetch the complete documentation index at: https://jobo.world/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines Overview

> Build visual flow pipelines to transform, enrich, and remap job data with Liquid templates, AI models, and JSON merge.

<Warning>
  Pipelines are currently **Under Maintenance**. Some features may have limited
  availability.
</Warning>

## What are Pipelines?

Transformation Pipelines let you **reshape, enrich, and transform** job data before it's pushed to an external destination via an [Outbound Feed](/data-platform/outbound-feed). Build visual flows by dragging nodes onto a canvas, connecting them, and configuring each step — no code required.

Pipelines are attached to Outbound Feeds and run automatically on every sync cycle. Each job passes through your pipeline before being written to the destination (PostgreSQL, Elasticsearch, Algolia, etc.).

<Info>
  Pipelines can **only** be used with Outbound Feeds. They run automatically
  during each feed sync cycle — you cannot trigger them independently (though
  you can test them standalone in the editor).
</Info>

## Feature Highlights

<CardGroup cols={3}>
  <Card title="Liquid Templates" icon="code">
    Rename fields, filter arrays, format dates, and reshape JSON using the
    [Liquid](https://shopify.github.io/liquid/) templating language.
    Sub-millisecond execution.
  </Card>

  <Card title="AI Transforms" icon="brain">
    Send data to GPT-4o, Claude, Gemini, Llama, or any
    [OpenRouter](https://openrouter.ai/) model to extract skills, classify jobs,
    or rewrite descriptions.
  </Card>

  <Card title="JSON Merge" icon="code-merge">
    Combine AI output with original data using deep or shallow merge with
    configurable conflict resolution. Preserves all existing fields.
  </Card>
</CardGroup>

## Node Types

Every pipeline has exactly one **Input** node and one **Output** node. In between, you can add any combination of transformation nodes.

<CardGroup cols={3}>
  <Card title="Input" icon="right-to-bracket">
    Entry point — receives raw job JSON from the Outbound Feed sync engine
  </Card>

  <Card title="Liquid Template" icon="code" href="/data-platform/pipelines/node-types#liquid-template">
    Transform data using Liquid template syntax with custom filters
  </Card>

  <Card title="AI (OpenRouter)" icon="brain" href="/data-platform/pipelines/node-types#ai-openrouter">
    Process data with any LLM via OpenRouter with structured JSON output
  </Card>

  <Card title="JSON Merge" icon="code-merge" href="/data-platform/pipelines/node-types#json-merge">
    Combine two JSON inputs (base + patch) with configurable strategies
  </Card>

  <Card title="Output" icon="right-from-bracket">
    Exit point — final JSON pushed to your destination
  </Card>
</CardGroup>

## How It Works

Every pipeline is a **directed acyclic graph (DAG)** of nodes. Data flows from the Input node through transformation nodes and arrives at the Output node.

<Steps>
  <Step title="Create a Pipeline">
    Open the visual editor in **Data → Pipelines** in the dashboard. Start from
    a blank canvas or choose a pre-built template (field remapping, AI
    enrichment, etc.).
  </Step>

  <Step title="Add & Connect Nodes">
    Drag Liquid, AI, or JSON Merge nodes from the palette onto the canvas.
    Connect outputs to inputs to define the data flow.
  </Step>

  <Step title="Configure Each Node">
    Click a node to open its config panel. Write Liquid templates, set AI
    prompts and output schemas, or choose merge strategies.
  </Step>

  <Step title="Test with Real Data">
    Use the built-in test panel to run a real job from your feed through the
    pipeline and inspect each node's output before going live.
  </Step>

  <Step title="Attach to an Outbound Feed">
    Select your pipeline in the Outbound Feed settings. Every sync cycle will
    run jobs through the pipeline before pushing to the destination.
  </Step>
</Steps>

## Attaching to Outbound Feeds

Pipelines don't run in isolation — they're attached to [Outbound Feeds](/data-platform/outbound-feed). When a feed syncs, each job passes through the attached pipeline before being written to the destination.

1. **Create your pipeline** — build and test it in the Pipeline Editor
2. **Open your Outbound Feed** — navigate to the Outbound Feed page and select the feed you want to transform
3. **Select a pipeline** — in the feed settings, choose your pipeline from the dropdown

<Note>
  **One pipeline per feed.** Each outbound feed can have at most one pipeline
  attached. To apply different transformations, create separate feeds with
  different pipelines.
</Note>

## Execution Model

* Nodes execute in **topological order** using Kahn's algorithm (respecting dependencies)
* Independent nodes run in **parallel** for maximum throughput
* Each job runs through the pipeline independently — failures on one job don't affect others
* Each node produces an output JSON and timing info
* Failed nodes halt downstream execution but don't affect parallel branches

<Tip>
  **Recommended:** Keep pipelines under **10 nodes** for maintainability.
  Complex logic can often be simplified with better Liquid templates or more
  focused AI prompts.
</Tip>

## Next Steps

<CardGroup cols={2}>
  <Card title="Node Types Reference" icon="book" href="/data-platform/pipelines/node-types">
    Detailed configuration for each node type
  </Card>

  <Card title="Example Flows" icon="diagram-project" href="/data-platform/pipelines/example-flows">
    Pre-built pipeline examples for common use cases
  </Card>

  <Card title="Execution Model" icon="gears" href="/data-platform/pipelines/execution-model">
    How nodes are scheduled, parallelized, and errors are handled
  </Card>

  <Card title="Best Practices" icon="lightbulb" href="/data-platform/pipelines/best-practices">
    Tips for building efficient and reliable pipelines
  </Card>
</CardGroup>