> ## Documentation Index
> Fetch the complete documentation index at: https://docs.livepeer.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime AI Tutorial

> Set up live video-to-video AI processing using ComfyStream and the Cascade pipeline. A video stream enters, a transformed stream exits in under 100ms per frame.

export const StyledStep = ({title, icon, titleSize = 'h3', iconColor = null, titleColor = null, children, className = '', style = {}, ...rest}) => {
  const styledTitle = titleColor ? <span style={{
    color: titleColor
  }}>{title}</span> : title;
  return <Step title={styledTitle} icon={icon} iconColor={iconColor || undefined} titleSize={titleSize} className={className} style={style} {...rest}>
      {children}
    </Step>;
};

export const StyledSteps = ({children, iconColor, titleColor, lineColor, iconSize = '24px', className = '', style = {}, ...rest}) => {
  const resolvedIconColor = iconColor || 'var(--accent-dark, #18794E)';
  const resolvedTitleColor = titleColor || 'var(--lp-color-accent)';
  const resolvedLineColor = lineColor || 'var(--lp-color-accent)';
  return <div className={['docs-styled-steps', className].filter(Boolean).join(' ')} style={style} {...rest}>
      <style>{`
        .docs-styled-steps .steps > div > div.absolute > div {
          background-color: ${resolvedIconColor};
        }
        .docs-styled-steps .steps > div > div.w-full > p {
          color: ${resolvedTitleColor};
        }
        .docs-styled-steps .steps > div > div.absolute.w-px {
          background-color: ${resolvedLineColor};
        }
        .docs-styled-steps .steps > div:last-child > div.absolute.w-px::after {
          content: '';
          position: absolute;
          bottom: 0;
          left: 50%;
          transform: translateX(-50%);
          width: 6px;
          height: 6px;
          background-color: ${resolvedLineColor};
          transform: translateX(-50%) rotate(45deg);
        }
      `}</style>
      <div>
        <Steps>{children}</Steps>
      </div>
    </div>;
};

export const TableCell = ({children, align = "left", header = false, style = {}, className = "", ...rest}) => {
  const Component = header ? "th" : "td";
  return <Component className={className} style={{
    padding: "0.75rem 1rem",
    textAlign: align,
    border: header ? "none" : "1px solid var(--lp-color-border-default)",
    ...style
  }} {...rest}>
      {children}
    </Component>;
};

export const TableRow = ({children, header = false, hover = false, style = {}, className = "", ...rest}) => {
  const rowId = `table-row-${Math.random().toString(36).substr(2, 9)}`;
  return <>
      {hover && <style>{`
          #${rowId}:hover {
            background-color: var(--lp-color-bg-card);
          }
        `}</style>}
      <tr id={rowId} className={className} style={{
    ...header && ({
      backgroundColor: "var(--lp-color-accent-strong)",
      color: "var(--lp-color-on-accent)",
      fontWeight: "bold"
    }),
    ...style
  }} {...rest}>
        {children}
      </tr>
    </>;
};

export const StyledTable = ({children, variant = "default", style = {}, className = "", ...rest}) => {
  const wrapperVariants = {
    default: {
      border: "1px solid var(--lp-color-border-default)",
      backgroundColor: "var(--lp-color-bg-card)",
      overflow: "hidden"
    },
    bordered: {
      border: "2px solid var(--lp-color-accent)",
      backgroundColor: "var(--lp-color-bg-page)",
      overflow: "hidden"
    },
    minimal: {
      border: "none",
      backgroundColor: "transparent",
      overflow: "visible"
    }
  };
  return <div data-docs-styled-table-shell className={className} style={{
    width: "100%",
    padding: 0,
    margin: 0,
    ...wrapperVariants[variant],
    ...style
  }} {...rest}>
      <table data-docs-styled-table style={{
    width: "100%",
    borderCollapse: "collapse",
    borderSpacing: 0,
    margin: 0,
    backgroundColor: "transparent"
  }}>
        {children}
      </table>
    </div>;
};

export const LinkArrow = ({href, label, description, newline = true, borderColor, className = '', style = {}, ...rest}) => {
  const linkArrowStyle = {
    display: 'inline-flex',
    alignItems: 'center',
    justifyContent: 'center',
    gap: "var(--lp-spacing-1)",
    width: 'fit-content',
    ...borderColor && ({
      borderColor
    })
  };
  return <span className={className} style={style} {...rest}>
      {newline && <br />}
      <span style={linkArrowStyle}>
        <a href={href} target="_blank" rel="noopener noreferrer">
          {label}
        </a>
        <Icon icon="arrow-up-right" size={14} color="var(--lp-color-accent)" />
      </span>
      {description && description}
      {description && <div style={{
    height: "var(--lp-spacing-3)"
  }} />}
    </span>;
};

export const CustomDivider = ({color = "var(--lp-color-border-default)", middleText = "", spacing = "default", style = {}, className = "", ...rest}) => {
  const spacingPresets = {
    default: {
      margin: "24px 0"
    },
    overlap: {
      margin: "-1rem 0 -1rem 0"
    },
    tight: {
      margin: "0 0 -1rem 0"
    },
    section: {
      margin: "0 0 -2rem 0"
    },
    sectionOverlap: {
      margin: "-1rem 0 -2rem 0"
    },
    deepOverlap: {
      margin: "-1rem 0 -1.5rem 0"
    }
  };
  const spacingStyle = spacingPresets[spacing] || spacingPresets.default;
  return <div role="separator" aria-orientation="horizontal" className={className} style={{
    display: "flex",
    alignItems: "center",
    ...spacingStyle,
    fontSize: style?.fontSize || "16px",
    height: "fit-content",
    ...style
  }} {...rest}>
      <span style={{
    marginRight: "var(--lp-spacing-px-8)",
    opacity: 0.2
  }}>
        <Icon icon="/snippets/assets/logos/Livepeer-Logo-Symbol-Theme.svg" />
      </span>
      <div style={{
    flex: 1,
    height: "1px",
    background: "var(--lp-color-border-default)",
    opacity: 0.4
  }}></div>
      {middleText && <>
          <Icon icon="circle" size={2} />
          <span style={{
    margin: "0 8px",
    fontWeight: "bold",
    color: color,
    opacity: 0.7
  }}>
            {middleText}
          </span>
          <Icon icon="circle" size={2} />
        </>}
      <div style={{
    flex: 1,
    height: "1px",
    background: "var(--lp-color-border-default)",
    opacity: 0.4
  }}></div>
      <span style={{
    marginLeft: "var(--lp-spacing-px-8)",
    opacity: 0.2
  }}>
        <span style={{
    display: "inline-block",
    transform: "scaleX(-1)"
  }}>
          <Icon icon="/snippets/assets/logos/Livepeer-Logo-Symbol-Theme.svg" />
        </span>
      </span>
    </div>;
};

<Tip>
  Realtime AI is Livepeer's highest-value compute offering. A continuous stream of video frames enters the Orchestrator, AI transforms each frame, and a processed stream exits. This is fundamentally different from batch inference - there is no request-response cycle.
</Tip>

***

This tutorial sets up a working `live-video-to-video` pipeline using the Cascade architecture and ComfyStream. By the end, a live video stream enters the Orchestrator, StreamDiffusion transforms each frame in a continuous low-latency pipeline, and the output stream is viewable. Estimated time: **3 hours** (most of this is model download time).

**What you will verify:**

* The `livepeer/ai-runner:live-base` container starts cleanly with GPU access
* The `live-video-to-video` pipeline registers at `tools.livepeer.cloud/ai/network-capabilities`
* A test stream sends successfully and the transformed output is visible

<CustomDivider />

## How realtime AI differs from batch

<StyledTable variant="bordered">
  <thead>
    <TableRow header>
      <TableCell header />

      <TableCell header>Batch AI</TableCell>
      <TableCell header>Realtime AI (Cascade)</TableCell>
    </TableRow>
  </thead>

  <tbody>
    <TableRow>
      <TableCell>**Input**</TableCell>
      <TableCell>Discrete file or prompt</TableCell>
      <TableCell>Continuous WebRTC video stream</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>**Output**</TableCell>
      <TableCell>Result returned once</TableCell>
      <TableCell>Continuous processed stream</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>**Latency target**</TableCell>
      <TableCell>Seconds per request</TableCell>
      <TableCell>\<100ms per frame</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>**Runtime**</TableCell>
      <TableCell>`livepeer/ai-runner`</TableCell>
      <TableCell>`livepeer/ai-runner:live-base`</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>**Min VRAM**</TableCell>
      <TableCell>4–24 GB</TableCell>
      <TableCell>24 GB recommended</TableCell>
    </TableRow>
  </tbody>
</StyledTable>

At 30 fps, the frame budget is 33 ms. The pipeline must receive, process, and emit each frame within that window. StreamDiffusion's architecture is purpose-built for this: stream batching, residual CFG, and stochastic similarity filtering combine to achieve 30+ fps on an RTX 4090.

<CustomDivider />

## Prerequisites

<StyledTable variant="bordered">
  <thead>
    <TableRow header>
      <TableCell header>Requirement</TableCell>
      <TableCell header>Notes</TableCell>
    </TableRow>
  </thead>

  <tbody>
    <TableRow>
      <TableCell>NVIDIA GPU, 24 GB VRAM</TableCell>
      <TableCell>RTX 4090 strongly recommended. RTX 3090 functional with less headroom. A100/H100 for production multi-stream. Cards below 24 GB are typically insufficient.</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>CUDA 12.0+ drivers</TableCell>
      <TableCell>`nvidia-smi` shows driver and CUDA version. Min driver: `525.60.13`.</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>Docker + NVIDIA Container Toolkit</TableCell>
      <TableCell>Verify: `docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi`</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>go-livepeer running with `-aiWorker`</TableCell>
      <TableCell>Use AI Earning Quickstart for the base AI node setup, then return here for the live pipeline.</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>30 GB free disk space</TableCell>
      <TableCell>StreamDiffusion model weights + ComfyStream dependencies</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>FFmpeg</TableCell>
      <TableCell>For sending a test RTMP stream. Install: `apt-get install ffmpeg`</TableCell>
    </TableRow>

    <TableRow>
      <TableCell>CPU: 8+ cores</TableCell>
      <TableCell>WebRTC frame encode/decode is CPU-bound</TableCell>
    </TableRow>
  </tbody>
</StyledTable>

<Warning>
  GPUs below 24 GB VRAM (RTX 3080 10 GB, RTX 3060 12 GB) are typically insufficient for live-video inference at acceptable frame rates. StreamDiffusion's stream batch buffers, model weights, and ControlNet adapters combined exhaust available VRAM on these cards.
</Warning>

<CustomDivider />

## Step 1: Verify GPU and Docker access

```bash icon="terminal" filename="verify-gpu" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
nvidia-smi
```

Note the GPU name, VRAM total, and driver version. Driver must be `525.60.13` or newer.

Confirm Docker GPU access:

```bash icon="terminal" filename="verify-docker-gpu" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
```

The GPU table should appear inside the container output. Re-install the NVIDIA Container Toolkit until this command succeeds.

<CustomDivider />

## Step 2: Pull the live-base AI Runner image

The `live-base` image is separate from the standard `livepeer/ai-runner` used for batch pipelines. It includes ComfyStream, ComfyUI, and StreamDiffusion dependencies:

```bash icon="terminal" filename="pull-live-runner" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/ai-runner:live-base
```

Verify the image is available:

```bash icon="terminal" filename="verify-live-runner" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker images | grep "ai-runner.*live-base"
```

Verify CUDA works inside the container:

```bash icon="terminal" filename="verify-cuda-in-container" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run --gpus all --rm livepeer/ai-runner:live-base \
  python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')"
```

Expected: `CUDA available: True, GPU: NVIDIA GeForce RTX 4090`

<CustomDivider />

## Step 3: Download ComfyStream model weights

ComfyStream requires model weights before the container starts. Clone the ComfyStream repository and run the download script:

```bash icon="terminal" filename="clone-comfystream" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
git clone https://github.com/livepeer/comfystream
cd comfystream
pip install -r requirements.txt
```

Download StreamDiffusion and base models:

```bash icon="terminal" filename="download-models" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
python scripts/download_models.py
```

This downloads approximately 15 to 20 GB. Wait for completion. Models are downloaded to the directory that will be mounted into the AI Runner container via `-aiModelsDir`.

Verify the download:

```bash icon="terminal" filename="verify-models" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
ls -lh ~/.lpData/models/ | head -20
```

<CustomDivider />

## Step 4: Configure aiModels.json for live pipeline

For the live pipeline, `model_id` names the ComfyUI workflow or pipeline. The underlying models load inside the ComfyStream container.

```bash icon="terminal" filename="write-aimodels" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
cat > ~/.lpData/aiModels.json << 'EOF'
[
  {
    "pipeline": "live-video-to-video",
    "model_id": "streamdiffusion",
    "price_per_unit": 500,
    "warm": true
  }
]
EOF
```

`price_per_unit` for the live pipeline is charged per frame, unlike batch pipelines that charge per pixel or per millisecond. Set a value at or below the current Gateway caps in `-maxPricePerCapability` for `live-video-to-video`. Check current rates at [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities).

<CustomDivider />

## Step 5: Start go-livepeer with live AI flags

Existing AI nodes should stop and restart with the updated `aiModels.json`. Fresh setups should use:

```bash icon="terminal" filename="start-orchestrator" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run -d \
  --name livepeer-orchestrator \
  -v ~/.lpData/:/root/.lpData/ \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --network host \
  --gpus all \
  livepeer/go-livepeer:latest \
  -network arbitrum-one-mainnet \
  -ethUrl https://arb-mainnet.g.alchemy.com/v2/YOUR_API_KEY \
  -orchestrator \
  -transcoder \
  -nvidia 0 \
  -pricePerUnit 1000 \
  -serviceAddr YOUR_PUBLIC_IP:8935 \
  -aiWorker \
  -aiModels /root/.lpData/aiModels.json \
  -aiModelsDir /root/.lpData/models \
  -v 6
```

Wait for the live runner container to start and the pipeline to warm. This takes longer than batch pipelines - ComfyStream loads the full ComfyUI environment and StreamDiffusion model stack:

```bash icon="terminal" filename="watch-startup" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker logs -f livepeer-orchestrator 2>&1 | grep -i "live\|cascade\|pipeline\|warm\|error"
```

Expected after a typical 5 to 10 minute warm-up:

```text icon="terminal" title="Expected live-runner startup log" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Starting AI worker
Starting live-video-to-video pipeline: streamdiffusion
ComfyStream container started
Warm model loaded: streamdiffusion
```

Check the live runner container is running:

```bash icon="terminal" filename="check-containers" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker ps | grep livepeer
```

Two containers should be running: `livepeer-orchestrator` and the AI Runner container for the live pipeline.

<CustomDivider />

## Step 6: Set up the Gateway for live routing

Start an off-chain Gateway that routes `live-video-to-video` jobs to the Orchestrator for this local test:

```bash icon="terminal" filename="start-gateway" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run -d \
  --name livepeer-gateway-live \
  -v ~/.lpData-gateway-live/:/root/.lpData/ \
  --network host \
  livepeer/go-livepeer:latest \
  -gateway \
  -cliAddr 127.0.0.1:7936 \
  -httpAddr 0.0.0.0:8936 \
  -rtmpAddr 0.0.0.0:1935 \
  -orchAddr http://127.0.0.1:8935 \
  -httpIngest \
  -remoteSignerAddr https://signer.eliteencoder.net \
  -network offchain
```

Verify the Gateway started:

```bash icon="terminal" filename="check-gateway" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker logs livepeer-gateway-live 2>&1 | grep -i "started\|gateway\|rtmp\|http" | head -10
```

Expected:

```text icon="terminal" title="Expected gateway startup log" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Gateway started on :8936
RTMP server listening on :1935
```

<Note>
  For a production Gateway routing live AI jobs on-chain, configure `-maxPricePerCapability` with a cap for `live-video-to-video`. The Gateway routes only to Orchestrators priced at or below this cap, regardless of hardware capability.
</Note>

<CustomDivider />

## Step 7: Send a test stream

Send a test RTMP stream through the Gateway using FFmpeg. This simulates a camera or OBS stream:

```bash icon="terminal" filename="send-test-stream" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Generate a synthetic test pattern and stream it via RTMP
ffmpeg \
  -re \
  -f lavfi -i "testsrc=size=512x512:rate=30" \
  -f lavfi -i "sine=frequency=440:sample_rate=44100" \
  -vcodec libx264 \
  -preset ultrafast \
  -tune zerolatency \
  -b:v 2000k \
  -acodec aac \
  -f flv \
  rtmp://localhost:1935/live/test-stream-key
```

This streams a synthetic test pattern at 30 fps. The stream should be processed by the `live-video-to-video` pipeline. Keep this running while checking the output.

In a second terminal, watch the Orchestrator process frames:

```bash icon="terminal" filename="watch-orchestrator" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker logs -f livepeer-orchestrator 2>&1 | grep -i "frame\|stream\|cascade\|inference" | head -20
```

Expected:

```text icon="terminal" title="Expected frame-processing log" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Received live stream: test-stream-key
Dispatching to live-video-to-video pipeline
Processing frame 0
Processing frame 1
...
```

<CustomDivider />

## Step 8: Verify the transformed output

Retrieve the processed output stream from the Gateway:

```bash icon="terminal" filename="get-output" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Pull the transformed output HLS stream
curl -o output-manifest.m3u8 http://localhost:8936/hls/test-stream-key/index.m3u8
```

A non-empty manifest confirms the live pipeline is processing frames and delivering output.

To view the output stream in VLC or another player:

```bash icon="terminal" filename="view-output" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
ffplay http://localhost:8936/hls/test-stream-key/index.m3u8
```

Check the network registration:

Open [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities) and search for the Orchestrator address. The `live-video-to-video` pipeline should appear with **Warm** status.

**Latency check:**

Monitor frame processing times in the Orchestrator logs. At 30 fps, each frame should be processed in under 33 ms. Repeated frame times above 33 ms show the pipeline is falling behind the incoming stream:

```bash icon="terminal" filename="check-latency" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker logs livepeer-orchestrator 2>&1 | grep -i "frame.*ms\|latency\|processing time" | tail -20
```

<CustomDivider />

## Troubleshooting

**Frames dropping or high latency:**

* The model is running too slowly for the target fps. StreamDiffusion at 2 steps is the minimum viable configuration for 30 fps on an RTX 4090. Try reducing output resolution.
* VRAM OOM: reduce `stream_batch_size` in the StreamDiffusion config.
* CPU bottleneck: WebRTC frame encode/decode is CPU-bound. Monitor CPU with `htop`.

**Pipeline job registration check:**

* Confirm `live-video-to-video` appears at [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities).
* Verify the live runner container is running: `docker ps --filter name=livepeer`.
* Check the container started cleanly: `docker logs <live-runner-container-name>`.

**ComfyStream container failing to start:**

```bash icon="terminal" filename="debug-container" theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run --gpus all --rm livepeer/ai-runner:live-base \
  python -c "import torch; print(torch.cuda.is_available())"
```

This command should return `True`. Any other result means CUDA is unavailable inside the container, so re-install the NVIDIA Container Toolkit.

<CustomDivider />

## What happened

The Cascade architecture processed a live stream end-to-end:

1. **FFmpeg** sent an RTMP stream to the Gateway at `:1935`.
2. **The Gateway** routed the stream to the Orchestrator at `:8935` with a `live-video-to-video` capability match.
3. **The Orchestrator** dispatched the stream to the `livepeer/ai-runner:live-base` container.
4. **ComfyStream** received each frame via WebRTC, ran it through the StreamDiffusion workflow, and emitted the processed frame.
5. **The Orchestrator** collected processed frames and returned the output stream through the Gateway.
6. **The HLS output** was available at the Gateway's `/hls/` endpoint.

Payment for live streams uses an interval-based model instead of per-frame settlement: the Gateway sends periodic PM tickets at a configurable interval (`-livePaymentInterval`, default 5 seconds) instead of one ticket per frame. This reduces payment overhead for continuous streams.

<CustomDivider />

## Related pages

<CardGroup cols={2}>
  <Card title="Realtime AI Setup" icon="video" href="/v2/orchestrators/guides/ai-and-job-workloads/realtime-ai-setup" arrow horizontal>
    Full reference for Cascade architecture, ComfyStream workflows, ControlNet variants, and multi-stream capacity.
  </Card>

  <Card title="Full AI Pipeline Tutorial" icon="diagram-project" href="/v2/orchestrators/guides/tutorials/full-ai-pipeline-tutorial" arrow horizontal>
    Batch inference end-to-end - the alternative pipeline for request-response AI workloads.
  </Card>

  <Card title="Capacity Planning" icon="gauge" href="/v2/orchestrators/guides/config-and-optimisation/capacity-planning" arrow horizontal>
    VRAM budgeting for realtime workloads and the one-warm-model-per-GPU constraint.
  </Card>

  <Card title="Gateway-Orchestrator Interface" icon="handshake" href="/v2/orchestrators/guides/advanced-operations/gateway-orchestrator-interface" arrow horizontal>
    Production combined setup with port allocation and pricing alignment.
  </Card>
</CardGroup>