> ## Documentation Index
> Fetch the complete documentation index at: https://docs.livepeer.org/llms.txt
> Use this file to discover all available pages before exploring further.
# Start your AI Orchestrator
The Livepeer AI network is currently in its **Beta** stage and is undergoing
active development. Running it on the same machine as your main Orchestrator
or Gateway node may cause stability issues. Please proceed with caution.
The Livepeer AI network is currently in **Beta** but is already integrated into
the main [go-livepeer](https://github.com/livepeer/go-livepeer) software. You
can run the Livepeer AI software using one of the following methods:
* **Docker** (Recommended): The simplest and preferred method.
* **Pre-built Binaries**: An alternative if you prefer not to use Docker.
## Orchestrator Node Architecture
In the Livepeer AI network, orchestrator operations rely on two primary **node
types**:
* **Orchestrator**: Manages and routes incoming jobs to available compute
resources.
* **Worker**: Performs the actual computation tasks.
The simplest configuration combines both roles on a single machine, utilizing
the machine's GPUs for AI inference tasks, where the orchestrator also functions
as a worker (known as a **combined AI orchestrator**). In this setup, capacity
is limited by the available GPUs and is set as
`runner container count per pipeline/model_id = capacity per pipeline/model_id`.
For expanded scalability, operators can deploy dedicated (remote) worker nodes
that connect to the orchestrator, increasing overall compute capacity.
Instructions for setting up remote workers are available on the
[next page](/ai/orchestrators/ai-worker).
## Start a Combined AI Orchestrator
Please follow the steps below to start your **combined AI orchestrator** node.
Fetch the latest Livepeer AI Docker image from the [Livepeer Docker Hub](https://hub.docker.com/r/livepeer/go-livepeer) with the following command:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/go-livepeer:master
```
The Livepeer AI network employs a [containerized workflow](https://www.ibm.com/topics/containerization) for running AI models. Fetch the latest [AI Runner](https://hub.docker.com/r/livepeer/ai-runner) image with this command:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/ai-runner:latest
```
Next, pull any pipeline-specific images if needed. Check the [pipelines](/ai/pipelines/overview) documentation for more information. For example, to pull the image for the [segment-anything-2](/ai/pipelines/segment-anything-2#pipeline-specific-image) pipeline:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/ai-runner:segment-anything-2
```
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
Confirm that the AI models are correctly set up in the `aiModels.json` file in the `~/.lpData/` directory. For guidance on configuring the `aiModels.json` file, refer to the [AI Models Configuration](/ai/orchestrators/models-config) page. The configuration file should resemble:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
[
{
"pipeline": "text-to-image",
"model_id": "ByteDance/SDXL-Lightning",
"price_per_unit": 4768371,
"warm": true
}
]
```
Execute the Livepeer AI Docker image using the following command:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker run \
--name livepeer_ai_orchestrator \
-v ~/.lpData/:/root/.lpData/ \
-v /var/run/docker.sock:/var/run/docker.sock \
--network host \
--gpus all \
livepeer/go-livepeer:master \
-orchestrator \
-serviceAddr 0.0.0.0:8936 \
-v 6 \
-nvidia 0 \
-aiWorker \
-aiModels /root/.lpData/aiModels.json \
-aiModelsDir ~/.lpData/models \
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
```
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
* `-aiWorker`: This flag enables the AI Worker functionality.
* `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
* `-aiModelsDir`: This flag indicates the directory where the AI models are stored on the host machine.
* `-aiRunnerImage`: This optional flag specifies which version of the ai-runner image is used. Example: `livepeer/ai-runner:0.0.2`
Moreover, the `--network host` flag facilitates communication between the AI Orchestrator and the AI Runner container.
Lastly, the `-nvidia` can be configured in a few ways. Use a comma seperated list of GPUs ie. `0,1` to activate specific GPU slots, each GPU will need it's own config item in `aiModels.json`. Alternativly we can use `"all"` to activate all GPUs on the machine with a single model loaded in `aiModels.json` (Warning: If different RAM size GPUs are installed it may cause containers to fail if they have less than the required RAM).
Please note that since we use [docker-out-of-docker](https://tdongsi.github.io/blog/2017/04/23/docker-out-of-docker/), the `aiModelsDir` path should be defined as being on the host machine.
If your Livepeer AI Orchestrator node is functioning correctly, you should see the following output:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
2024/05/01 09:01:39 INFO Starting managed container gpu=0 name=text-to-image_ByteDance_SDXL-Lightning modelID=ByteDance/SDXL-Lightning
...
I0405 22:03:17.427058 2655655 rpc.go:301] Connecting RPC to uri=https://0.0.0.0:8936
I0405 22:03:17.430371 2655655 rpc.go:254] Received Ping request
```
To make your Livepeer AI Orchestrator node accessible from the internet, you need to configure your network settings. Ensure that port `8936` is unblocked on your machine. Additionally, consider setting up port forwarding on your router, allowing the Gateway node to be reachable from the internet.
Download the latest Livepeer AI binary for your system:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
wget https://build.livepeer.live/go-livepeer/livepeer--gpu-.tar.gz
```
Replace `` and `` with your system's operating system and architecture. For example, for a Linux system with an AMD64 architecture, the command would be:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
wget https://build.livepeer.live/go-livepeer/livepeer-linux-gpu-amd64.tar.gz
```
See the [go-livepeer installation guide](/orchestrators/guides/install-go-livepeer#install-using-a-binary-release) for more information on the available binaries.
The Windows and MacOS (amd64) binaries of **Livepeer AI** are not available yet.
Once downloaded, extract the binary to a directory of your choice.
The Livepeer AI network employs a [containerized workflow](https://www.ibm.com/topics/containerization) for running AI models. Fetch the latest [AI Runner](https://hub.docker.com/r/livepeer/ai-runner) image with this command:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/ai-runner:latest
```
Next, pull any pipeline-specific images if needed. Check the [pipelines](/ai/pipelines/overview) documentation for more information. For example, to pull the image for the [segment-anything-2](/ai/pipelines/segment-anything-2#pipeline-specific-image) pipeline:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
docker pull livepeer/ai-runner:segment-anything-2
```
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
Confirm that the AI models are correctly set up in the `aiModels.json` file in the `~/.lpData/` directory. For guidance on configuring the `aiModels.json` file, refer to the [AI Models Configuration](/ai/orchestrators/models-config) page. The configuration file should resemble:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
[
{
"pipeline": "text-to-image",
"model_id": "ByteDance/SDXL-Lightning",
"price_per_unit": 4768371,
"warm": true
}
]
```
Run the following command to start your Livepeer AI Orchestrator node:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
./livepeer \
-orchestrator \
-serviceAddr 0.0.0.0:8936 \
-v 6 \
-nvidia "all" \
-aiWorker \
-aiModels ~/.lpData/aiModels.json \
-aiModelsDir ~/.lpData/models \
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
```
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
* `-aiWorker`: This flag enables the AI Worker functionality.
* `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
* `-aiModelsDir`: This flag indicates the directory where the AI models are stored.
* `-aiRunnerImage`: This optional flag specifies which version of the ai-runner image is used. Example: `livepeer/ai-runner:0.0.2`
If your Livepeer AI Orchestrator node is functioning correctly, you should see the following output:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
2024/05/01 09:01:39 INFO Starting managed container gpu=0 name=text-to-image_ByteDance_SDXL-Lightning modelID=ByteDance/SDXL-Lightning
...
I0405 22:03:17.427058 2655655 rpc.go:301] Connecting RPC to uri=https://0.0.0.0:8936
I0405 22:03:17.430371 2655655 rpc.go:254] Received Ping request
```
To make your Livepeer AI Orchestrator node accessible from the internet, you need to configure your network settings. Ensure that port `8936` is unblocked on your machine. Additionally, consider setting up port forwarding on your router, allowing the Gateway node to be reachable from the internet.
If no binaries are available for your system, you can build the [master branch](https://github.com/livepeer/go-livepeer/tree/master) of [go-livepeer](https://github.com/livepeer/go-livepeer) from source by following the instructions in the [Livepeer repository](/orchestrators/guides/install-go-livepeer) or by reaching out to the Livepeer community on [Discord](https://discord.gg/livepeer).
## Verify Combined AI Orchestrator Operation
Once your **combined Livepeer AI Orchestrator** node is running, verify that the
worker is operational by sending an AI inference request directly to the
[ai-runner](https://hub.docker.com/r/livepeer/ai-runner) container. You can
either use the [Swagger UI](https://fastapi.tiangolo.com/features/) interface or
a `curl` command for this check.
Open your web browser and navigate to `http://localhost:8000/docs` to access the Swagger UI interface.
In the Swagger UI, locate the `POST /text-to-image` endpoint and click the `Try it out` button. Use the following example JSON payload:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"prompt": "A cool cat on the beach."
}
```
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"images": [
{
"url": "data:image/png;base64,iVBORw0KGgoAA...",
"seed": 2724904334
}
]
}
```
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to PNG, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
Alternatively, you can use the `curl` command to test the AI inference capabilities directly. Run the following command, replacing `` with the IP address of your worker node:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST "http://localost:8000/text-to-image" -H "Content-Type: application/json" -d '{"prompt": "A cool cat on the beach."}'
```
This sends a POST request to the `text-to-image` endpoint on the worker node with the specified JSON payload.
If the AI Worker node is functioning correctly, you should receive a response similar to this:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"images": [
{
"url": "data:image/png;base64,iVBORw0KGgoAA...",
"seed": 2724904334
}
]
}
```
As with the Swagger UI response, the `url` field contains a base64 encoded image that can be decoded into PNG format using a tool like [Base64.guru](https://base64.guru/converter/decode/image/png).