} ; }; export const CustomDivider = ({color = "var(--lp-color-border-default)", middleText = "", spacing = "default", style = {}, className = "", ...rest}) => { const spacingPresets = { default: { margin: "24px 0" }, overlap: { margin: "-1rem 0 -1rem 0" }, tight: { margin: "0 0 -1rem 0" }, section: { margin: "0 0 -2rem 0" }, sectionOverlap: { margin: "-1rem 0 -2rem 0" }, deepOverlap: { margin: "-1rem 0 -1.5rem 0" } }; const spacingStyle = spacingPresets[spacing] || spacingPresets.default; return

{middleText && <> {middleText} }

; }; # Livepeer Troubleshooting Index Aggregated troubleshooting reference across audience-specific operational pages. Each section is mirrored from an audience tab; the original page remains available there with full operational context. ## Source pages | Section | Source page | Audience | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------ | | Gateway troubleshooting | [/v2/Gateways/guides/monitoring-and-tooling/troubleshooting](/v2/Gateways/guides/monitoring-and-tooling/troubleshooting) | Gateway | | Orchestrator troubleshooting | [/v2/Orchestrators/guides/monitoring-and-tooling/troubleshooting](/v2/Orchestrators/guides/monitoring-and-tooling/troubleshooting) | Orchestrator | ## Gateway troubleshooting Source: [/v2/Gateways/guides/monitoring-and-tooling/troubleshooting](/v2/Gateways/guides/monitoring-and-tooling/troubleshooting) errors operators encounter. For "how does X work?" questions, see the . ## Video Issues Work through these causes in order. The first one confirmed is the fix. **1. ETH deposit or reserve is depleted.** When the deposit reaches zero, the Gateway silently stops routing - no error is surfaced to the stream sender. Check immediately: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} livepeer_cli -host 127.0.0.1 -http 5935 # Select Option 1: Get node status # Check: Deposit and Reserve under BROADCASTER STATS ``` The CLI uses the legacy term "Broadcaster" for Gateway. If either value is zero, top up via Option 11 (Deposit broadcasting funds). See . **2. `-maxPricePerUnit` is set too low.** `-maxPricePerUnit` is the maximum wei per pixel the Gateway pays Orchestrators. If Orchestrators on the network charge more than the cap, none accept jobs. Community standard for testing: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -maxPricePerUnit 300 -pixelsPerUnit 1 ``` Adjust upward in increments if jobs are still not routing after confirming the deposit is funded. **3. No reachable Orchestrators.** If a manual `-orchAddr` list is specified, verify those addresses are active on [Livepeer Explorer](https://explorer.livepeer.org). Remove unreachable addresses and add active ones. **4. Arbitrum RPC is disconnected.** The Gateway requires a working Arbitrum RPC to issue and track payment tickets. See the Arbitrum RPC accordion below. An Arbitrum RPC URL is required for all on-chain Gateway operations. **Test the RPC endpoint:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl -X POST \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' ``` A working endpoint returns a JSON object with a `result` field containing a hex block number. A failed one returns an HTTP error or times out. **Common causes and fixes:** Cause Fix Free-tier rate limit exceeded Upgrade to paid Infura/Alchemy tier, or use a different provider Missing `https://` prefix Add `https://` to the start of the `-ethUrl` value Wrong network (Ethereum mainnet instead of Arbitrum) Use an Arbitrum One endpoint, not Ethereum mainnet Stale/deprecated public RPC Switch to `https://arb1.arbitrum.io/rpc` as fallback **RPC endpoint options:** * Infura: [Infura.io](https://www.infura.io) (free tier available) * Alchemy: [Alchemy.com](https://www.alchemy.com) (free tier available) * Public fallback: `https://arb1.arbitrum.io/rpc` (rate-limited, not for production) * Self-hosted: [Arbitrum full node guide](https://docs.arbitrum.io/node-running/how-tos/running-a-full-node) Update the startup command: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -ethUrl https:// ``` Bridging ETH from Ethereum mainnet to Arbitrum One transfers ETH to the wallet address on Arbitrum, but does **not** automatically allocate it as a Gateway deposit. Deposit and reserve must be set explicitly via `livepeer_cli` after the bridge confirms. ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} livepeer_cli -host 127.0.0.1 -http 5935 # Select Option 11: Deposit broadcasting funds # Enter deposit amount (e.g. 0.07 ETH) # Enter reserve amount (e.g. 0.03 ETH) ``` After the transaction confirms, run Option 1 (Get node status) to verify the deposit and reserve appear in the BROADCASTER STATS section (legacy term for Gateway). If ETH has not arrived in the Arbitrum wallet at all, check the bridge transaction on Arbiscan. Arbitrum bridge deposits are typically confirmed within 15 minutes but can take longer during congestion. ```icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} TicketParams expired ``` This error means the Gateway sent a payment ticket with parameters that had expired by the time the Orchestrator processed it. The expiration window is measured in Ethereum L1 blocks. This is not an operator error - it is expected behaviour when there is a delay between the Gateway requesting Orchestrator info and sending the subsequent segment. The Gateway automatically retries with fresh ticket parameters. **When to investigate further:** * If this error appears *constantly* (not intermittently) across all sessions, the Arbitrum RPC may be returning stale block numbers. Test the RPC with `eth_blockNumber` and compare to Arbiscan's current block. * If retries are consistently failing after the error, check `livepeer_payment_create_errors` in Prometheus. For context on the expiry mechanism, see [go-livepeer issue #1343](https://github.com/livepeer/go-livepeer/issues/1343). Log patterns: ```icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} Transcode loop timed out Segment loop timed out ``` These are not errors - they indicate a session was cleaned up because no segments arrived for an extended period. This is expected when streams end. If these appear during active streams: 1. **Re-publish the stream** - the Gateway selects a new Orchestrator and session automatically. 2. **If it happens repeatedly:** the Orchestrators in the pool may be dropping sessions. Remove underperforming addresses from `-orchAddr` using Explorer to find more reliable alternatives. 3. **If CUDA errors appear in logs** (`CUDA_ERROR_UNKNOWN`, `Cannot allocate memory`) these originate on the Orchestrator side. The Gateway retries with a different Orchestrator automatically. Enable verbose logging for session-level detail: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -v 6 ``` If the Gateway is unexpectedly calling the reward function and spending gas, ensure `-reward=false` is set in the launch command even when using a config file. Config file values can be overridden by command-line flags; always set `reward=false` explicitly on the command line. ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} livepeer -gateway -reward=false [other flags...] ``` ## AI Issues The Livepeer AI Gateway binary is **not currently available for Windows or macOS (Intel)**. The AI inference stack depends on a GPU/CUDA toolchain tested only on Linux. **Workaround: Docker on Linux.** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} docker run \ --name livepeer_ai_gateway \ -v ~/.lpData2/:/root/.lpData2 \ -p 8935:8935 \ --network host \ livepeer/go-livepeer:master \ -datadir /root/.lpData2 \ -gateway \ -orchAddr \ -httpAddr 0.0.0.0:8935 \ -v 6 \ -httpIngest ``` On Windows, install [Docker Desktop](https://www.docker.com/products/docker-desktop/) with WSL2 backend and run the container inside WSL2. The standard video Gateway binary (`livepeer-windows-amd64`) remains available and works normally for video transcoding workloads on Windows. Off-chain AI Gateways route jobs only to Orchestrators specified in `-orchAddr` that are currently online and running the requested pipeline. On-chain AI Gateways using `-aiServiceRegistry` discover Orchestrators automatically via the protocol. **Check the `-orchAddr` list (off-chain):** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Verify the Orchestrator is reachable curl https://:/getOrchestratorInfo ``` If the Orchestrator responds with its capability list and does not include the requested pipeline, it will not accept jobs regardless of connectivity. **Finding AI-capable Orchestrators:** * [Discord #local-Gateways](https://discord.gg/livepeer) - community operators share endpoints * [Livepeer Explorer](https://explorer.livepeer.org) - contact active Orchestrators directly * [Livepeer Forum](https://forum.livepeer.org) - search for community AI Orchestrator lists **Common mistake:** pointing an AI Gateway at video-only Orchestrators. These reject AI jobs silently. Each AI pipeline requires specific models to be loaded on the Orchestrator. Errors on specific pipelines typically indicate: 1. **The pipeline is not supported** by the connected Orchestrators - they may support `text-to-image` but not `image-to-video`. 2. **The model is not loaded** - the Orchestrator supports the pipeline but the specific model variant (e.g. `ByteDance/SDXL-Lightning`) is not warm. Some Orchestrators support cold loading (slower first request). 3. **Model ID mismatch** - the model ID string in the request must exactly match what the Orchestrator has. Case and formatting matter. **Diagnostic approach:** Start with `text-to-image` using a common model against a known-good Orchestrator before testing less common pipelines. There is currently no unified registry or discovery endpoint listing all AI capabilities network-wide. Per-Orchestrator capability query: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl https://:/getOrchestratorInfo ``` Community discussion on a unified discovery mechanism is active in [Discord #local-Gateways](https://discord.gg/livepeer) and the [Livepeer Forum](https://forum.livepeer.org). ## Dual Issues When video transcoding and AI inference run simultaneously, they compete for GPU memory and compute. Signs of contention: * AI model loading failures during active video sessions * Increased transcoding latency correlated with AI request peaks * GPU memory at or near 100% in `/hardware/stats` **Diagnose:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Watch GPU stats every 10 seconds watch -n 10 'curl -s http://localhost:8935/hardware/stats' ``` **Mitigations in order of escalation:** 1. Reduce AI session concurrency with `-maxSessions` 2. Schedule AI-heavy workloads during lower video traffic periods 3. Separate video and AI onto dedicated nodes If GPU memory consistently exceeds 85% under normal load, dedicated nodes are the right path. See . Both video and AI endpoints share the same `-httpAddr` port (default 8935). A Dual Gateway does not open two separate ports - the `-httpIngest` flag enables AI endpoints on the same port used for video. Service Default port Notes HTTP API (video + AI) 8935 Set via `-httpAddr`. `-httpIngest` enables AI endpoints on this port. RTMP ingest 1935 Set via `-rtmpAddr`. Video ingest only. CLI API 5935 Set via `-cliAddr`. Used by livepeer\_cli. If the default port is occupied, check for lingering processes: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} lsof -i :8935 ``` Override the port: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -httpAddr 0.0.0.0:9000 ``` ## Common Errors Another process is already occupying the port. **Find and resolve (Linux/macOS):** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} lsof -i :8935 # Note the PID in the output kill -9 ``` **Find and resolve (Windows):** ```cmd icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} netstat -ano | findstr :8935 taskkill /PID /F ``` **Change the port instead:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -httpAddr 0.0.0.0:9000 ``` Most common cause: a previous `livepeer` process that did not exit cleanly. Check for lingering processes with `ps aux | grep livepeer` (Linux) or Task Manager (Windows) before restarting. The GPU-accelerated binary (`livepeer-linux-gpu-amd64`) was downloaded but CUDA Toolkit v12 is not installed, or is installed at a non-standard path. **Fix 1 - Use the non-GPU binary** (most Gateway operators do not need local GPU transcoding): ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} wget https://github.com/livepeer/go-livepeer/releases/download//livepeer-linux-amd64.tar.gz ``` **Fix 2 - Point to CUDA libraries:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH ./livepeer -gateway ... ``` The GPU binary is only needed if the Gateway performs local transcoding with an NVIDIA GPU. Most operators route transcoding to network Orchestrators. The Gateway cannot reach its Arbitrum RPC. ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl -X POST \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' ``` A working RPC returns a JSON response. Fix the `-ethUrl` flag and restart. See the Arbitrum RPC accordion under Video Issues for full diagnosis. The Gateway cannot find or unlock the keystore file. **Check the keystore path:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} ls ~/.lpData//keystore/ ``` The keystore directory should contain a file with a name matching the ETH account address. **Check the password file for trailing newlines:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} cat -A /path/to/eth-password.txt # A trailing $ followed by nothing is fine; a trailing ^J means a newline that may cause issues ``` **Recreate the password file safely:** ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} echo -n "your-password" > /path/to/eth-password.txt ``` If a different `datadir` or `network` was used in the past, copy the keystore file to the expected location: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} cp ~/.lpData//keystore/* ~/.lpData//keystore/ ``` If the password to the keystore is lost, it cannot be recovered. Create a new wallet and fund the new address. The container is crashing on startup. Always check logs first: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} docker logs --tail 50 ``` **Common causes:** 1. **ETH password file not mounted or wrong path inside container:** The volume mount must include the password file, and the path in `-ethPassword` must be the path *inside the container*, not on the host. ```yaml icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} volumes: - /host/path/to/password.txt:/run/secrets/eth-password.txt command: "-gateway -ethPassword /run/secrets/eth-password.txt ..." ``` 2. **`-cliAddr` using `localhost` instead of the container hostname:** ```yaml icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} command: "-cliAddr gateway:5935 ..." ``` 3. **Deprecated `version:` field in docker-compose.yml:** Remove the `version: '3.9'` line at the top of the compose file - it is deprecated in current Docker Compose versions and can cause unexpected behaviour on some systems. 4. **Missing required flags:** On-chain Gateways require at minimum: `-network`, `-ethUrl`, `-ethKeystorePath`, `-ethPassword`. This error on startup with the `-nvidia` flag means GPU encoding/decoding session limits were hit during startup tests. Different NVIDIA GPUs have different limits. This is a hardware constraint. Options: * Check the GPU's session limits in the [NVIDIA encode/decode support matrix](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new) * Reduce concurrent sessions via `-maxSessions` * Upgrade to a GPU with higher session limits for production workloads A stream is being pushed with a pixel format that go-livepeer cannot transcode. This is not an operator-actionable error - the stream source needs to encode in a supported format (typically `yuv420p`). If the stream is operator-controlled, add `-pix_fmt yuv420p` to the FFmpeg command. ## Log Interpretation Enable verbose logging when diagnosing issues: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} -v 6 ``` Save logs to file while also displaying in terminal: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} ./livepeer -gateway [flags] 2>&1 | tee livepeer.log ``` ### Severity guide Pattern Severity Action `INFO` messages with normal activity Normal None `Transcode loop timed out` Normal None - expected when streams end `Segment loop timed out` Normal None - expected when streams end `TicketParams expired` Info None - Gateway retries automatically `MB rate > Level limit` Warning None - does not impact transcoding `Could not connect to Ethereum node` Error Fix RPC immediately `TicketParams expired` (constant) Warning Investigate RPC block sync `CUDA_ERROR_UNKNOWN` Error (Orchestrator-side) Gateway retries; investigate if persistent `payment_create_errors` increasing Error Investigate deposit balance + RPC ## Escalation If none of the above resolves the issue, include the following in a support request: * Gateway node type (Video / AI / Dual) * go-livepeer version (`./livepeer --version`) * Operating system and architecture * Exact error message (copy from logs) * First 50 lines of startup logs (`-v 6`) * Output of `livepeer_cli` Option 1 (Get node status) - redact the ETH private key path Primary Gateway operator channel. Most operators respond within a few hours. Search existing threads before posting. Tag with `gateway` and `go-livepeer`. If an error is not covered here, open a docs issue with Gateway type, version, OS, and error. ## Orchestrator troubleshooting Source: [/v2/Orchestrators/guides/monitoring-and-tooling/troubleshooting](/v2/Orchestrators/guides/monitoring-and-tooling/troubleshooting) Errors are grouped by category. Use the index below to jump straight to what you are seeing, or read through the category that matches your situation. **Symptom quick-nav** * [Transcoding errors](#transcoding-errors) * [GPU and memory errors](#GPU-and-memory-errors) * [Reward and gas errors](#reward-and-gas-errors) * [AI runner errors](#ai-runner-errors) * [Networking and connectivity](#networking-and-connectivity) * [Account and keystore errors](#account-and-keystore-errors) * [General diagnostics](#general-diagnostics) Common external checks used throughout this page: * H.264 levels reference: [Wikipedia AVC levels](https://en.wikipedia.org/wiki/Advanced_Video_Coding#Levels) * NVIDIA GPU support matrix: [NVIDIA video encode/decode support matrix](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new) * Arbiscan: [Arbiscan.io](https://arbiscan.io) * Explorer Orchestrator list: [explorer.livepeer.org/Orchestrators](https://explorer.livepeer.org/orchestrators) * Cloud SPE AI registry: [tools.Livepeer.cloud](https://tools.livepeer.cloud) * Ollama version endpoint: `http://localhost:11434/api/version` * Example local and public service URIs: `https://127.0.0.1:4433`, `https://121.5.10.8:8935`, `https://121.5.10.8:8935;`, `https://:8935/status`, `https://orch.yourdomain.com:8935` ## Transcoding errors **What it means:** Your Orchestrator has reached its session limit and is rejecting new work from Gateways. **Where you see it:** Gateway logs (not your Orchestrator logs) report this error when they try to send you a job. **How to fix it:** 1. Check your current session count against your configured limit: ```bash icon="terminal" title="Check current session limit" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl http://localhost:7935/status | jq '.SessionLimit' ``` 2. If you have spare GPU capacity, increase `-maxSessions` in your launch command 3. If you are already at GPU VRAM limits, you cannot safely increase sessions – you need to reduce the model size, reduce output dimensions, or add GPU capacity 4. In a split setup, verify both the transcoder machine and the Orchestrator have capacity headroom See [Session Limits](/v2/Orchestrators/guides/config-and-optimisation/capacity-planning) for the full calculation methodology. **What it means:** A session that was previously processing a stream was cleaned up because no new segments arrived for a while. **This is normal.** These messages appear when a live stream ends or pauses. They are not errors and do not indicate a problem with your node. You can safely ignore them. **What it means:** The source video segment being transcoded has a bitrate that exceeds the H.264 level limit for its resolution. See the [H.264 levels reference](https://en.wikipedia.org/wiki/Advanced_Video_Coding#Levels) for technical detail. **In practice:** Transcoding usually completes and returns results to the Gateway despite this warning. When these warnings rise alongside failed transcodes, inspect the source segment properties. The fault sits with the Gateway or broadcaster input, not with your Orchestrator. **What it means:** A source video segment has properties that prevent it from being processed – unsupported codec, unusual encoding parameters, or a corrupt segment. **Your action:** None required. This is a Gateway or broadcaster responsibility. The Gateway is sending segments that the Livepeer Network cannot process. Your node correctly rejects them. If you are seeing a large volume of these errors from one Gateway, consider flagging it in the community Discord. **What it means:** The source video uses a pixel format that go-livepeer cannot transcode. Go-livepeer requires YUV 4:2:0 (planar or interleaved) input format. Any other pixel format returns this error. **Your action:** None. The broadcaster submitted an unsupported format. There is nothing an Orchestrator can do to transcode an unsupported pixel format. ## GPU and memory errors **What it means:** go-livepeer runs a GPU test on startup to verify it can encode and decode using your NVIDIA GPU. This error means the test failed because your GPU has already reached its maximum concurrent NVENC/NVDEC session count. **Consumer NVIDIA GPU session limits:** Most consumer NVIDIA GPUs (GTX/RTX series) have a hardware-enforced limit of 3–8 concurrent NVENC sessions per GPU. If other processes have those sessions open, go-livepeer's startup test cannot allocate one and fails. **How to fix it:** 1. Check what is using NVENC sessions on the GPU: `nvidia-smi` 2. Stop any processes consuming NVENC sessions (video encoding software, other Livepeer processes) 3. If you need more concurrent sessions than your consumer GPU allows, look into driver patching (the `nvenc-patch` approach) or upgrade to a data centre class GPU (RTX A-series or above) which has no session cap **Reference:** The [NVIDIA Video Encode and Decode GPU Support Matrix](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new) shows session limits by GPU model. **What it means:** Your GPU ran out of VRAM while trying to load or run a model. **How to fix it:** 1. Check current VRAM usage: `nvidia-smi --query-gpu=memory.used,memory.free --format=csv` 2. Reduce the `capacity` value in your `aiModels.json` for the affected pipeline 3. If you have multiple warm models loaded, consider making some cold (remove from warm list) to free VRAM for others 4. Check whether you have set dimensions in your inference requests that exceed what your VRAM can handle See [Model VRAM Reference](/v2/Orchestrators/guides/operator-considerations/requirements#batch-ai-inference) for per-pipeline minimums. **What it means:** go-livepeer is falling back to CPU transcoding even though you specified `-nvidia`. **Checklist:** * Run `nvidia-smi` – confirm the GPU is visible to the OS * Check go-livepeer startup logs for a GPU detection line * Verify `LD_LIBRARY_PATH` includes CUDA shared libraries if not installed to `/usr/local/cuda` * Confirm NVIDIA Container Toolkit is installed if you are running in Docker: `docker run --gpus all nvidia/cuda:11.0-base nvidia-smi` From `go-livepeer/doc/gpu.md`: if the CUDA location differs from `/usr/local/cuda`, set `LD_LIBRARY_PATH=` when launching. ## Reward and gas errors **What it means:** Your node tried to submit a transaction (reward call, ticket redemption, or other on-chain action) but your ETH wallet does not have enough ETH to cover gas. **How to fix it:** 1. Check your Orchestrator wallet ETH balance on [Arbiscan](https://arbiscan.io) – bridge or transfer ETH to it on Arbitrum One 2. As a preventive measure, keep at least 0.02–0.05 ETH in your Orchestrator wallet at all times 3. If you are using OrchestratorSiphon, configure `eth_warn` and `eth_minval` thresholds to receive warnings before the wallet goes dry Arbitrum gas is very cheap – reward calls cost approximately $0.01–$0.12 each. Wallet depletion typically happens either from a price spike in ETH or from high-volume ticket redemptions. **What it means:** You set `-reward=false` but your node is still submitting reward transactions and spending gas. **Why this happens:** If you are running the Orchestrator and transcoder as separate processes (split setup), `-reward=false` must be set in **every** launch command. A transcoder process running with the same Ethereum wallet and a separate config may be calling reward independently. **How to fix it:** 1. Audit all running `livepeer` processes: `ps aux | grep livepeer` 2. Add `-reward=false` to every launch command 3. As an extra precaution, remove the `-ethUrl` option from any transcoder process that shares the same wallet. Without an ETH URL, the transcoder cannot submit on-chain transactions at all. 4. When using a `.conf` file for configuration, the command-line flag overrides the file. Always pass `-reward=false` explicitly at launch. **What it means:** Your Orchestrator is running and you have not set `-reward=false`, but the Explorer shows missed rounds. **Diagnose in order:** 1. **ETH balance** – low balance causes reward calls to fail silently. Check `http://localhost:7935/status` for ETH balance or look at Arbiscan. 2. **Node was offline** – a node down at the round boundary (\~every 22 hours on Arbitrum) misses the call. Check your systemd or service uptime logs. 3. **Multiple processes competing** – two go-livepeer processes sharing the same wallet can submit a failing duplicate. For persistent missed rounds, consider the [Siphon split setup](/v2/Orchestrators/guides/deployment-details/siphon-setup) which runs reward calling on a dedicated stable machine independently of your GPU workload. **What it means:** A Gateway sent a payment ticket with parameters that have expired by the time your node processed it. **Cause:** A delay between when the Gateway retrieved your Orchestrator info and when it sent the segment, or a delay in your node polling L1 blocks for expiry validation. **Your action:** None required. The Gateway will automatically retry the request with fresh ticket parameters. This error is transient and self-resolving. If you are seeing a very high rate of TicketParams expired errors from one Gateway, it may indicate that Gateway has an unusually slow L1 block polling rate. ## AI Runner errors **Common causes and fixes:** Cause How to confirm Fix Wrong Docker image tag `docker logs livepeer-ai-runner` shows image pull error Use the correct versioned tag from `aiModels.json` documentation VRAM insufficient `docker logs livepeer-ai-runner` shows CUDA OOM at startup Reduce loaded models, free VRAM, or check minimum requirements NVIDIA Container Toolkit missing `docker run --gpus all nvidia/cuda:11.0-base nvidia-smi` fails Install `nvidia-container-toolkit` – see [Hardware Reference](/v2/Orchestrators/guides/operator-considerations/requirements#driver-requirements) Docker not in GPU mode Container starts but no GPU visible inside Pass `--gpus all` or `--gpus device=` in Docker run **What it means:** The model ID in your `aiModels.json` does not resolve to a valid model. Model IDs are **case-sensitive** and must include the organisation prefix. For example: * ✓ `stabilityai/stable-diffusion-xl-base-1.0` * ✗ `stable-diffusion-xl-base-1.0` (missing org prefix) * ✗ `StabilityAI/stable-diffusion-xl-base-1.0` (wrong case) Ollama-based LLM models use a different format – do not use Ollama model tags (`llama3:8b`) in `aiModels.json`. Use the HuggingFace model ID format instead. **Diagnose in order:** 1. **Outside the Active Set** – check [Livepeer Explorer](https://explorer.livepeer.org/orchestrators). Nodes outside the top 100 by stake receive no transcoding or AI jobs. 2. **Price too high** – your per-capability price exceeds the Gateway's `-maxPricePerCapability` limit. Check market rates on [tools.Livepeer.cloud](https://tools.livepeer.cloud) and compare with other operators. 3. **Model is cold** – the warm model is not loaded in VRAM. Jobs may time out before the model loads, and cold starts often take 30 to 120 seconds. Make sure your intended warm model is listed in the `warm` section of `aiModels.json`. 4. **Capability not registered** – query your node's registered capabilities: `curl http://localhost:7935/getNetworkCapabilities | jq`. If the pipeline is missing, check your `aiModels.json` configuration and that the AI Runner started successfully. **What it means:** The inference job started (model was loaded), but ran out of VRAM during processing – typically because output dimensions (resolution, frame count) exceed what your VRAM can hold during the forward pass. **How to fix it:** 1. Reduce the `capacity` value for the pipeline in `aiModels.json` 2. Reduce `maxSessions` for AI inference specifically 3. If the OOM happens for large output requests but not small ones, consider whether you need to restrict accepted request dimensions at the Gateway level **Checklist:** 1. Model weights are downloaded and accessible at the path configured in your ComfyStream setup 2. CUDA toolkit version matches what the container expects – check `nvidia-smi` for your driver version and confirm container CUDA compatibility 3. The container has sufficient VRAM for the workflow – live-video streaming requires the model to remain resident during the stream 4. Check container logs for the specific Python exception: `docker logs -f ` See [Live-Video AI Setup](/v2/Orchestrators/guides/ai-and-job-workloads/realtime-ai-setup#troubleshooting) for ComfyStream-specific troubleshooting. **Checklist:** 1. Verify Ollama container is running and accessible: `curl http://localhost:11434/api/version` 2. Confirm go-livepeer can reach the Ollama endpoint. Containerised deployments usually need both services on the same Docker network. 3. Re-register the LLM capability: restart go-livepeer to force capability re-advertisement 4. Check that the model ID in `aiModels.json` matches an installed Ollama model: `ollama list` ## Networking and connectivity **Error:** ```text icon="terminal" title="Service address mismatch warning" theme={"theme":{"light":"github-light","dark":"dark-plus"}} Service address https://127.0.0.1:4433 did not match discovered address https://121.5.10.8:8935; set the correct address in livepeer_cli or use -serviceAddr ``` **What it means:** On startup, go-livepeer checks whether your current public IP matches the service URI stored on-chain. They do not match, which may mean Gateways cannot reach you. **How to fix it:** 1. If your IP changed: update your on-chain service URI using `livepeer_cli` 2. If your IP is correct but different from what the node auto-detects: override with `-serviceAddr :8935` 3. Confirm your node is actually reachable at that address: `curl -v https://:8935/status` **Full diagnostic checklist:** ```bash icon="terminal" title="Job receipt diagnostic checklist" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # 1. Check your service URI is reachable from outside curl -v https://:8935/status # 2. Check your current price (must be below gateway max) curl http://localhost:7935/status | jq '.PricePerUnit' # 3. Check capabilities are registered curl http://localhost:7935/getNetworkCapabilities | jq # 4. Verify on-chain service URI matches your running node # Check Explorer: explorer.livepeer.org/accounts/

/orchestrating ``` If step 1 times out or refuses connection: port 8935 is not reachable from the internet. Check your firewall rules and, if behind a NAT, configure port forwarding. **The situation:** Gateways reach Orchestrators via the public IP registered on-chain. If your machine is behind a NAT, the public IP points to your router, not directly to your node. **Options:** * **Port forwarding** – forward port 8935 on your router to your node's local IP. This is the standard approach. * **DMZ** – place your node in the router's DMZ to receive all unsolicited inbound traffic. Less secure but simpler. * **Hairpinning (if needed)** – some networks require iptables rules to handle internal-to-external traffic loops: ```bash icon="terminal" title="Example hairpinning rule" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Allow internal traffic to reach the node via the external IP iptables -t nat -A POSTROUTING -p tcp -s 10.0.0.10 -d 10.0.0.10 -j SNAT --to-source ``` Running an Orchestrator from a home connection exposes a publicly accessible port on a residential network. Ensure you understand the security implications. A VPS or dedicated server is strongly recommended for production operation. Your service URI is the address Gateways use to connect to your node. It must be publicly accessible on port 8935. * **IP address:** `https://121.5.10.8:8935` – static IPs are preferred for consistency * **DNS name:** `https://orch.yourdomain.com:8935` – allowed and recommended if you want flexibility to change the underlying IP without an on-chain transaction The URI is stored on-chain via the Livepeer Protocol. You register it during setup, and it stays until you update it via `livepeer_cli`. Use DNS if you anticipate IP changes. ## Account and keystore errors **What it means:** go-livepeer could not find or load your Ethereum account. This usually means the keystore file is in the wrong location, has incorrect permissions, or the wrong network is specified. **Keystore default location:** ```icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} ~/.lpData//keystore/ ``` For Arbitrum mainnet: `~/.lpData/arbitrum-one-mainnet/keystore/` **How to fix it:** 1. List files in your keystore directory: `ls -la ~/.lpData//keystore/` 2. Confirm the file matching your `-ethAcctAddr` is present (UTC-- prefixed JSON file) 3. Check file permissions – the keystore file should be readable by the user running go-livepeer: `chmod 600 ` 4. If you used a different `-datadir` in the past, the keystore may be under a different path. Locate it and copy it to the correct location. Never copy a keystore file over an unencrypted connection. Use `scp` with SSH key authentication or another encrypted transfer method. ## General diagnostics ### How to confirm your node is receiving work ```bash icon="terminal" title="General diagnostic commands" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Check current session count curl http://localhost:7935/status | jq # Enable verbose logging for transcoding activity livepeer -orchestrator -transcoder -v 6 ... # Watch logs in real time journalctl -u livepeer -f # or if using tee: tail -f /var/log/livepeer/livepeer.log ``` ### How to capture logs to a file ```bash icon="terminal" title="Capture logs with tee" theme={"theme":{"light":"github-light","dark":"dark-plus"}} livepeer \ -orchestrator \ -transcoder \ ... \ 2>&1 | tee /var/log/livepeer/livepeer.log ``` This pipes both stdout and stderr to both the terminal and the log file simultaneously. ### Checking node status via CLI port ```bash icon="terminal" title="Check node status and metrics" theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Node status curl http://localhost:7935/status | jq # Registered capabilities curl http://localhost:7935/getNetworkCapabilities | jq # Prometheus metrics curl http://localhost:7935/metrics ``` ## Escalation paths The #Orchestrators and #support channels have active community members. Search before posting – most errors have been seen before. Search open and closed issues for your error message. Many edge cases are documented here. Set up Prometheus to see what is happening inside your node before errors surface. Read your on-chain state – stake, reward calls, earnings – to spot issues from the outside.