> ## Documentation Index
> Fetch the complete documentation index at: https://docs.livepeer.org/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM

> Generate text using a language model.

<Note>
  The LLM pipeline is OpenAI API-compatible but does **not** implement all
  features of the OpenAI API.
</Note>

<Info>
  The default Gateway used in this guide is the public
  [Livepeer.cloud](https://www.livepeer.cloud/) Gateway. It is free to use but
  not intended for production-ready applications. For production-ready
  applications, consider using the [Livepeer Studio](https://livepeer.studio/)
  Gateway, which requires an API token. Alternatively, you can set up your own
  Gateway node or partner with one via the `ai-video` channel on
  [Discord](https://discord.gg/livepeer).
</Info>

### Streaming Responses

<Note>
  Ensure your client supports SSE and processes each `data:` line as it arrives.
</Note>

By default, the `/llm` endpoint returns a single JSON response in the OpenAI
[chat/completions](https://platform.openai.com/docs/api-reference/chat/object)
format, as shown in the sidebar.

To receive responses token-by-token, set `"stream": true` in the request body.
The server will then use **Server-Sent Events (SSE)** to stream output in real
time.

Each streamed chunk will look like:

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
data: {
  "choices": [
    {
      "delta": {
        "content": "...token...",
        "role": "assistant"
      },
      "finish_reason": null
    }
  ]
}
```

The final chunk will have empty content and `"finish_reason": "stop"`:

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
data: {
  "choices": [
    {
      "delta": {
        "content": "",
        "role": "assistant"
      },
      "finish_reason": "stop"
    }
  ]
}
```


## OpenAPI

````yaml post /llm
openapi: 3.1.0
info:
  title: Livepeer AI Runner
  description: An application to run AI pipelines
  version: 0.0.0
servers:
  - url: https://dream-gateway.livepeer.cloud
    description: Livepeer Cloud Community Gateway
  - url: https://livepeer.studio/api/beta/generate
    description: Livepeer Studio Gateway
security: []
paths:
  /llm:
    post:
      tags:
        - generate
      summary: LLM
      description: Generate text using a language model.
      operationId: genLLM
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/LLMRequest'
        required: true
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/LLMResponse'
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPError'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPError'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
        '500':
          description: Internal Server Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPError'
      security:
        - HTTPBearer: []
components:
  schemas:
    LLMRequest:
      properties:
        messages:
          items:
            $ref: '#/components/schemas/LLMMessage'
          type: array
          title: Messages
        model:
          type: string
          title: Model
          default: ''
        temperature:
          type: number
          title: Temperature
          default: 0.7
        max_tokens:
          type: integer
          title: Max Tokens
          default: 256
        top_p:
          type: number
          title: Top P
          default: 1
        top_k:
          type: integer
          title: Top K
          default: -1
        stream:
          type: boolean
          title: Stream
          default: false
      type: object
      required:
        - messages
      title: LLMRequest
    LLMResponse:
      properties:
        id:
          type: string
          title: Id
        model:
          type: string
          title: Model
        created:
          type: integer
          title: Created
        usage:
          $ref: '#/components/schemas/LLMTokenUsage'
        choices:
          items:
            $ref: '#/components/schemas/LLMChoice'
          type: array
          title: Choices
      type: object
      required:
        - id
        - model
        - created
        - usage
        - choices
      title: LLMResponse
    HTTPError:
      properties:
        detail:
          allOf:
            - $ref: '#/components/schemas/APIError'
          description: Detailed error information.
      type: object
      required:
        - detail
      title: HTTPError
      description: HTTP error response model.
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    LLMMessage:
      properties:
        role:
          type: string
          title: Role
        content:
          type: string
          title: Content
      type: object
      required:
        - role
        - content
      title: LLMMessage
    LLMTokenUsage:
      properties:
        prompt_tokens:
          type: integer
          title: Prompt Tokens
        completion_tokens:
          type: integer
          title: Completion Tokens
        total_tokens:
          type: integer
          title: Total Tokens
      type: object
      required:
        - prompt_tokens
        - completion_tokens
        - total_tokens
      title: LLMTokenUsage
    LLMChoice:
      properties:
        index:
          type: integer
          title: Index
        finish_reason:
          type: string
          title: Finish Reason
          default: ''
        delta:
          allOf:
            - $ref: '#/components/schemas/LLMMessage'
        message:
          allOf:
            - $ref: '#/components/schemas/LLMMessage'
      type: object
      required:
        - index
      title: LLMChoice
    APIError:
      properties:
        msg:
          type: string
          title: Msg
          description: The error message.
      type: object
      required:
        - msg
      title: APIError
      description: API error response model.
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
  securitySchemes:
    HTTPBearer:
      type: http
      scheme: bearer

````