> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat Completions

> OpenAI-compatible chat completions that answer from your Moss indexes.

`POST /v1/chat/completions` searches your project's indexes for the user's question, adds what it finds to the prompt, and returns the reply in OpenAI's shape. Any OpenAI client works: change the base URL and the key.

> **Base URL**
>
> ```text theme={null}
> https://inference.moss.dev/v1
> ```

## Authentication

Send the project id and project key from the [Moss portal](https://portal.usemoss.dev) as one bearer token:

```text theme={null}
Authorization: Bearer <project_id>:<project_key>
```

## Request

The body is an OpenAI [chat completion request](https://platform.openai.com/docs/api-reference/chat/create).

| Field | Required | Notes |
| - | - | - |
| `model` | no | `moss` (the default), or a vendor model with your own key. See [Models](#models). |
| `messages` | yes | OpenAI message list. System messages are supported. |
| `stream` | no | `true` streams OpenAI `chat.completion.chunk` events, ending with `data: [DONE]`. See [Streaming](#streaming). Omitted returns one `chat.completion`. |
| `temperature`, `max_tokens`, `tools`, `response_format`, ... | no | Passed to the model. A reply without `stream` has one choice and no `logprobs`. |

| Header | Default | What it does |
| - | - | - |
| `X-Moss-Indexes` | every index of the project | Comma-separated index names to search, for example `faq,policies`. Required when the project has more than 32 indexes. An index name outside this project returns 403. |
| `X-Moss-Retrieve` | `on` | `off` skips the search. |
| `X-Moss-Cache` | `on` | `off` skips the reply cache. A repeated question otherwise returns the cached reply with `usage.prompt_tokens` of 0. |
| `X-Moss-Provider-Key` | none | Your OpenAI, Anthropic, or Google API key, for a vendor `model`. |

`usage.prompt_tokens` includes the retrieved facts.

A reply that reached the model carries `X-Moss-Request-Id`: include it when you report a problem. The `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and `x-ratelimit-reset-*` headers, for `requests` and, on `moss`, `tokens`, show how much of the project's [limits](#limits) is left.

## Streaming

The last chunk before `data: [DONE]` has empty `choices` and the whole reply's `usage`; no other chunk carries it. `{"include_usage": false}` in `stream_options` leaves it out. Check `choices` before reading `choices[0]`, as the examples below do.

```text theme={null}
data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "Yes, we ship"}, "finish_reason": null}]}

data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [{"index": 0, "delta": {"content": " to Canada."}, "finish_reason": "stop"}]}

data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [], "usage": {"prompt_tokens": 165, "completion_tokens": 7, "total_tokens": 172}}

data: [DONE]
```

## Examples

The Python and JavaScript examples below use OpenAI's own clients. Every Moss SDK also has its own client for this endpoint: [Python](https://docs.moss.dev/docs/reference/python/inference.md), [JavaScript](https://docs.moss.dev/docs/reference/js/inference.md), [Swift](https://docs.moss.dev/docs/reference/swift/inference.md), [Android](https://docs.moss.dev/docs/reference/android/inference.md), [Elixir](https://docs.moss.dev/docs/reference/elixir/inference.md), [Browser](https://docs.moss.dev/docs/reference/browser/inference.md), and [C](https://docs.moss.dev/docs/reference/c/inference.md).

```bash curl theme={null}
curl -N https://inference.moss.dev/v1/chat/completions \
  -H "Authorization: Bearer $MOSS_PROJECT_ID:$MOSS_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Moss-Indexes: faq" \
  -d '{
    "model": "moss",
    "stream": true,
    "messages": [
      { "role": "system", "content": "You are the support agent for Acme." },
      { "role": "user", "content": "Do you ship to Canada?" }
    ]
  }'
```

```python Python theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.moss.dev/v1",
    api_key=f"{os.environ['MOSS_PROJECT_ID']}:{os.environ['MOSS_PROJECT_KEY']}",
    default_headers={"X-Moss-Indexes": "faq"},
)
stream = client.chat.completions.create(
    model="moss",
    messages=[
        {"role": "system", "content": "You are the support agent for Acme."},
        {"role": "user", "content": "Do you ship to Canada?"},
    ],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
```

```typescript JavaScript theme={null}
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://inference.moss.dev/v1",
  apiKey: `${process.env.MOSS_PROJECT_ID}:${process.env.MOSS_PROJECT_KEY}`,
  defaultHeaders: { "X-Moss-Indexes": "faq" },
});
const stream = await client.chat.completions.create({
  model: "moss",
  messages: [
    { role: "system", content: "You are the support agent for Acme." },
    { role: "user", content: "Do you ship to Canada?" },
  ],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}
```

## Models

| `model` | Where it runs | Key |
| - | - | - |
| `moss`, or omitted | The Moss model on Moss GPUs. | Project key only. |
| `gpt-*`, `o*`, `chatgpt-*` | OpenAI | `X-Moss-Provider-Key` |
| `claude-*` | Anthropic | `X-Moss-Provider-Key` |
| `gemini-*` | Google | `X-Moss-Provider-Key` |

Retrieval works the same for every model. A vendor model without `X-Moss-Provider-Key`, or any other model name, returns `400`. The provider key is sent only to that vendor and is never stored.

## Limits

Per project:

| Limit | `moss` | Vendor model |
| - | - | - |
| Requests per minute (sustained / burst) | 120 / 10 | 300 / 50 |
| Input tokens per minute | 600,000 | none |
| Prompt tokens per request | 16,384 | 200,000 |
| Indexes per request | 32 | 32 |
| Request body | 256 KB | 256 KB |

On `moss` the 16,384 tokens are the model's whole context: the prompt, the retrieved facts and the reply share them, so leave room for `max_tokens`. A prompt over it is `400` with `type` `invalid_request_error` and `code: "context_length_exceeded"`.

Over a per-minute limit the response is `429` with `type` `requests` or `tokens` and `code: "rate_limit_exceeded"`. When Moss is at capacity it is `429` with `type` `capacity` and `code: "server_overloaded"`. Both carry `Retry-After`, and the OpenAI SDKs retry on their own. A paused project gets `429` with `code: "usage_limit_exceeded"` and stays refused until it is unpaused in the portal.

```json 429 - Too Many Requests theme={null}
{
  "error": {
    "message": "Rate limit reached for project 1a2b3c: too many requests per minute. Please try again in 1s.",
    "type": "requests",
    "code": "rate_limit_exceeded"
  }
}
```

## Errors

Errors use OpenAI's `{"error": {"message", "type", "code"}}` shape.

| Status | `type` | Cause |
| - | - | - |
| `400` | `invalid_request_error` | Malformed body, unknown model, vendor model without a key, more than 32 indexes. A prompt over the token limit has `code: "context_length_exceeded"`. |
| `400` | `model_error` | The model rejected the prompt. |
| `401` | `authentication_error` | Missing or wrong `Authorization` token. |
| `403` | `permission_error` | `X-Moss-Indexes` names an index outside this project. |
| `413` | `invalid_request_error` | Body over 256 KB. |
| `429` | `requests`, `tokens`, `capacity`, `usage_limit` | See [Limits](#limits). |
| vendor `4xx` | `model_error` | The vendor's own status, with its message after `The model rejected the prompt`. A bad key is `401` from OpenAI and Anthropic, and `400` from Google. |
| `502` | `model_error` | The model failed, or a vendor returned `5xx`. Retry with backoff. |
| `503` | `server_error` | Moss could not check the project. Retry with backoff. |

A failure after a stream has started arrives as a final `data:` frame carrying `error` instead of `choices`, then `data: [DONE]`.

## Tool calls

`tools`, `tool_choice` and `parallel_tool_calls` take OpenAI's shapes and go to the model. Your code runs the tools. A `tools` value that is not a list of objects returns `400`.

A streamed reply sends each call in pieces in `delta.tool_calls`, with `index`, `id`, `function.name` and `function.arguments`, then a chunk with `finish_reason` `tool_calls`. A reply without `stream` has the calls joined in `choices[0].message.tool_calls`, with the same `finish_reason`.

Send the assistant message back unchanged, then one `tool` message per call, and call again:

```json theme={null}
{
  "messages": [
    {"role": "user", "content": "Where is order A1?"},
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": {"name": "lookup_order", "arguments": "{\"id\": \"A1\"}"}
      }]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "Shipped on Oct 20."}
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "lookup_order",
      "parameters": {"type": "object", "properties": {"id": {"type": "string"}}}
    }
  }]
}
```

A tool call can carry fields beyond OpenAI's, such as Gemini's `extra_content`. Keep them: the model needs them on the next turn.

## MCP servers

A `tools` entry of type `mcp` names an MCP server. Moss Inference connects to it, offers its tools to the model, runs the calls the model makes, and streams the final answer. One request can hold your own tools and MCP servers together.

```json theme={null}
{
  "type": "mcp",
  "server_label": "crm",
  "server_url": "https://mcp.example.com/mcp",
  "headers": {"Authorization": "Bearer CRM_TOKEN"},
  "allowed_tools": ["lookup_order"]
}
```

| Field | Required | Notes |
| - | - | - |
| `type` | yes | `mcp` |
| `server_label` | yes | A name for the server, different for each server in the request. |
| `server_url` | yes | A public `https` URL on port 443. Any other URL returns `400`. |
| `headers` | no | Headers sent to that server only, such as `Authorization`. They never reach the model. A public server, such as `https://mcp.deepwiki.com/mcp`, needs none. |
| `allowed_tools` | no | Tool names. The model sees only these tools of the server. |
| `transport` | no | `streamable_http`, the default, or `sse` for an older server. |

MCP calls are not added to the message history. Keep what you need from the [tool steps](#tool-steps) in your own messages. A request with MCP servers can run for a few minutes.

### Tool steps

A stream carries one extra chunk before each MCP call and one after it, with an empty `delta` and a `moss_tool_step`:

```text theme={null}
data: {"object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "finish_reason": null}], "moss_tool_step": {"type": "call", "id": "call_1", "server_label": "crm", "name": "lookup_order", "arguments": "{\"id\": \"A1\"}"}}

data: {"object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "finish_reason": null}], "moss_tool_step": {"type": "result", "id": "call_1", "server_label": "crm", "name": "lookup_order", "is_error": false, "duration_ms": 41.2, "output": "Shipped on Oct 20."}}
```

`type` is `call` before the tool runs, with its `arguments`, and `result` after it, with `is_error`, `duration_ms` and `output`. Both have the same `id`, the `server_label` and the tool's `name`. A reply without `stream` lists the steps in order in a top-level `moss_tool_steps` array. OpenAI clients ignore both fields.

### MCP errors

| Status | `type` | `code` | Cause |
| - | - | - | - |
| `400` | `invalid_request_error` | none | A bad `tools` entry, a `server_label` or tool name used twice, or too many servers or tools. |
| `400` | `mcp_error` | `mcp_server_error` | `server_url` is not a public `https` URL on port 443. |
| `424` | `mcp_error` | `mcp_server_error` | The server did not connect, refused the credentials, or failed to list its tools. |
| `429` | `capacity` | `mcp_loops_busy` | Too many MCP requests at once. Retry after `Retry-After`. |

A tool may already have changed data when a call fails after it, so Moss Inference never retries that call. Without `stream`, it returns `424` with `code` `failed_after_tool_call`, the `moss_tool_steps` that ran, and the header `x-should-retry: false`. On a stream, it is an `error` event after a step, which the Moss SDKs report with the code `failed_after_tool_call`. Do not replay such a call.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.