> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions that answer from your Moss indexes, with OpenAI's arguments and replies.

`MossInference` is the Python client for
[Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), a drop-in for OpenAI's chat
completions. Its `chat.completions.create` takes OpenAI's arguments, searches your indexes in the
cloud, adds what it finds to the prompt, and returns the reply with OpenAI's field names.

> [!NOTE]
>
> Requires `moss` 1.21.0 or later, installed as in [Install](https://docs.moss.dev/docs/reference/python/api.md#install). The API is async only.

## Create a client

```python theme={null}
import os
from moss import MossInference

client = MossInference(os.environ["MOSS_PROJECT_ID"], os.environ["MOSS_PROJECT_KEY"])
```

The project id and key come from the [Moss portal](https://portal.usemoss.dev). Build one client
and reuse it for every call. `await client.close()`, or the end of an `async with client:` block,
closes its connections. With `MOSS_PROJECT_ID` and `MOSS_PROJECT_KEY` set,
`from moss import inference` gives the same `inference.chat.completions.create` with no client
to build.

## Get a reply

```python theme={null}
completion = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Do you ship to Canada?"}],
    indexes=["faq", "policies"],
)
print(completion.choices[0].message.content)
```

`completion.request_id` is the response's `X-Moss-Request-Id`, and so is
`stream.request_id` on a stream. Include it when you report a problem. Both are `None` when
the response has none.

## Stream a reply

```python theme={null}
stream = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Do you ship to Canada?"}],
    indexes=["faq", "policies"],
    stream=True,
)
async with stream:
    async for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
```

To stop early, `break` out of the loop: leaving the `async with stream:` block closes the
connection, which stops the model.

Only the last chunk carries `usage`, and it has no choices, which is why the loop checks `chunk.choices`.

## Errors

A failed call raises `moss.inference.InferenceError`, from the `async for` loop once a stream
has started. `status` is the HTTP status of a refused request, `type` names the failure, and
`message`, also `str(error)`, describes it. `code` is the error code Moss Inference sent, such as
`"rate_limit_exceeded"`, `request_id` is the response's `X-Moss-Request-Id`, and `retry_after` is
the wait in seconds that a `Retry-After` header asks for. A field the failure lacks is `None`.

The SDK sets these types itself, with `status` `None`:

| `type` | Cause |
| - | - |
| `invalid_request_error` | An argument the SDK rejects before sending, such as an empty key, or an index name that is empty or holds a comma. |
| `connection_error` | Moss Inference could not be reached, or the connection broke. |
| `timeout` | A time limit ran out, such as `timeout` or `idle_timeout`. |
| `incomplete_stream` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

Pass OpenAI function tools in `tools`. When the reply calls tools, append its `message` to
`messages`, then one `tool` message per call, and call again, as in
[Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls):

```python theme={null}
import json

tools = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "parameters": {"type": "object", "properties": {"id": {"type": "string"}}},
    },
}]
messages = [{"role": "user", "content": "Where is order A1?"}]
while True:
    stream = await client.chat.completions.create(
        messages=messages, tools=tools, stream=True
    )
    async with stream:
        async for chunk in stream:
            if chunk.choices:
                print(chunk.choices[0].delta.content or "", end="", flush=True)
        completion = await stream.get_final_completion()
    message = completion.choices[0].message
    messages.append(message)
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = lookup_order(**json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id, "content": output})
```

Here `lookup_order` is your own function, and it returns text. `stream.get_final_completion()`
joins every chunk into the `ChatCompletion` that `create` returns without `stream=True`, and
raises `InferenceError` with `type` `incomplete_stream` when the stream closed before its end.
Append the `message` itself, not `dataclasses.asdict(message)`, so each tool call goes back with
every field the model sent.

## MCP servers

An `McpServer` in `tools` names an MCP server whose tools Moss Inference runs for the model, so
your code sends no `tool` messages for them:

```python theme={null}
from moss.inference import McpServer

crm = McpServer(
    server_label="crm",
    server_url="https://mcp.example.com/mcp",
    headers={"Authorization": "Bearer your-crm-token"},
)
stream = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Where is order A1?"}],
    tools=[crm],
    stream=True,
)
async with stream:
    async for chunk in stream:
        if chunk.tool_step:
            print(chunk.tool_step.type, chunk.tool_step.name)
        elif chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
```

Steps arrive in `chunk.tool_step` and `completion.tool_steps`, and MCP failures raise
`InferenceError` with `status`, `type`, `code` and `retry_after`, as in
[MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Options

`MossInference(project_id, project_key, ...)`:

| Option | Default | Notes |
| - | - | - |
| `project_id`, `project_key` | required | Your project id and key. |
| `api_key` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key raises `InferenceError`. |
| `timeout` | none | Seconds for a whole call, the stream included. 0 or less turns it off. |
| `idle_timeout` | `120` | Seconds to wait for the reply to start, and for each pause in it. 0 or less turns it off, and `None` keeps 120. |
| `max_retries` | `2` | Retries before the reply starts. `0` turns them off. |
| `default_headers` | none | Headers sent with every call. |

`chat.completions.create(...)` takes keyword arguments only:

| Option | Default | Notes |
| - | - | - |
| `model` | `"moss"` | The Moss model, or a vendor model with `api_key`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `messages` | required | OpenAI messages as dicts. A reply's `message` can go in too. |
| `stream` | `False` | `True` returns a `ChatCompletionStream` of `ChatCompletionChunk`, otherwise one `ChatCompletion`. |
| `indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve=False` does. |
| `retrieve` | `True` | `False` skips the search, and the model answers from the messages alone. |
| `cache` | `True` | Moss can answer an identical request from its reply cache. `False` always asks the model. |
| `api_key` | the client's | Your provider key for this call, in place of the client's. |
| `timeout`, `idle_timeout` | the client's | This call's limits, in place of the client's. 0 or less turns one off, and `None` keeps the client's. |
| `max_retries` | the client's | This call's retries, in place of the client's. |
| `extra_headers` | none | Headers for this call, over `default_headers`. |
| `extra_body` | none | Extra body fields for the model. |
| Other OpenAI arguments | none | `temperature`, `max_tokens`, `tools` and the rest go to the model. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.