> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions in OpenAI's shape that answer from your Moss indexes, from an Android app.

`MossInference` is a Kotlin client for
[Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), which
searches your indexes in the cloud and answers in OpenAI's chat completions
shape, with camelCase fields. `chat.completions.create` returns the whole
reply, and `createStreaming` streams it.

> [!NOTE]
>
> Requires the `moss-android` AAR 0.28.0 or later, installed as in
> [Install](https://docs.moss.dev/docs/reference/android/getting-started.md#install).

## Create a client

```kotlin theme={null}
import dev.usemoss.moss.MossInference

val inference = MossInference(projectId = projectId, projectKey = projectKey)
```

The project id and key come from the [Moss portal](https://portal.usemoss.dev).
Build one client and reuse it for every call, from any thread. Calls and
stream reads block, and throw `MossException` on the main thread, so run them
from `Dispatchers.IO`.

## Get a reply

```kotlin theme={null}
import dev.usemoss.moss.ChatMessage
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

val messages = listOf(
    ChatMessage.system("You are the support agent for Acme."),
    ChatMessage.user("Do you ship to Canada?"),
)

val completion = withContext(Dispatchers.IO) {
    inference.chat.completions.create(messages, indexes = listOf("faq"))
}
println(completion.choices[0].message.content)
```

`completion.requestId` is the response's `X-Moss-Request-Id`, and so is
`stream.requestId` on a stream. Include it when you report a problem. Both are
`null` when the response has none.

## Stream a reply

```kotlin theme={null}
import kotlinx.coroutines.ensureActive

withContext(Dispatchers.IO) {
    inference.chat.completions.createStreaming(messages, indexes = listOf("faq")).use { stream ->
        for (chunk in stream) {
            ensureActive()
            chunk.choices.firstOrNull()?.delta?.content?.let(::print)
        }
    }
}
```

To stop early, leave `use` with a `break` or `return`: that closes the
connection, which stops the model. Cancelling the coroutine alone does not, so
the loop calls `ensureActive()`, whose throw leaves `use` the same way.

Only the last chunk carries `usage`, and it has no choices, which is why the loop reads
`choices.firstOrNull()`.

## Errors

A failed call throws `InferenceError`, from the `for` loop once a stream has
started. `status` is the HTTP status of a refused request, `type` names the
failure, and `message` describes it. `code` is the error code Moss Inference
sent, such as `"rate_limit_exceeded"`, `requestId` is the response's
`X-Moss-Request-Id`, and `retryAfterMs` is the wait in milliseconds that a
`Retry-After` header asks for. A field the failure lacks is `null`.

The SDK raises these itself, the last three as an `InferenceError` with that
`type` and `status` `null`:

| Error | Cause |
| - | - |
| `IllegalArgumentException` | An argument the SDK rejects before sending, such as a blank key, or an index name that is blank or holds a comma. |
| `connection_error` | Moss Inference could not be reached, or the connection broke. |
| `timeout` | A time limit ran out, such as `timeoutMs` or `idleTimeoutMs`. |
| `incomplete_stream` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

Pass `tools`, and the model may answer with `toolCalls`. `create` reads the
whole stream and joins each call's arguments. Append the reply with
`ChatMessage.assistant(reply)`, which sends it back unchanged, then one
`ChatMessage.tool(callId, content)` per call, and call again, as in
[Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls):

```kotlin theme={null}
import dev.usemoss.moss.Tool

val tools = listOf(
    Tool.function(
        name = "lookup_order",
        description = "Finds an order by its id.",
        parameters = """{"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}""",
    ),
)

val answer = withContext(Dispatchers.IO) {
    val chat = messages.toMutableList()
    val reply = inference.chat.completions.create(chat, tools = tools).choices[0].message
    chat += ChatMessage.assistant(reply)
    for (call in reply.toolCalls.orEmpty()) {
        chat += ChatMessage.tool(call.id.orEmpty(), lookupOrder(call.function?.arguments.orEmpty()))
    }
    inference.chat.completions.create(chat, tools = tools).choices[0].message.content
}
println(answer)
```

`lookupOrder` is your own function, which takes the call's JSON arguments and
returns the result as text. `createStreaming` streams each call in pieces, in
`delta.toolCalls`, so use `create` for a reply you send back.

## MCP servers

A `Tool.mcp` entry names an MCP server whose tools Moss Inference runs for the
model during the same call, so your code sends no `ChatMessage.tool` for them:

```kotlin theme={null}
import dev.usemoss.moss.McpServer

val crm = McpServer(
    serverLabel = "crm",
    serverUrl = "https://mcp.example.com/mcp",
    headers = mapOf("Authorization" to "Bearer $crmToken"),
)

withContext(Dispatchers.IO) {
    inference.chat.completions.createStreaming(messages, tools = listOf(Tool.mcp(crm))).use { stream ->
        for (chunk in stream) {
            ensureActive()
            chunk.toolStep?.let { println("${it.type}: ${it.serverLabel} ${it.name}") }
            chunk.choices.firstOrNull()?.delta?.content?.let(::print)
        }
    }
}
```

`McpServer` also takes `allowedTools` and `transport`,
`McpServer.Transport.STREAMABLE_HTTP` or `SSE`. Steps arrive in
`chunk.toolStep` and `completion.toolSteps`, with camelCase fields such as
`isError`, and MCP failures throw `InferenceError` with `status`, `type`,
`code` and `retryAfterMs`, as in
[MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Options

`MossInference(...)`:

| Parameter | Default | Notes |
| - | - | - |
| `projectId`, `projectKey` | required | Your project id and key. |
| `apiKey` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. A blank key throws `IllegalArgumentException`. |
| `timeoutMs` | none | Milliseconds for a whole call, the stream included. `null` turns it off. |
| `idleTimeoutMs` | `120_000` | Milliseconds to wait for the reply to start, and for each pause in it. `null` turns it off. |
| `maxRetries` | `2` | Retries before the reply starts. `0` turns them off. |

`chat.completions.create` and `createStreaming` take the same parameters,
`messages` first:

| Parameter | Default | Notes |
| - | - | - |
| `messages` | required | A `List<ChatMessage>`. `ChatMessage.system`, `user` and `assistant` build the common messages, `ChatMessage.assistant(reply)` sends a reply back, `ChatMessage.tool(callId, content)` answers a tool call, and `ChatMessage.raw` takes any other as the text of a JSON object. |
| `model` | `"moss"` | The Moss model, or a vendor model with `apiKey`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve = false` does. |
| `retrieve` | `true` | `false` skips the search, and the model answers from the messages alone. |
| `cache` | `true` | Moss can answer an identical request from its reply cache. `false` always asks the model. |
| `apiKey` | the client's | Your provider key for this call, in place of the client's. |
| `extra` | none | Other OpenAI fields, such as `temperature` or `max_tokens`, as JSON values. The call's `model` and `messages` win over `extra`, and so do `tools`, `toolChoice` and `parallelToolCalls` when given. `stream` is always `true`. |
| `tools` | none | A `List<Tool>`: `Tool.function` for a function of yours, `Tool.mcp` for an MCP server, and `Tool.raw` for any other tool object. See [Tool calls](#tool-calls) and [MCP servers](#mcp-servers). |
| `toolChoice` | none | `ToolChoice.AUTO`, `NONE`, `REQUIRED`, `ToolChoice.function(name)`, or `ToolChoice.raw(json)` for any other JSON object. |
| `parallelToolCalls` | none | Whether the model may call several tools at once. |
| `timeoutMs`, `idleTimeoutMs` | the client's | This call's limits, in place of the client's. `null` turns one off. |
| `maxRetries` | the client's | This call's retries, in place of the client's. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.