> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions that answer from your indexes, in OpenAI's shape.

`MossInference` is a drop-in for OpenAI's client: its `chat.completions.create` takes OpenAI's
arguments plus `indexes` and returns the same shapes. Each call goes to
[Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), which searches those indexes
and adds what it finds to the prompt.

> [!NOTE]
>
> Requires `@moss-js/moss` 1.21.0 or later, installed as in [Install](https://docs.moss.dev/docs/reference/js/api.md#install).

## Create a client

```typescript theme={null}
import { MossInference } from '@moss-js/moss'

const client = new MossInference(process.env.MOSS_PROJECT_ID!, process.env.MOSS_PROJECT_KEY!)
```

The project id and key come from the [Moss portal](https://portal.usemoss.dev). Build one client
and reuse it for every call. It has nothing to close. With `MOSS_PROJECT_ID` and `MOSS_PROJECT_KEY` set,
`import { inference } from '@moss-js/moss'` gives the same `inference.chat.completions.create`
with no client to build.

## Get a reply

```typescript theme={null}
const completion = await client.chat.completions.create({
  messages: [
    { role: 'system', content: 'You are the support agent for Acme.' },
    { role: 'user', content: 'Do you ship to Canada?' },
  ],
  indexes: ['faq', 'policies'],
})
console.log(completion.choices[0].message.content)
```

`completion.requestId` is the response's `X-Moss-Request-Id`, and so is `stream.requestId` on a
stream. Include it when you report a problem. Both are `undefined` when the response has none.

## Stream a reply

```typescript theme={null}
const stream = await client.chat.completions.create({
  messages: [{ role: 'user', content: 'Do you ship to Canada?' }],
  indexes: ['faq'],
  stream: true,
})
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? '')
}
```

To stop early, leave the loop or call `stream.controller.abort()`: either closes the connection,
which stops the model, and neither throws.

Only the last chunk carries `usage`, and it has no choices, which is why the loop reads `choices[0]?.`.

## Errors

Every failure other than an abort throws `InferenceError`, from the `for await` loop once a stream
has started. `status` is the HTTP status of a refused request, `type` names the failure, and
`message` describes it. `errorCode` is the error code Moss Inference sent, such as
`"rate_limit_exceeded"`, `requestId` is the response's `X-Moss-Request-Id`, and `retryAfter` is
the wait in milliseconds that a `Retry-After` header asks for. A field the failure lacks is
`undefined`. The error's Moss `code` lets [`isMossError`](./functions/isMossError) accept it.

The SDK sets these types itself, with `status` `undefined`:

| `type` | Cause |
| - | - |
| `invalid_request_error` | An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, or an unset `MOSS_PROJECT_ID` for `inference`. |
| `connection_error` | Moss Inference could not be reached, or the connection broke. |
| `timeout` | A time limit ran out, such as `timeout` or `idleTimeout`. |
| `incomplete_stream` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

`tools`, `tool_choice` and `parallel_tool_calls` take OpenAI's shapes. Stream the reply, then
`stream.finalChatCompletion()` joins it into one completion. Push its message onto `messages`, add
one `tool` message per call, and call again until the reply has no tool calls, as in
[Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls):

```typescript theme={null}
import type { ChatCompletionMessageParam, ChatCompletionTool } from '@moss-js/moss'

const tools: ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'check_stock',
      description: 'How many units of a product are in stock.',
      parameters: { type: 'object', properties: { sku: { type: 'string' } }, required: ['sku'] },
    },
  },
]
const messages: ChatCompletionMessageParam[] = [
  { role: 'user', content: 'Is the EQB jacket in stock?' },
]

for (;;) {
  const stream = await client.chat.completions.create({ messages, tools, stream: true })
  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta.content ?? '')
  }
  const { message } = (await stream.finalChatCompletion()).choices[0]
  messages.push(message)
  if (!message.tool_calls?.length) break
  for (const call of message.tool_calls) {
    const { sku } = JSON.parse(call.function.arguments)
    messages.push({ role: 'tool', tool_call_id: call.id, content: await checkStock(sku) })
  }
}
```

`checkStock` stands for your own code. Push the message unchanged, so each tool call keeps every
field the model sent. `finalChatCompletion()` also works without the `for await` loop. Without
`stream: true`, `create` returns the joined completion directly.

## MCP servers

An `McpServer` entry in `tools` names an MCP server whose tools Moss Inference calls for the model
during the turn. The stream carries each tool step, then the answer:

```typescript theme={null}
import type { McpServer } from '@moss-js/moss'

const crm: McpServer = {
  type: 'mcp',
  server_label: 'crm',
  server_url: 'https://mcp.example.com/mcp',
  headers: { Authorization: `Bearer ${process.env.CRM_TOKEN}` },
}
const stream = await client.chat.completions.create({
  messages: [{ role: 'user', content: 'Where is order A1?' }],
  tools: [crm],
  stream: true,
})
for await (const chunk of stream) {
  const step = chunk.moss_tool_step
  if (step) console.log(step.type, step.server_label, step.name)
  else process.stdout.write(chunk.choices[0]?.delta.content ?? '')
}
const { moss_tool_steps } = await stream.finalChatCompletion()
console.log(moss_tool_steps)
```

Steps arrive in `chunk.moss_tool_step` and `completion.moss_tool_steps`, and MCP failures throw
`InferenceError` with `status`, `type`, `errorCode` and `retryAfter`, as in
[MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Options

`new MossInference(projectId, projectKey, options?)`:

| Option | Default | Notes |
| - | - | - |
| `apiKey` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key throws `InferenceError`. |
| `timeout` | none | Milliseconds for a whole call, the stream included. `null` turns it off. |
| `idleTimeout` | `120_000` | Milliseconds to wait for the reply to start, and for each pause in it. `null` turns it off. |
| `maxRetries` | `2` | Retries before the reply starts. `0` turns them off. |
| `headers` | none | Headers sent with every call. |

`chat.completions.create(params, options?)` takes OpenAI's fields and the Moss fields in `params`,
and settings for this call only in `options`:

| Option | Default | Notes |
| - | - | - |
| `model` | `"moss"` | The Moss model, or a vendor model with `apiKey`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `messages` | required | OpenAI's message list. |
| `stream` | `false` | `true` resolves to a `ChatCompletionStream` of chunks, otherwise to one `ChatCompletion`. |
| `indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve: false` does. |
| `cache` | `true` | Moss can answer an identical request from its reply cache. `false` always asks the model. |
| `retrieve` | `true` | `false` skips the search, and the model answers from the messages alone. |
| Other OpenAI fields | none | `temperature`, `max_tokens`, `tools` and the rest go to the model. An `apiKey` or `api_key` field throws. |
| `options.apiKey` | the client's | Your provider key for this call, in place of the client's. |
| `options.timeout`, `options.idleTimeout` | the client's | This call's limits, in place of the client's. `null` turns one off. |
| `options.maxRetries` | the client's | This call's retries, in place of the client's. |
| `options.headers` | none | Headers for this call. They add to the client's or replace them by name. |
| `options.signal` | none | An `AbortSignal` that cancels the call. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.