> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions that answer from your Moss indexes, with OpenAI's arguments and replies, in the browser.

[Browser SDK](https://docs.moss.dev/docs/reference/browser/api.md) / Inference

`MossInference` takes OpenAI's `chat.completions.create` arguments plus `indexes`, and returns
OpenAI's shapes from [Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), which
searches your indexes in the cloud, so the page loads no index. It works like the
[JavaScript client](https://docs.moss.dev/docs/reference/js/inference.md), and this page covers what differs in the browser.

> [!NOTE]
>
> Requires `@moss-dev/moss-web` 1.7.0 or later, installed as in [Install](https://docs.moss.dev/docs/reference/browser/api.md#install).

## Create a client

```typescript theme={null}
import { MossInference } from "@moss-dev/moss-web";

const client = new MossInference("your_project_id", "your_project_key");
```

`new MossInference(projectId, projectKey, options)` takes your project id and key from the
[Moss portal](https://portal.usemoss.dev). CORS is open, so the page can call Moss Inference
directly, but a project key in page code is visible to every visitor and can also create and
delete your indexes. Keep direct calls to prototypes, local demos and internal tools. On a public
page, pass `null` for both and a `baseURL` path served by
[a route of yours](#keep-the-project-key-on-your-server) that adds them. Build one client and reuse it for every call. It has nothing to close.

## Get a reply

```typescript theme={null}
const completion = await client.chat.completions.create({
  messages: [{ role: "user", content: "Do you ship to Canada?" }],
  indexes: ["faq"],
});
document.querySelector("#answer")!.textContent = completion.choices[0].message.content;
```

`completion.requestId` is the response's `X-Moss-Request-Id`, and so is `stream.requestId` on a
stream. Include it when you report a problem. Both are `undefined` when the response has none.

## Stream a reply

```typescript theme={null}
const answer = document.querySelector("#answer")!;

const stream = await client.chat.completions.create({
  messages: [{ role: "user", content: "Do you ship to Canada?" }],
  indexes: ["faq"],
  stream: true,
});
document.querySelector("#stop")!.addEventListener("click", () => stream.controller.abort());
for await (const chunk of stream) {
  answer.textContent += chunk.choices[0]?.delta.content ?? "";
}
```

To stop early, leave the loop or call `stream.controller.abort()`, as the stop button does:
either closes the connection, which stops the model, and neither throws.

Only the last chunk carries `usage`, and it has no choices, which is why the loop reads `choices[0]?.`.

## Errors

Every failure other than an abort throws `InferenceError`, from the `for await` loop once a stream
has started. `status` is the HTTP status of a refused request, `type` names the failure, and
`message` describes it. `errorCode` is the error code Moss Inference sent, such as
`"rate_limit_exceeded"`, `requestId` is the response's `X-Moss-Request-Id`, and `retryAfter` is
the wait in milliseconds that a `Retry-After` header asks for. A field the failure lacks is
`undefined`. A call without `baseURL` gets no `retryAfter`, since Moss Inference does not expose
`Retry-After` to the page.

The SDK sets these types itself, with `status` `undefined`:

| `type` | Cause |
| - | - |
| `invalid_request_error` | An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, a `null` id and key without `baseURL`, or a `baseURL` that is not a path on the page's origin. |
| `connection_error` | Your route or Moss Inference could not be reached, the browser blocked the call, for example over CORS, or the connection broke. |
| `timeout` | A time limit ran out, such as `timeout` or `idleTimeout`. |
| `incomplete_stream` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

`tools`, `tool_choice` and `parallel_tool_calls` take OpenAI's shapes, and your code runs each
call. After the loop, `stream.finalChatCompletion()` joins every chunk into the completion that a
call without `stream: true` returns. Push its message onto `messages` as it is, then one `tool`
message per call, and call again, as in
[Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls):

```typescript theme={null}
import type { ChatCompletionMessageParam, ChatCompletionTool } from "@moss-dev/moss-web";
import { orderStatus } from "./orders";

const tools: ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "order_status",
      parameters: { type: "object", properties: { id: { type: "string" } }, required: ["id"] },
    },
  },
];
const messages: ChatCompletionMessageParam[] = [{ role: "user", content: "Where is order A1?" }];

const stream = await client.chat.completions.create({ messages, tools, stream: true });
for await (const chunk of stream) {
  answer.textContent += chunk.choices[0]?.delta.content ?? "";
}
const { message } = (await stream.finalChatCompletion()).choices[0];
messages.push(message);
for (const call of message.tool_calls ?? []) {
  const status = await orderStatus(JSON.parse(call.function.arguments).id);
  messages.push({ role: "tool", tool_call_id: call.id, content: status });
}
const reply = await client.chat.completions.create({ messages, tools });
```

`orderStatus` is your own function, and it returns text. `finalChatCompletion()` also works
without the `for await` loop, and a second call returns the same completion. It rejects with what
the loop would throw, or with the signal's reason after an abort.

## MCP servers

Add MCP entries in your [server route](#keep-the-project-key-on-your-server), never in page code.
Steps arrive in `chunk.moss_tool_step` and `completion.moss_tool_steps`, and MCP failures throw
`InferenceError` with `status`, `type`, `errorCode` and `retryAfter`, as in
[MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Keep the project key on your server

A public page calls Moss Inference through a route like this one, which adds the key on your server so it never reaches the page:

```typescript theme={null}
// Serves POST /moss/v1/chat/completions. Every call spends your project's quota,
// so the route answers only a visitor your own session check accepts.
import { isSignedIn } from "./session";

const ENDPOINT = "https://inference.moss.dev/v1/chat/completions";
const INDEXES = ["faq"];
const FORWARD = ["content-type", "accept"];
const REPLY = ["content-type", "retry-after", "x-moss-request-id"];

export async function POST(request: Request): Promise<Response> {
  if (!(await isSignedIn(request))) {
    return Response.json({ detail: "Sign in first." }, { status: 401 });
  }
  const { MOSS_PROJECT_ID, MOSS_PROJECT_KEY } = process.env;
  const indexes = request.headers.get("x-moss-indexes")?.split(",") ?? INDEXES;
  if (!indexes.every((name) => INDEXES.includes(name))) {
    return Response.json({ detail: "This page cannot search that index." }, { status: 403 });
  }
  const headers = new Headers();
  for (const name of FORWARD) {
    const value = request.headers.get(name);
    if (value !== null) headers.set(name, value);
  }
  headers.set("X-Moss-Indexes", indexes.join(","));
  headers.set("Authorization", `Bearer ${MOSS_PROJECT_ID}:${MOSS_PROJECT_KEY}`);
  const reply = await fetch(ENDPOINT, {
    method: "POST",
    headers,
    body: await request.text(),
    signal: request.signal,
    redirect: "manual",
  });
  const replyHeaders = [...reply.headers].filter(([name]) => REPLY.includes(name));
  return new Response(reply.body, { status: reply.status, headers: replyHeaders });
}
```

Every visitor can set any header, so the route forwards only the ones it names, refuses an
index outside its list with `403`, and always sends `X-Moss-Indexes`, because a call without it
searches every index of the project. `retrieve`, `cache`, an empty `indexes` list and `apiKey` in
page code therefore do not reach Moss Inference, and a route that wants them sets `X-Moss-Retrieve`,
`X-Moss-Cache` or `X-Moss-Provider-Key` itself.
With `null` credentials the client sends no `Authorization` header, so one you pass in
`headers`, such as the visitor's session token, reaches the route.

## Options

`new MossInference(projectId, projectKey, options?)`:

| Option | Default | Notes |
| - | - | - |
| `baseURL` | none | A path on the page's origin, such as `/moss/v1`, with no query or fragment. Each call resolves it against the page, so a client built during server rendering works in the browser. Without it, calls go to `https://inference.moss.dev/v1`. Required when the id and key are `null`. |
| `apiKey` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. In page code every visitor can read it, like a project key. An empty key throws `InferenceError`. |
| `timeout` | none | Milliseconds for a whole call, the stream included. `null` turns it off. |
| `idleTimeout` | `120_000` | Milliseconds to wait for the reply to start, and for each pause in it. `null` turns it off. |
| `maxRetries` | `2` | Retries before the reply starts. `0` turns them off. |
| `headers` | none | Headers sent with every call. |

`chat.completions.create(params, options?)` takes OpenAI's fields and the Moss fields in `params`,
and settings for this call only in `options`:

| Option | Default | Notes |
| - | - | - |
| `model` | `"moss"` | The Moss model, or a vendor model with `apiKey`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `messages` | required | OpenAI's message list. |
| `stream` | `false` | `true` resolves to a `ChatCompletionStream` of chunks, otherwise to one `ChatCompletion`. |
| `indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve: false` does, except through a [server route](#keep-the-project-key-on-your-server), which sends its own list. |
| `cache` | `true` | Moss can answer an identical request from its reply cache. `false` always asks the model. |
| `retrieve` | `true` | `false` skips the search, and the model answers from the messages alone. |
| Other OpenAI fields | none | `temperature`, `max_tokens`, `tools` and the rest go to the model. An `apiKey` or `api_key` field throws. |
| `options.apiKey` | the client's | Your provider key for this call, in place of the client's. |
| `options.timeout`, `options.idleTimeout` | the client's | This call's limits, in place of the client's. `null` turns one off. |
| `options.maxRetries` | the client's | This call's retries, in place of the client's. |
| `options.headers` | none | Headers for this call. They add to the client's or replace them by name. |
| `options.signal` | none | An `AbortSignal` that cancels the call. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.