> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions that answer from your Moss indexes, in OpenAI's shape, from Elixir.

[Elixir SDK](https://docs.moss.dev/docs/reference/elixir/api.md) / Inference

`Moss.Inference.Chat.Completions.create/3` sends OpenAI's chat completions body, as a map, to
[Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), which searches the indexes
you name, adds what it finds to the prompt and replies with OpenAI's field names as string keys.
Code written for OpenAI keeps its body and adds `:indexes`.

> [!NOTE]
>
> Requires `moss` 1.10.0 or later, installed as in [Install](https://docs.moss.dev/docs/reference/elixir/api.md#install).

## Create a client

```elixir theme={null}
project_id = System.fetch_env!("MOSS_PROJECT_ID")
project_key = System.fetch_env!("MOSS_PROJECT_KEY")

client = Moss.Inference.new(project_id, project_key)
```

The project id and key come from the [Moss portal](https://portal.usemoss.dev). Build one client
and reuse it for every call, from any process. It is plain data with nothing to close.

## Get a reply

```elixir theme={null}
alias Moss.Inference.Chat.Completions

messages = [%{role: "user", content: "Do you ship to Canada?"}]

{:ok, completion} = Completions.create(client, %{messages: messages}, indexes: ["faq"])

IO.puts(hd(completion["choices"])["message"]["content"])
```

`completion["request_id"]` is the response's `X-Moss-Request-Id`, and so is `stream.request_id`
on a stream. Include it when you report a problem. Both are `nil` when the response has none.

## Stream a reply

```elixir theme={null}
{:ok, stream} =
  Completions.create(client, %{messages: messages, stream: true}, indexes: ["faq"])

for chunk <- stream, choice <- chunk["choices"] do
  IO.write(choice["delta"]["content"] || "")
end
```

Read the stream once, in the process that called `create/3`. To stop early, halt it, for example
with `Enum.take/2`, or call `close/1` from any process: either closes the connection, which stops
the model.

```elixir theme={null}
Moss.Inference.ChatCompletionStream.close(stream)
```

Only the last chunk carries `usage`, and it has no choices, which is why the comprehension above
skips it.

## Errors

A failed call gives a `Moss.Inference.Error`: `new/3` raises it, `create/3` returns it as
`{:error, error}`, and a stream raises it while you read it. `status` is the HTTP status of a
refused request, `type` names the failure, and `message` describes it. `code` is the error code
Moss Inference sent, such as `"rate_limit_exceeded"`, `request_id` is the response's
`X-Moss-Request-Id`, and `retry_after` is the wait in milliseconds that a `Retry-After` header
asks for. A field the failure lacks is `nil`.

The SDK sets these types itself, with `status` `nil`:

| `type` | Cause |
| - | - |
| `invalid_request_error` | An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, or an option inside `params`, such as `:api_key` or `apiKey`. |
| `connection_error` | Moss Inference could not be reached, or the connection broke. |
| `timeout` | A time limit ran out, such as `:timeout` or `:idle_timeout`. |
| `incomplete_stream` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

`tools`, `tool_choice` and `parallel_tool_calls` go in `params` as in OpenAI's API, and your code
runs the tools. With `stream: true`, each call arrives in pieces in `choice["delta"]["tool_calls"]`.
Without it, `create/3` joins them into the reply's message, with `""` for a missing id, name or
arguments. Append that message to `messages` as it is, add one `tool` message per call, and call
again, as in [Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls).

```elixir theme={null}
lookup = %{
  type: "function",
  function: %{name: "lookup_order", parameters: %{type: "object", properties: %{}}}
}

{:ok, completion} =
  Completions.create(client, %{messages: messages, tools: [lookup]}, indexes: ["faq"])

message = hd(completion["choices"])["message"]

results =
  for call <- message["tool_calls"] || [] do
    %{role: "tool", tool_call_id: call["id"], content: lookup_order(call["function"]["arguments"])}
  end

params = %{messages: messages ++ [message | results], tools: [lookup]}
{:ok, completion} = Completions.create(client, params, indexes: ["faq"])
```

`lookup_order/1` stands for your own code. It gets the arguments as a JSON string and returns the
result as a string.

## MCP servers

A `Moss.Inference.McpServer` in `tools` lets Moss Inference call that server's tools itself
during the turn.

```elixir theme={null}
crm = %Moss.Inference.McpServer{
  server_label: "crm",
  server_url: "https://mcp.example.com/mcp",
  headers: %{"Authorization" => "Bearer " <> System.fetch_env!("CRM_TOKEN")}
}

{:ok, stream} =
  Completions.create(client, %{messages: messages, tools: [crm], stream: true}, indexes: ["faq"])

for chunk <- stream do
  case chunk do
    %{"moss_tool_step" => step} ->
      IO.puts("\n#{step["type"]} #{step["server_label"]}.#{step["name"]}")

    %{"choices" => choices} ->
      for choice <- choices, do: IO.write(choice["delta"]["content"] || "")
  end
end
```

Steps arrive in `chunk["moss_tool_step"]` and `completion["moss_tool_steps"]`, and MCP failures
give a `Moss.Inference.Error` with `status`, `type`, `code` and `retry_after`, as in
[MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Options

`Moss.Inference.new(project_id, project_key, opts)`:

| Option | Default | Notes |
| - | - | - |
| `:api_key` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key raises `Moss.Inference.Error`. |
| `:timeout` | none | Milliseconds for a whole call, the stream included. `:infinity` turns it off. |
| `:idle_timeout` | `120_000` | Milliseconds to wait for the reply to start, and for each pause in it. `:infinity` turns it off. |
| `:max_retries` | `2` | Retries before the reply starts. `0` turns them off. |

`Moss.Inference.Chat.Completions.create(client, params, opts)`:

| Argument | Default | Notes |
| - | - | - |
| `params` | required | The OpenAI request body as a map, with atom or string keys. It needs `messages`. |
| `model` in `params` | `"moss"` | The Moss model, or a vendor model with `:api_key`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `stream` in `params` | `false` | `true` returns `{:ok, stream}` once the reply starts, an `Enumerable` of `chat.completion.chunk` maps, otherwise `{:ok, completion}`, one `chat.completion` map. |
| `:indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve: false` does. |
| `:retrieve` | `true` | `false` skips the search, and the model answers from the messages alone. |
| `:cache` | `true` | Moss can answer an identical request from its reply cache. `false` always asks the model. |
| `:api_key` | the client's | Your provider key for this call, in place of the client's. |
| `:timeout`, `:idle_timeout` | the client's | This call's limits, in place of the client's. `:infinity` turns one off. |
| `:max_retries` | the client's | This call's retries, in place of the client's. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.