> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference

> Chat completions that answer from your Moss indexes, in OpenAI's shape, from C.

[C SDK](https://docs.moss.dev/docs/reference/c/getting-started.md) / Inference

`MossInference` is the C client for
[Moss Inference](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md), a drop-in
for OpenAI's chat completions. `moss_inference_chat_completions_create` takes an
OpenAI request as JSON, searches your indexes in the cloud, adds what it finds
to the prompt, and hands back the reply as OpenAI's JSON.

> [!NOTE]
>
> Requires `libmoss` 0.28.0 or later, installed as in [Install](https://docs.moss.dev/docs/reference/c/getting-started.md#install). A
> `libmoss.h` with these functions defines `MOSS_INFERENCE_API_VERSION`. From
> version `2`, the calls keep every tool call field and report MCP tool steps.

## Create a client

```c theme={null}
#include "libmoss.h"
#include <stdio.h>

MossInference *client = NULL;
const char *options =
    "{\"project_id\": \"your-project-id\","
    " \"project_key\": \"your-project-key\"}";
if (moss_inference_new(options, &client) != OK) {
    fprintf(stderr, "%s\n", moss_last_error());
    return 1;
}
```

The project id and key come from the [Moss portal](https://portal.usemoss.dev).
Build one client and reuse it for every call, from any thread. Free it with
`moss_inference_free` once no call on it is running.

## Get a reply

With `NULL` for the callback, `out_json` gets the reply as the JSON of one OpenAI
`chat.completion`:

```c theme={null}
char *completion = NULL;
const char *request =
    "{\"indexes\": [\"faq\", \"policies\"],"
    " \"messages\": [{\"role\": \"user\", \"content\": \"Do you ship to Canada?\"}]}";
if (moss_inference_chat_completions_create(client, request, NULL, NULL, NULL,
                                           &completion) == OK) {
    printf("%s\n", completion);
    moss_free_string(completion);
} else {
    fprintf(stderr, "%s\n", moss_last_error());
}
```

The text is in `choices[0].message.content`, which is `null` when the reply
only calls tools.

The completion's `request_id` is the response's `X-Moss-Request-Id`. Include it
when you report a problem. It is `null` when the response has none.

## Stream a reply

```c theme={null}
static int32_t print_chunk(const char *chunk_json, void *user_data) {
    (void)user_data;
    printf("%s\n", chunk_json);
    return 0; // any other value stops the stream
}

const char *request =
    "{\"indexes\": [\"faq\", \"policies\"],"
    " \"messages\": [{\"role\": \"user\", \"content\": \"Do you ship to Canada?\"}]}";
if (moss_inference_chat_completions_create(client, request, NULL, print_chunk, NULL,
                                           NULL) != OK) {
    fprintf(stderr, "%s\n", moss_last_error());
}
```

The call blocks until the reply ends, and hands the callback each chunk as the
JSON text of one OpenAI `chat.completion.chunk`, with the new text in
`choices[0].delta.content` when `choices` is not empty. To stop early, return a
value other than `0` from the callback: the call closes the connection, which
stops the model, and returns `OK`.

Only the last chunk carries `usage`, and it has no choices, so check `choices` before
reading `choices[0]`.

## Errors

A failed call returns a negative `MossResult`, a streaming one after the chunks
the callback already got. Right after it, on the same thread, `moss_last_error()`
holds the message, and `moss_last_error_detail()` a JSON object in which `status`
is the HTTP status of a refused request, `type` names the failure, `code` is the
error code Moss Inference sent, such as `"rate_limit_exceeded"`, `request_id` is
the response's `X-Moss-Request-Id`, and `retry_after_ms` is the wait in
milliseconds that a `Retry-After` header asks for. A field the failure lacks is
`null`.

The SDK sets these itself:

| `type` | Returns | Cause |
| - | - | - |
| none, the detail is `NULL` | `ERR_NULL_POINTER`, `ERR_INVALID_ARG` | An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, a `NULL` client or request, or no callback and no `out_json`. |
| `connection_error` | `ERR_CLOUD` | Moss Inference could not be reached, or the connection broke. |
| `timeout` | `ERR_CLOUD` | A time limit ran out, such as `timeout_ms` or `idle_timeout_ms`. |
| `incomplete_stream` | `ERR_CLOUD` | The stream ended before `[DONE]`. |

Every other error reports what Moss Inference sent. The call returns
`ERR_UNAUTHORIZED` for `authentication_error` and `permission_error`,
`ERR_INVALID_ARG` for `invalid_request_error`, `ERR_DESERIALIZATION` for a
stream event that is not a JSON object, and `ERR_CLOUD` for the rest. See
[Errors](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#errors), and
[Limits](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#limits) for a `429`.

## Tool calls

`tools`, `tool_choice` and `parallel_tool_calls` go in the request JSON as in
OpenAI's API, and your code runs the tools. Pass a callback to stream the reply
and `out_json` to get it joined, in the same call:

```c theme={null}
const char *request =
    "{\"messages\": [{\"role\": \"user\", \"content\": \"Where is order A1?\"}],"
    " \"tools\": [{\"type\": \"function\", \"function\": {\"name\": \"lookup_order\","
    " \"parameters\": {\"type\": \"object\","
    " \"properties\": {\"id\": {\"type\": \"string\"}}}}}]}";
char *completion = NULL;
if (moss_inference_chat_completions_create(client, request, NULL, print_chunk, NULL,
                                           &completion) == OK) {
    // With your JSON library, run each call in choices[0].message.tool_calls.
    moss_free_string(completion);
} else {
    fprintf(stderr, "%s\n", moss_last_error());
}
```

`print_chunk` is the callback from [Stream a reply](#stream-a-reply). The
joined message keeps every field of each call, with `""` for an id, name or
arguments the model left out. Append it to `messages` as it is, then one `tool`
message per call, and call again with the same `tools`, as in
[Tool calls](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#tool-calls).

## MCP servers

A `tools` entry of type `"mcp"` names an MCP server whose tools Moss Inference
runs for the model within the same call, so you send no `tool` messages for them:

```c theme={null}
const char *request =
    "{\"messages\": [{\"role\": \"user\", \"content\": \"Where is order A1?\"}],"
    " \"tools\": [{\"type\": \"mcp\", \"server_label\": \"crm\","
    " \"server_url\": \"https://mcp.example.com/mcp\","
    " \"headers\": {\"Authorization\": \"Bearer your-crm-token\"}}]}";
char *completion = NULL;
if (moss_inference_chat_completions_create(client, request, NULL, print_chunk, NULL,
                                           &completion) == OK) {
    printf("%s\n", completion);
    moss_free_string(completion);
} else {
    fprintf(stderr, "%s\n", moss_last_error());
}
```

Steps arrive in each chunk's `moss_tool_step` and the completion's
`moss_tool_steps`, and MCP failures return a negative `MossResult`, with
`status`, `type`, `code` and `retry_after_ms` in `moss_last_error_detail()`, as
in [MCP servers](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#mcp-servers).

## Memory and the chunk callback

| Pointer | Owner | Release |
| - | - | - |
| `MossInference *` from `moss_inference_new` | You | `moss_inference_free`, once no call on it is running. `NULL` is a no-op. |
| `*out_json` from `moss_inference_chat_completions_create` | You | `moss_free_string`. It is set only when the call returns `OK`. |
| `chunk_json` in the callback | `libmoss` | Nothing. It is valid until the callback returns, so copy it to keep it. |
| `moss_last_error()`, `moss_last_error_detail()` | `libmoss` | Nothing. They are valid until the next `libmoss` call on the same thread. |
| `options_json`, `request_json` | You | Your own. They only need to live until the call returns. |

* The callback runs on the thread that made the call, once per chunk, and never
  after the call returns. `libmoss` never reads or frees `user_data`.
* The next chunk is read only after the callback returns, so a slow callback
  holds up the stream, and its time counts toward `timeout_ms`.
* A streaming call can pass `out_json` too, and gets the reply as one
  `chat.completion`. When the callback stops the stream, that reply ends with
  the chunk it stopped at.

## Options

```c theme={null}
typedef int32_t (*MossInferenceChunkFn)(const char *chunk_json, void *user_data);
MossResult moss_inference_new(const char *options_json, MossInference **out);
void moss_inference_free(MossInference *client);
MossResult moss_inference_chat_completions_create(
    const MossInference *client, const char *request_json, const char *options_json,
    MossInferenceChunkFn on_chunk, void *user_data, char **out_json);
```

`options_json` of `moss_inference_new`:

| Key | Default | Notes |
| - | - | - |
| `project_id`, `project_key` | required | Your project id and key. |
| `api_key` | none | Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key returns `ERR_INVALID_ARG`. |
| `timeout_ms` | none | Milliseconds for a whole call, the stream included. `0` turns it off. |
| `idle_timeout_ms` | `120000` | Milliseconds to wait for the reply to start, and for each pause in it. `0` turns it off. |
| `max_retries` | `2` | Retries before the reply starts. `0` turns them off. |

`options_json` of `moss_inference_chat_completions_create`, which may be `NULL`:

| Key | Default | Notes |
| - | - | - |
| `api_key` | the client's | Your provider key for this call, in place of the client's. |
| `timeout_ms`, `idle_timeout_ms` | the client's | This call's limits, in place of the client's. `0` turns one off. |
| `max_retries` | the client's | This call's retries, in place of the client's. |

`request_json` of `moss_inference_chat_completions_create`:

| Key | Default | Notes |
| - | - | - |
| `model` | `"moss"` | The Moss model, or a vendor model with `api_key`. See [Models](https://docs.moss.dev/docs/api-reference/v1/inference/chat-completions.md#models). |
| `messages` | required | OpenAI messages. A reply's `message` can go in too. |
| `indexes` | every index of the project | Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `"retrieve": false` does. |
| `retrieve` | `true` | `false` skips the search, and the model answers from the messages alone. |
| `cache` | `true` | Moss can answer an identical request from its reply cache. `false` always asks the model. |
| `tools`, `tool_choice`, `parallel_tool_calls` | none | OpenAI's tool fields. A `tools` entry can also name an MCP server. See [Tool calls](#tool-calls) and [MCP servers](#mcp-servers). |
| Other OpenAI fields | none | `temperature`, `max_tokens` and the rest go to the model. `stream` is always `true`. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.