Moss - Real-time Semantic Search for Conversational AI

MossInference is the Python client for Moss Inference, a drop-in for OpenAI’s chat completions. Its chat.completions.create takes OpenAI’s arguments, searches your indexes in the cloud, adds what it finds to the prompt, and returns the reply with OpenAI’s field names.

Requires moss 1.21.0 or later, installed as in Install. The API is async only.

Create a client

import os
from moss import MossInference

client = MossInference(os.environ["MOSS_PROJECT_ID"], os.environ["MOSS_PROJECT_KEY"])

The project id and key come from the Moss portal. Build one client and reuse it for every call. await client.close(), or the end of an async with client: block, closes its connections. With MOSS_PROJECT_ID and MOSS_PROJECT_KEY set, from moss import inference gives the same inference.chat.completions.create with no client to build.

Get a reply

completion = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Do you ship to Canada?"}],
    indexes=["faq", "policies"],
)
print(completion.choices[0].message.content)

completion.request_id is the response’s X-Moss-Request-Id, and so is stream.request_id on a stream. Include it when you report a problem. Both are None when the response has none.

Stream a reply

stream = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Do you ship to Canada?"}],
    indexes=["faq", "policies"],
    stream=True,
)
async with stream:
    async for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)

To stop early, break out of the loop: leaving the async with stream: block closes the connection, which stops the model. Only the last chunk carries usage, and it has no choices, which is why the loop checks chunk.choices.

Errors

A failed call raises moss.inference.InferenceError, from the async for loop once a stream has started. status is the HTTP status of a refused request, type names the failure, and message, also str(error), describes it. code is the error code Moss Inference sent, such as "rate_limit_exceeded", request_id is the response’s X-Moss-Request-Id, and retry_after is the wait in seconds that a Retry-After header asks for. A field the failure lacks is None. The SDK sets these types itself, with status None:

`type`	Cause
`invalid_request_error`	An argument the SDK rejects before sending, such as an empty key, or an index name that is empty or holds a comma.
`connection_error`	Moss Inference could not be reached, or the connection broke.
`timeout`	A time limit ran out, such as `timeout` or `idle_timeout`.
`incomplete_stream`	The stream ended before `[DONE]`.

Every other error reports what Moss Inference sent. See Errors, and Limits for a 429.

Tool calls

Pass OpenAI function tools in tools. When the reply calls tools, append its message to messages, then one tool message per call, and call again, as in Tool calls:

import json

tools = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "parameters": {"type": "object", "properties": {"id": {"type": "string"}}},
    },
}]
messages = [{"role": "user", "content": "Where is order A1?"}]
while True:
    stream = await client.chat.completions.create(
        messages=messages, tools=tools, stream=True
    )
    async with stream:
        async for chunk in stream:
            if chunk.choices:
                print(chunk.choices[0].delta.content or "", end="", flush=True)
        completion = await stream.get_final_completion()
    message = completion.choices[0].message
    messages.append(message)
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = lookup_order(**json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id, "content": output})

Here lookup_order is your own function, and it returns text. stream.get_final_completion() joins every chunk into the ChatCompletion that create returns without stream=True, and raises InferenceError with type incomplete_stream when the stream closed before its end. Append the message itself, not dataclasses.asdict(message), so each tool call goes back with every field the model sent.

MCP servers

An McpServer in tools names an MCP server whose tools Moss Inference runs for the model, so your code sends no tool messages for them:

from moss.inference import McpServer

crm = McpServer(
    server_label="crm",
    server_url="https://mcp.example.com/mcp",
    headers={"Authorization": "Bearer your-crm-token"},
)
stream = await client.chat.completions.create(
    messages=[{"role": "user", "content": "Where is order A1?"}],
    tools=[crm],
    stream=True,
)
async with stream:
    async for chunk in stream:
        if chunk.tool_step:
            print(chunk.tool_step.type, chunk.tool_step.name)
        elif chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)

Steps arrive in chunk.tool_step and completion.tool_steps, and MCP failures raise InferenceError with status, type, code and retry_after, as in MCP servers.

Options

MossInference(project_id, project_key, ...):

Option	Default	Notes
`project_id`, `project_key`	required	Your project id and key.
`api_key`	none	Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See Models. It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key raises `InferenceError`.
`timeout`	none	Seconds for a whole call, the stream included. 0 or less turns it off.
`idle_timeout`	`120`	Seconds to wait for the reply to start, and for each pause in it. 0 or less turns it off, and `None` keeps 120.
`max_retries`	`2`	Retries before the reply starts. `0` turns them off.
`default_headers`	none	Headers sent with every call.

chat.completions.create(...) takes keyword arguments only:

Option	Default	Notes
`model`	`"moss"`	The Moss model, or a vendor model with `api_key`. See Models.
`messages`	required	OpenAI messages as dicts. A reply’s `message` can go in too.
`stream`	`False`	`True` returns a `ChatCompletionStream` of `ChatCompletionChunk`, otherwise one `ChatCompletion`.
`indexes`	every index of the project	Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve=False` does.
`retrieve`	`True`	`False` skips the search, and the model answers from the messages alone.
`cache`	`True`	Moss can answer an identical request from its reply cache. `False` always asks the model.
`api_key`	the client’s	Your provider key for this call, in place of the client’s.
`timeout`, `idle_timeout`	the client’s	This call’s limits, in place of the client’s. 0 or less turns one off, and `None` keeps the client’s.
`max_retries`	the client’s	This call’s retries, in place of the client’s.
`extra_headers`	none	Headers for this call, over `default_headers`.
`extra_body`	none	Extra body fields for the model.
Other OpenAI arguments	none	`temperature`, `max_tokens`, `tools` and the rest go to the model.

Python

JavaScript

Swift (iOS)

Android

Elixir

Browser

C

Inference

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options

​Create a client

​Get a reply

​Stream a reply

​Errors

​Tool calls

​MCP servers

​Options

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options