Moss - Real-time Semantic Search for Conversational AI

POST /v1/chat/completions searches your project’s indexes for the user’s question, adds what it finds to the prompt, and returns the reply in OpenAI’s shape. Any OpenAI client works: change the base URL and the key.

Base URL

https://inference.moss.dev/v1

Authentication

Send the project id and project key from the Moss portal as one bearer token:

Authorization: Bearer <project_id>:<project_key>

Request

The body is an OpenAI chat completion request.

Field	Required	Notes
`model`	no	`moss` (the default), or a vendor model with your own key. See Models.
`messages`	yes	OpenAI message list. System messages are supported.
`stream`	no	`true` streams OpenAI `chat.completion.chunk` events, ending with `data: [DONE]`. See Streaming. Omitted returns one `chat.completion`.
`temperature`, `max_tokens`, `tools`, `response_format`, …	no	Passed to the model. A reply without `stream` has one choice and no `logprobs`.

Header	Default	What it does
`X-Moss-Indexes`	every index of the project	Comma-separated index names to search, for example `faq,policies`. Required when the project has more than 32 indexes. An index name outside this project returns 403.
`X-Moss-Retrieve`	`on`	`off` skips the search.
`X-Moss-Cache`	`on`	`off` skips the reply cache. A repeated question otherwise returns the cached reply with `usage.prompt_tokens` of 0.
`X-Moss-Provider-Key`	none	Your OpenAI, Anthropic, or Google API key, for a vendor `model`.

usage.prompt_tokens includes the retrieved facts. A reply that reached the model carries X-Moss-Request-Id: include it when you report a problem. The x-ratelimit-limit-*, x-ratelimit-remaining-* and x-ratelimit-reset-* headers, for requests and, on moss, tokens, show how much of the project’s limits is left.

Streaming

The last chunk before data: [DONE] has empty choices and the whole reply’s usage; no other chunk carries it. {"include_usage": false} in stream_options leaves it out. Check choices before reading choices[0], as the examples below do.

data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "Yes, we ship"}, "finish_reason": null}]}

data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [{"index": 0, "delta": {"content": " to Canada."}, "finish_reason": "stop"}]}

data: {"id": "chatcmpl-1", "object": "chat.completion.chunk", "created": 1761177600, "model": "moss", "choices": [], "usage": {"prompt_tokens": 165, "completion_tokens": 7, "total_tokens": 172}}

data: [DONE]

Examples

The Python and JavaScript examples below use OpenAI’s own clients. Every Moss SDK also has its own client for this endpoint: Python, JavaScript, Swift, Android, Elixir, Browser, and C.

curl -N https://inference.moss.dev/v1/chat/completions \
  -H "Authorization: Bearer $MOSS_PROJECT_ID:$MOSS_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Moss-Indexes: faq" \
  -d '{
    "model": "moss",
    "stream": true,
    "messages": [
      { "role": "system", "content": "You are the support agent for Acme." },
      { "role": "user", "content": "Do you ship to Canada?" }
    ]
  }'

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.moss.dev/v1",
    api_key=f"{os.environ['MOSS_PROJECT_ID']}:{os.environ['MOSS_PROJECT_KEY']}",
    default_headers={"X-Moss-Indexes": "faq"},
)
stream = client.chat.completions.create(
    model="moss",
    messages=[
        {"role": "system", "content": "You are the support agent for Acme."},
        {"role": "user", "content": "Do you ship to Canada?"},
    ],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://inference.moss.dev/v1",
  apiKey: `${process.env.MOSS_PROJECT_ID}:${process.env.MOSS_PROJECT_KEY}`,
  defaultHeaders: { "X-Moss-Indexes": "faq" },
});
const stream = await client.chat.completions.create({
  model: "moss",
  messages: [
    { role: "system", content: "You are the support agent for Acme." },
    { role: "user", content: "Do you ship to Canada?" },
  ],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}

Models

`model`	Where it runs	Key
`moss`, or omitted	The Moss model on Moss GPUs.	Project key only.
`gpt-`, `o`, `chatgpt-*`	OpenAI	`X-Moss-Provider-Key`
`claude-*`	Anthropic	`X-Moss-Provider-Key`
`gemini-*`	Google	`X-Moss-Provider-Key`

Retrieval works the same for every model. A vendor model without X-Moss-Provider-Key, or any other model name, returns 400. The provider key is sent only to that vendor and is never stored.

Limits

Per project:

Limit	`moss`	Vendor model
Requests per minute (sustained / burst)	120 / 10	300 / 50
Input tokens per minute	600,000	none
Prompt tokens per request	16,384	200,000
Indexes per request	32	32
Request body	256 KB	256 KB

On moss the 16,384 tokens are the model’s whole context: the prompt, the retrieved facts and the reply share them, so leave room for max_tokens. A prompt over it is 400 with type invalid_request_error and code: "context_length_exceeded". Over a per-minute limit the response is 429 with type requests or tokens and code: "rate_limit_exceeded". When Moss is at capacity it is 429 with type capacity and code: "server_overloaded". Both carry Retry-After, and the OpenAI SDKs retry on their own. A paused project gets 429 with code: "usage_limit_exceeded" and stays refused until it is unpaused in the portal.

429 - Too Many Requests

{
  "error": {
    "message": "Rate limit reached for project 1a2b3c: too many requests per minute. Please try again in 1s.",
    "type": "requests",
    "code": "rate_limit_exceeded"
  }
}

Errors

Errors use OpenAI’s {"error": {"message", "type", "code"}} shape.

Status	`type`	Cause
`400`	`invalid_request_error`	Malformed body, unknown model, vendor model without a key, more than 32 indexes. A prompt over the token limit has `code: "context_length_exceeded"`.
`400`	`model_error`	The model rejected the prompt.
`401`	`authentication_error`	Missing or wrong `Authorization` token.
`403`	`permission_error`	`X-Moss-Indexes` names an index outside this project.
`413`	`invalid_request_error`	Body over 256 KB.
`429`	`requests`, `tokens`, `capacity`, `usage_limit`	See Limits.
vendor `4xx`	`model_error`	The vendor’s own status, with its message after `The model rejected the prompt`. A bad key is `401` from OpenAI and Anthropic, and `400` from Google.
`502`	`model_error`	The model failed, or a vendor returned `5xx`. Retry with backoff.
`503`	`server_error`	Moss could not check the project. Retry with backoff.

A failure after a stream has started arrives as a final data: frame carrying error instead of choices, then data: [DONE].

Tool calls

tools, tool_choice and parallel_tool_calls take OpenAI’s shapes and go to the model. Your code runs the tools. A tools value that is not a list of objects returns 400. A streamed reply sends each call in pieces in delta.tool_calls, with index, id, function.name and function.arguments, then a chunk with finish_reason tool_calls. A reply without stream has the calls joined in choices[0].message.tool_calls, with the same finish_reason. Send the assistant message back unchanged, then one tool message per call, and call again:

{
  "messages": [
    {"role": "user", "content": "Where is order A1?"},
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": {"name": "lookup_order", "arguments": "{\"id\": \"A1\"}"}
      }]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "Shipped on Oct 20."}
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "lookup_order",
      "parameters": {"type": "object", "properties": {"id": {"type": "string"}}}
    }
  }]
}

A tool call can carry fields beyond OpenAI’s, such as Gemini’s extra_content. Keep them: the model needs them on the next turn.

MCP servers

A tools entry of type mcp names an MCP server. Moss Inference connects to it, offers its tools to the model, runs the calls the model makes, and streams the final answer. One request can hold your own tools and MCP servers together.

{
  "type": "mcp",
  "server_label": "crm",
  "server_url": "https://mcp.example.com/mcp",
  "headers": {"Authorization": "Bearer CRM_TOKEN"},
  "allowed_tools": ["lookup_order"]
}

Field	Required	Notes
`type`	yes	`mcp`
`server_label`	yes	A name for the server, different for each server in the request.
`server_url`	yes	A public `https` URL on port 443. Any other URL returns `400`.
`headers`	no	Headers sent to that server only, such as `Authorization`. They never reach the model. A public server, such as `https://mcp.deepwiki.com/mcp`, needs none.
`allowed_tools`	no	Tool names. The model sees only these tools of the server.
`transport`	no	`streamable_http`, the default, or `sse` for an older server.

MCP calls are not added to the message history. Keep what you need from the tool steps in your own messages. A request with MCP servers can run for a few minutes.

Tool steps

A stream carries one extra chunk before each MCP call and one after it, with an empty delta and a moss_tool_step:

data: {"object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "finish_reason": null}], "moss_tool_step": {"type": "call", "id": "call_1", "server_label": "crm", "name": "lookup_order", "arguments": "{\"id\": \"A1\"}"}}

data: {"object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "finish_reason": null}], "moss_tool_step": {"type": "result", "id": "call_1", "server_label": "crm", "name": "lookup_order", "is_error": false, "duration_ms": 41.2, "output": "Shipped on Oct 20."}}

type is call before the tool runs, with its arguments, and result after it, with is_error, duration_ms and output. Both have the same id, the server_label and the tool’s name. A reply without stream lists the steps in order in a top-level moss_tool_steps array. OpenAI clients ignore both fields.

MCP errors

Status	`type`	`code`	Cause
`400`	`invalid_request_error`	none	A bad `tools` entry, a `server_label` or tool name used twice, or too many servers or tools.
`400`	`mcp_error`	`mcp_server_error`	`server_url` is not a public `https` URL on port 443.
`424`	`mcp_error`	`mcp_server_error`	The server did not connect, refused the credentials, or failed to list its tools.
`429`	`capacity`	`mcp_loops_busy`	Too many MCP requests at once. Retry after `Retry-After`.

A tool may already have changed data when a call fails after it, so Moss Inference never retries that call. Without stream, it returns 424 with code failed_after_tool_call, the moss_tool_steps that ran, and the header x-should-retry: false. On a stream, it is an error event after a step, which the Moss SDKs report with the code failed_after_tool_call. Do not replay such a call.

Getting Started

Index Management

Web Sources

Document Operations

Inference

Chat Completions

Authentication

Request

Streaming

Examples

Models

Limits

Errors

Tool calls

MCP servers

Tool steps

MCP errors

​Authentication

​Request

​Streaming

​Examples

​Models

​Limits

​Errors

​Tool calls

​MCP servers

​Tool steps

​MCP errors

Authentication

Request

Streaming

Examples

Models

Limits

Errors

Tool calls

MCP servers

Tool steps

MCP errors