Moss - Real-time Semantic Search for Conversational AI

Sub-200ms time to first token, with retrieval built in. Moss Inference is a low-latency inference API built for real-time AI agents. It brings optimized model inference and context retrieval together, enabling agents to start responding in under 200ms on supported fast-lane workloads. Run Moss’s optimized inference model or access models from OpenAI, Anthropic, Google, and other providers through a single OpenAI-compatible endpoint. Ground responses in your Moss indexes without building or managing a separate retrieval pipeline.

Private beta: Moss Inference is currently in private beta. Request access through the Moss portal.

Why Moss Inference?

Traditional AI applications retrieve context and generate responses through separate systems. Even after retrieving the right information, an agent still needs to send that context to an LLM and wait for generation to begin. Moss Inference brings retrieval and generation closer together, reducing the overhead between finding relevant information and generating a response.

Sub-200ms time to first token

Start generating responses in under 200ms with Moss’s optimized inference model on supported workloads. Built for applications where response latency matters, including conversational AI, interactive agents, and real-time workflows.

Retrieval and inference, colocated

Moss runs its optimized inference model alongside its retrieval runtime, allowing relevant context to be retrieved and used without a separate network round trip between the retrieval and generation systems. Specify a Moss index in your inference request, and Moss retrieves relevant context to ground the model’s response.

One OpenAI-compatible API

Access Moss’s optimized model and supported third-party models through a single endpoint. Use the OpenAI-compatible Responses API, with an optional indexes parameter to enable context retrieval. You can use Moss for inference alone or combine it with your existing Moss indexes.

How Moss Inference works

Moss Inference combines context retrieval and model execution into a single request.

Send a request. Your agent calls the Moss Inference API, specifying a model and optionally one or more Moss indexes.
Retrieve relevant context. When indexes are specified, Moss searches them and adds relevant information to the model’s input.
Generate a response. The selected model generates a response, which can be streamed back to your application.

With Moss’s optimized model, retrieval and inference run on colocated infrastructure. When using a third-party model, Moss provides access through the same API, with optional retrieval grounding. API endpoint https://inference.moss.dev/v1

Available models

Moss Inference supports two ways to run inference.

Model type	Description
Moss optimized model	Moss-hosted inference optimized for low-latency response generation, with colocated retrieval
Third-party models	Access supported models from OpenAI, Anthropic, Google, and other providers through the Moss endpoint

Both options can use your Moss indexes to provide relevant context during inference. Models lists every id with its prompt limit and prices.

How it fits

You need	Moss Inference gives you
A model	`moss/fastlane-1` on Moss GPUs, or a vendor model, on one project key
Time to first token	Under 200 ms on `moss/fastlane-1`, grounded. A vendor model adds its own latency
Grounding	A search of the indexes a call names, hits added to the prompt
Tools	OpenAI function tools run by your code, MCP servers run by Moss
Streaming	OpenAI’s Responses events or Chat Completions chunks
A client	Any OpenAI client with a new base URL, or a Moss SDK on seven platforms
Billing	Credits per token and per retrieval, or vendor-billed tokens on your own key

Make one call

Run this after the Quickstart, which creates the faq index it searches.

curl -N https://inference.moss.dev/v1/responses \
  -H "Authorization: Bearer $MOSS_PROJECT_ID:$MOSS_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Moss-Indexes: faq" \
  -d '{
    "model": "moss/fastlane-1",
    "stream": true,
    "instructions": "You are the support agent for Acme.",
    "input": "Do you ship to Canada?"
  }'

import asyncio
import os

from moss import MossInference

client = MossInference(os.environ["MOSS_PROJECT_ID"], os.environ["MOSS_PROJECT_KEY"])


async def main():
    response = await client.responses.create(
        model="moss/fastlane-1",
        instructions="You are the support agent for Acme.",
        input="Do you ship to Canada?",
        indexes=["faq"],
    )
    print(response.output_text)


asyncio.run(main())

Moss searches the faq index for the question, adds what it finds to the prompt, and answers. The curl call streams the reply. The Python client installs with pip install moss; SDKs lists the other six.

What can you build?

Real-time voice agents Reduce the delay before an agent starts responding while retrieving relevant information from conversation context and business knowledge. Browser and computer-use agents Support agents that interact with websites and applications, using retrieved instructions, task context, and relevant knowledge to inform their next actions. Long-running agents Give agents access to relevant information from previous steps and persistent indexes as they work through multi-step tasks. AI copilots and assistants Build responsive applications that answer questions using your own documents, product information, and business knowledge.

Choose how to start

Quickstart

Get a grounded, streamed reply in five minutes.

Models

Every model, with its prompt limit, prices and notes.

Use an OpenAI client

Change the base URL and the key. What differs from OpenAI.

SDKs

Inference clients for seven platforms.

Terms

Term	Meaning	Called elsewhere
Index	A searchable set of your documents, built with the Moss SDK or API.	Vector store, knowledge base
Retrieval	The search of your indexes before the model answers, one per request.	RAG, grounding
Time to first token	The time from sending the request to the first streamed token, not to the whole reply.	TTFT, latency
Project id and key	The two portal credentials that together form the bearer token.	API key
Moss key	A vendor model called on Moss’s vendor account and billed to your credits.	Hosted model
Your own key	A vendor model called on your vendor account, with the key sent in `X-Moss-Provider-Key`.	BYOK
Model id	`provider/model`, such as `moss/fastlane-1` or `anthropic/claude-haiku-5.5`.	Model string
Reply cache	A repeated request answered without a model call, free.	Response cache
Prefix cache	Prompt tokens `moss/fastlane-1` still holds from an earlier call, billed at the cached price.	Prompt caching

In this section

Quickstart: key, install, one call, stream it, ground it.
Models: the table, routes, the Moss model, refused parameters.
Pricing: credits, tokens, caching, retrievals.
OpenAI compatibility: base URL, endpoints, headers, what differs.
Ground answers in your indexes: choose indexes, turn retrieval off, what it costs.
Streaming: events, chunks, stopping early, errors after the stream starts.
Tool calling: the loop, both shapes, structured output.
MCP servers: servers Moss runs for the model.
Bring your own key: vendor models on your account.
Errors and limits: every status and code, limits, retries.
SDKs: the seven inference clients.

Get started

Guides

SDKs

Reference

Moss Inference

Why Moss Inference?

Sub-200ms time to first token

Retrieval and inference, colocated

One OpenAI-compatible API

How Moss Inference works

Available models

How it fits

Make one call

What can you build?

Choose how to start

Quickstart

Models

Use an OpenAI client

SDKs

Terms

In this section

​Why Moss Inference?

​Sub-200ms time to first token

​Retrieval and inference, colocated

​One OpenAI-compatible API

​How Moss Inference works

​Available models

​How it fits

​Make one call

​What can you build?

​Choose how to start

Quickstart

Models

Use an OpenAI client

SDKs

​Terms

​In this section

Why Moss Inference?

Sub-200ms time to first token

Retrieval and inference, colocated

One OpenAI-compatible API

How Moss Inference works

Available models

How it fits

Make one call

What can you build?

Choose how to start

Terms

In this section