Moss - Real-time Semantic Search for Conversational AI

MossInference is a Kotlin client for Moss Inference, which searches your indexes in the cloud and answers in OpenAI’s chat completions shape, with camelCase fields. chat.completions.create returns the whole reply, and createStreaming streams it.

Requires the moss-android AAR 0.28.0 or later, installed as in Install.

Create a client

import dev.usemoss.moss.MossInference

val inference = MossInference(projectId = projectId, projectKey = projectKey)

The project id and key come from the Moss portal. Build one client and reuse it for every call, from any thread. Calls and stream reads block, and throw MossException on the main thread, so run them from Dispatchers.IO.

Get a reply

import dev.usemoss.moss.ChatMessage
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

val messages = listOf(
    ChatMessage.system("You are the support agent for Acme."),
    ChatMessage.user("Do you ship to Canada?"),
)

val completion = withContext(Dispatchers.IO) {
    inference.chat.completions.create(messages, indexes = listOf("faq"))
}
println(completion.choices[0].message.content)

completion.requestId is the response’s X-Moss-Request-Id, and so is stream.requestId on a stream. Include it when you report a problem. Both are null when the response has none.

Stream a reply

import kotlinx.coroutines.ensureActive

withContext(Dispatchers.IO) {
    inference.chat.completions.createStreaming(messages, indexes = listOf("faq")).use { stream ->
        for (chunk in stream) {
            ensureActive()
            chunk.choices.firstOrNull()?.delta?.content?.let(::print)
        }
    }
}

To stop early, leave use with a break or return: that closes the connection, which stops the model. Cancelling the coroutine alone does not, so the loop calls ensureActive(), whose throw leaves use the same way. Only the last chunk carries usage, and it has no choices, which is why the loop reads choices.firstOrNull().

Errors

A failed call throws InferenceError, from the for loop once a stream has started. status is the HTTP status of a refused request, type names the failure, and message describes it. code is the error code Moss Inference sent, such as "rate_limit_exceeded", requestId is the response’s X-Moss-Request-Id, and retryAfterMs is the wait in milliseconds that a Retry-After header asks for. A field the failure lacks is null. The SDK raises these itself, the last three as an InferenceError with that type and status null:

Error	Cause
`IllegalArgumentException`	An argument the SDK rejects before sending, such as a blank key, or an index name that is blank or holds a comma.
`connection_error`	Moss Inference could not be reached, or the connection broke.
`timeout`	A time limit ran out, such as `timeoutMs` or `idleTimeoutMs`.
`incomplete_stream`	The stream ended before `[DONE]`.

Every other error reports what Moss Inference sent. See Errors, and Limits for a 429.

Tool calls

Pass tools, and the model may answer with toolCalls. create reads the whole stream and joins each call’s arguments. Append the reply with ChatMessage.assistant(reply), which sends it back unchanged, then one ChatMessage.tool(callId, content) per call, and call again, as in Tool calls:

import dev.usemoss.moss.Tool

val tools = listOf(
    Tool.function(
        name = "lookup_order",
        description = "Finds an order by its id.",
        parameters = """{"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}""",
    ),
)

val answer = withContext(Dispatchers.IO) {
    val chat = messages.toMutableList()
    val reply = inference.chat.completions.create(chat, tools = tools).choices[0].message
    chat += ChatMessage.assistant(reply)
    for (call in reply.toolCalls.orEmpty()) {
        chat += ChatMessage.tool(call.id.orEmpty(), lookupOrder(call.function?.arguments.orEmpty()))
    }
    inference.chat.completions.create(chat, tools = tools).choices[0].message.content
}
println(answer)

lookupOrder is your own function, which takes the call’s JSON arguments and returns the result as text. createStreaming streams each call in pieces, in delta.toolCalls, so use create for a reply you send back.

MCP servers

A Tool.mcp entry names an MCP server whose tools Moss Inference runs for the model during the same call, so your code sends no ChatMessage.tool for them:

import dev.usemoss.moss.McpServer

val crm = McpServer(
    serverLabel = "crm",
    serverUrl = "https://mcp.example.com/mcp",
    headers = mapOf("Authorization" to "Bearer $crmToken"),
)

withContext(Dispatchers.IO) {
    inference.chat.completions.createStreaming(messages, tools = listOf(Tool.mcp(crm))).use { stream ->
        for (chunk in stream) {
            ensureActive()
            chunk.toolStep?.let { println("${it.type}: ${it.serverLabel} ${it.name}") }
            chunk.choices.firstOrNull()?.delta?.content?.let(::print)
        }
    }
}

McpServer also takes allowedTools and transport, McpServer.Transport.STREAMABLE_HTTP or SSE. Steps arrive in chunk.toolStep and completion.toolSteps, with camelCase fields such as isError, and MCP failures throw InferenceError with status, type, code and retryAfterMs, as in MCP servers.

Options

MossInference(...):

Parameter	Default	Notes
`projectId`, `projectKey`	required	Your project id and key.
`apiKey`	none	Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See Models. It is sent as the `X-Moss-Provider-Key` header, never in the body. A blank key throws `IllegalArgumentException`.
`timeoutMs`	none	Milliseconds for a whole call, the stream included. `null` turns it off.
`idleTimeoutMs`	`120_000`	Milliseconds to wait for the reply to start, and for each pause in it. `null` turns it off.
`maxRetries`	`2`	Retries before the reply starts. `0` turns them off.

chat.completions.create and createStreaming take the same parameters, messages first:

Parameter	Default	Notes
`messages`	required	A `List<ChatMessage>`. `ChatMessage.system`, `user` and `assistant` build the common messages, `ChatMessage.assistant(reply)` sends a reply back, `ChatMessage.tool(callId, content)` answers a tool call, and `ChatMessage.raw` takes any other as the text of a JSON object.
`model`	`"moss"`	The Moss model, or a vendor model with `apiKey`. See Models.
`indexes`	every index of the project	Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve = false` does.
`retrieve`	`true`	`false` skips the search, and the model answers from the messages alone.
`cache`	`true`	Moss can answer an identical request from its reply cache. `false` always asks the model.
`apiKey`	the client’s	Your provider key for this call, in place of the client’s.
`extra`	none	Other OpenAI fields, such as `temperature` or `max_tokens`, as JSON values. The call’s `model` and `messages` win over `extra`, and so do `tools`, `toolChoice` and `parallelToolCalls` when given. `stream` is always `true`.
`tools`	none	A `List<Tool>`: `Tool.function` for a function of yours, `Tool.mcp` for an MCP server, and `Tool.raw` for any other tool object. See Tool calls and MCP servers.
`toolChoice`	none	`ToolChoice.AUTO`, `NONE`, `REQUIRED`, `ToolChoice.function(name)`, or `ToolChoice.raw(json)` for any other JSON object.
`parallelToolCalls`	none	Whether the model may call several tools at once.
`timeoutMs`, `idleTimeoutMs`	the client’s	This call’s limits, in place of the client’s. `null` turns one off.
`maxRetries`	the client’s	This call’s retries, in place of the client’s.

Python

JavaScript

Swift (iOS)

Android

Elixir

Browser

C

Inference

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options

​Create a client

​Get a reply

​Stream a reply

​Errors

​Tool calls

​MCP servers

​Options

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options