Moss - Real-time Semantic Search for Conversational AI

MossInference is a drop-in for OpenAI’s client: its chat.completions.create takes OpenAI’s arguments plus indexes and returns the same shapes. Each call goes to Moss Inference, which searches those indexes and adds what it finds to the prompt.

Requires @moss-js/moss 1.21.0 or later, installed as in Install.

Create a client

import { MossInference } from '@moss-js/moss'

const client = new MossInference(process.env.MOSS_PROJECT_ID!, process.env.MOSS_PROJECT_KEY!)

The project id and key come from the Moss portal. Build one client and reuse it for every call. It has nothing to close. With MOSS_PROJECT_ID and MOSS_PROJECT_KEY set, import { inference } from '@moss-js/moss' gives the same inference.chat.completions.create with no client to build.

Get a reply

const completion = await client.chat.completions.create({
  messages: [
    { role: 'system', content: 'You are the support agent for Acme.' },
    { role: 'user', content: 'Do you ship to Canada?' },
  ],
  indexes: ['faq', 'policies'],
})
console.log(completion.choices[0].message.content)

completion.requestId is the response’s X-Moss-Request-Id, and so is stream.requestId on a stream. Include it when you report a problem. Both are undefined when the response has none.

Stream a reply

const stream = await client.chat.completions.create({
  messages: [{ role: 'user', content: 'Do you ship to Canada?' }],
  indexes: ['faq'],
  stream: true,
})
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? '')
}

To stop early, leave the loop or call stream.controller.abort(): either closes the connection, which stops the model, and neither throws. Only the last chunk carries usage, and it has no choices, which is why the loop reads choices[0]?..

Errors

Every failure other than an abort throws InferenceError, from the for await loop once a stream has started. status is the HTTP status of a refused request, type names the failure, and message describes it. errorCode is the error code Moss Inference sent, such as "rate_limit_exceeded", requestId is the response’s X-Moss-Request-Id, and retryAfter is the wait in milliseconds that a Retry-After header asks for. A field the failure lacks is undefined. The error’s Moss code lets isMossError accept it. The SDK sets these types itself, with status undefined:

`type`	Cause
`invalid_request_error`	An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, or an unset `MOSS_PROJECT_ID` for `inference`.
`connection_error`	Moss Inference could not be reached, or the connection broke.
`timeout`	A time limit ran out, such as `timeout` or `idleTimeout`.
`incomplete_stream`	The stream ended before `[DONE]`.

Every other error reports what Moss Inference sent. See Errors, and Limits for a 429.

Tool calls

tools, tool_choice and parallel_tool_calls take OpenAI’s shapes. Stream the reply, then stream.finalChatCompletion() joins it into one completion. Push its message onto messages, add one tool message per call, and call again until the reply has no tool calls, as in Tool calls:

import type { ChatCompletionMessageParam, ChatCompletionTool } from '@moss-js/moss'

const tools: ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'check_stock',
      description: 'How many units of a product are in stock.',
      parameters: { type: 'object', properties: { sku: { type: 'string' } }, required: ['sku'] },
    },
  },
]
const messages: ChatCompletionMessageParam[] = [
  { role: 'user', content: 'Is the EQB jacket in stock?' },
]

for (;;) {
  const stream = await client.chat.completions.create({ messages, tools, stream: true })
  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta.content ?? '')
  }
  const { message } = (await stream.finalChatCompletion()).choices[0]
  messages.push(message)
  if (!message.tool_calls?.length) break
  for (const call of message.tool_calls) {
    const { sku } = JSON.parse(call.function.arguments)
    messages.push({ role: 'tool', tool_call_id: call.id, content: await checkStock(sku) })
  }
}

checkStock stands for your own code. Push the message unchanged, so each tool call keeps every field the model sent. finalChatCompletion() also works without the for await loop. Without stream: true, create returns the joined completion directly.

MCP servers

An McpServer entry in tools names an MCP server whose tools Moss Inference calls for the model during the turn. The stream carries each tool step, then the answer:

import type { McpServer } from '@moss-js/moss'

const crm: McpServer = {
  type: 'mcp',
  server_label: 'crm',
  server_url: 'https://mcp.example.com/mcp',
  headers: { Authorization: `Bearer ${process.env.CRM_TOKEN}` },
}
const stream = await client.chat.completions.create({
  messages: [{ role: 'user', content: 'Where is order A1?' }],
  tools: [crm],
  stream: true,
})
for await (const chunk of stream) {
  const step = chunk.moss_tool_step
  if (step) console.log(step.type, step.server_label, step.name)
  else process.stdout.write(chunk.choices[0]?.delta.content ?? '')
}
const { moss_tool_steps } = await stream.finalChatCompletion()
console.log(moss_tool_steps)

Steps arrive in chunk.moss_tool_step and completion.moss_tool_steps, and MCP failures throw InferenceError with status, type, errorCode and retryAfter, as in MCP servers.

Options

new MossInference(projectId, projectKey, options?):

Option	Default	Notes
`apiKey`	none	Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See Models. It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key throws `InferenceError`.
`timeout`	none	Milliseconds for a whole call, the stream included. `null` turns it off.
`idleTimeout`	`120_000`	Milliseconds to wait for the reply to start, and for each pause in it. `null` turns it off.
`maxRetries`	`2`	Retries before the reply starts. `0` turns them off.
`headers`	none	Headers sent with every call.

chat.completions.create(params, options?) takes OpenAI’s fields and the Moss fields in params, and settings for this call only in options:

Option	Default	Notes
`model`	`"moss"`	The Moss model, or a vendor model with `apiKey`. See Models.
`messages`	required	OpenAI’s message list.
`stream`	`false`	`true` resolves to a `ChatCompletionStream` of chunks, otherwise to one `ChatCompletion`.
`indexes`	every index of the project	Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve: false` does.
`cache`	`true`	Moss can answer an identical request from its reply cache. `false` always asks the model.
`retrieve`	`true`	`false` skips the search, and the model answers from the messages alone.
Other OpenAI fields	none	`temperature`, `max_tokens`, `tools` and the rest go to the model. An `apiKey` or `api_key` field throws.
`options.apiKey`	the client’s	Your provider key for this call, in place of the client’s.
`options.timeout`, `options.idleTimeout`	the client’s	This call’s limits, in place of the client’s. `null` turns one off.
`options.maxRetries`	the client’s	This call’s retries, in place of the client’s.
`options.headers`	none	Headers for this call. They add to the client’s or replace them by name.
`options.signal`	none	An `AbortSignal` that cancels the call.

Python

JavaScript

Swift (iOS)

Android

Elixir

Browser

C

Inference

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options

​Create a client

​Get a reply

​Stream a reply

​Errors

​Tool calls

​MCP servers

​Options

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options