Moss - Real-time Semantic Search for Conversational AI

Elixir SDK / Inference Moss.Inference.Chat.Completions.create/3 sends OpenAI’s chat completions body, as a map, to Moss Inference, which searches the indexes you name, adds what it finds to the prompt and replies with OpenAI’s field names as string keys. Code written for OpenAI keeps its body and adds :indexes.

Requires moss 1.10.0 or later, installed as in Install.

Create a client

project_id = System.fetch_env!("MOSS_PROJECT_ID")
project_key = System.fetch_env!("MOSS_PROJECT_KEY")

client = Moss.Inference.new(project_id, project_key)

The project id and key come from the Moss portal. Build one client and reuse it for every call, from any process. It is plain data with nothing to close.

Get a reply

alias Moss.Inference.Chat.Completions

messages = [%{role: "user", content: "Do you ship to Canada?"}]

{:ok, completion} = Completions.create(client, %{messages: messages}, indexes: ["faq"])

IO.puts(hd(completion["choices"])["message"]["content"])

completion["request_id"] is the response’s X-Moss-Request-Id, and so is stream.request_id on a stream. Include it when you report a problem. Both are nil when the response has none.

Stream a reply

{:ok, stream} =
  Completions.create(client, %{messages: messages, stream: true}, indexes: ["faq"])

for chunk <- stream, choice <- chunk["choices"] do
  IO.write(choice["delta"]["content"] || "")
end

Read the stream once, in the process that called create/3. To stop early, halt it, for example with Enum.take/2, or call close/1 from any process: either closes the connection, which stops the model.

Moss.Inference.ChatCompletionStream.close(stream)

Only the last chunk carries usage, and it has no choices, which is why the comprehension above skips it.

Errors

A failed call gives a Moss.Inference.Error: new/3 raises it, create/3 returns it as {:error, error}, and a stream raises it while you read it. status is the HTTP status of a refused request, type names the failure, and message describes it. code is the error code Moss Inference sent, such as "rate_limit_exceeded", request_id is the response’s X-Moss-Request-Id, and retry_after is the wait in milliseconds that a Retry-After header asks for. A field the failure lacks is nil. The SDK sets these types itself, with status nil:

`type`	Cause
`invalid_request_error`	An argument the SDK rejects before sending, such as an empty key, an index name that is empty or holds a comma, or an option inside `params`, such as `:api_key` or `apiKey`.
`connection_error`	Moss Inference could not be reached, or the connection broke.
`timeout`	A time limit ran out, such as `:timeout` or `:idle_timeout`.
`incomplete_stream`	The stream ended before `[DONE]`.

Every other error reports what Moss Inference sent. See Errors, and Limits for a 429.

Tool calls

tools, tool_choice and parallel_tool_calls go in params as in OpenAI’s API, and your code runs the tools. With stream: true, each call arrives in pieces in choice["delta"]["tool_calls"]. Without it, create/3 joins them into the reply’s message, with "" for a missing id, name or arguments. Append that message to messages as it is, add one tool message per call, and call again, as in Tool calls.

lookup = %{
  type: "function",
  function: %{name: "lookup_order", parameters: %{type: "object", properties: %{}}}
}

{:ok, completion} =
  Completions.create(client, %{messages: messages, tools: [lookup]}, indexes: ["faq"])

message = hd(completion["choices"])["message"]

results =
  for call <- message["tool_calls"] || [] do
    %{role: "tool", tool_call_id: call["id"], content: lookup_order(call["function"]["arguments"])}
  end

params = %{messages: messages ++ [message | results], tools: [lookup]}
{:ok, completion} = Completions.create(client, params, indexes: ["faq"])

lookup_order/1 stands for your own code. It gets the arguments as a JSON string and returns the result as a string.

MCP servers

A Moss.Inference.McpServer in tools lets Moss Inference call that server’s tools itself during the turn.

crm = %Moss.Inference.McpServer{
  server_label: "crm",
  server_url: "https://mcp.example.com/mcp",
  headers: %{"Authorization" => "Bearer " <> System.fetch_env!("CRM_TOKEN")}
}

{:ok, stream} =
  Completions.create(client, %{messages: messages, tools: [crm], stream: true}, indexes: ["faq"])

for chunk <- stream do
  case chunk do
    %{"moss_tool_step" => step} ->
      IO.puts("\n#{step["type"]} #{step["server_label"]}.#{step["name"]}")

    %{"choices" => choices} ->
      for choice <- choices, do: IO.write(choice["delta"]["content"] || "")
  end
end

Steps arrive in chunk["moss_tool_step"] and completion["moss_tool_steps"], and MCP failures give a Moss.Inference.Error with status, type, code and retry_after, as in MCP servers.

Options

Moss.Inference.new(project_id, project_key, opts):

Option	Default	Notes
`:api_key`	none	Your own OpenAI, Anthropic or Google key, for a vendor `model` such as `gpt-4o-mini`. See Models. It is sent as the `X-Moss-Provider-Key` header, never in the body. An empty key raises `Moss.Inference.Error`.
`:timeout`	none	Milliseconds for a whole call, the stream included. `:infinity` turns it off.
`:idle_timeout`	`120_000`	Milliseconds to wait for the reply to start, and for each pause in it. `:infinity` turns it off.
`:max_retries`	`2`	Retries before the reply starts. `0` turns them off.

Moss.Inference.Chat.Completions.create(client, params, opts):

Argument	Default	Notes
`params`	required	The OpenAI request body as a map, with atom or string keys. It needs `messages`.
`model` in `params`	`"moss"`	The Moss model, or a vendor model with `:api_key`. See Models.
`stream` in `params`	`false`	`true` returns `{:ok, stream}` once the reply starts, an `Enumerable` of `chat.completion.chunk` maps, otherwise `{:ok, completion}`, one `chat.completion` map.
`:indexes`	every index of the project	Names of the indexes to search, at most 32. Required when the project has more than 32. An empty list skips the search, as `retrieve: false` does.
`:retrieve`	`true`	`false` skips the search, and the model answers from the messages alone.
`:cache`	`true`	Moss can answer an identical request from its reply cache. `false` always asks the model.
`:api_key`	the client’s	Your provider key for this call, in place of the client’s.
`:timeout`, `:idle_timeout`	the client’s	This call’s limits, in place of the client’s. `:infinity` turns one off.
`:max_retries`	the client’s	This call’s retries, in place of the client’s.

Python

JavaScript

Swift (iOS)

Android

Elixir

Browser

C

Inference

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options

​Create a client

​Get a reply

​Stream a reply

​Errors

​Tool calls

​MCP servers

​Options

Create a client

Get a reply

Stream a reply

Errors

Tool calls

MCP servers

Options