Moss - Real-time Semantic Search for Conversational AI

Register a website as a source and start a crawl that builds an index from its pages. The crawler stays on the site, uses its sitemap in addition to following links, respects robots.txt by default, and extracts the main content of each page (navigation and footer boilerplate are dropped). Linked PDF and DOCX files found during the crawl are parsed into the same index. Returns 202 Accepted with a jobId; poll getJobStatus until the job completes. Set refreshCadence to re-crawl on a schedule, or omit it and trigger re-crawls manually with resyncWebSource. Required fields: rootUrl, indexName

Field	Type	Required	Notes
`rootUrl`	string	✅	Publicly reachable `http`/`https` URL to start crawling from. Private and internal addresses are rejected.
`indexName`	string	✅	Name of the index to build. One web source per index name.
`maxPages`	integer	▶︎	Page cap for the crawl. Default `500`, max `5000`.
`maxDepth`	integer	▶︎	Link depth from `rootUrl`. Default `3`, max `10`.
`maxDocuments`	integer	▶︎	Cap on indexed chunks. Default and max `50000`.
`includePaths`	string[]	▶︎	Only crawl matching path globs (e.g. `["/blog/*"]`). Up to 50 entries. Omitted from responses when not set.
`excludePaths`	string[]	▶︎	Skip matching path globs. Up to 50 entries. Omitted from responses when not set.
`respectRobots`	boolean	▶︎	Honor robots.txt. Default `true`.
`parseDocuments`	boolean	▶︎	Parse linked PDF/DOCX files into the index. Default `true`.
`refreshCadence`	string	▶︎	`daily` or `weekly`. Omit for manual-only re-crawls.
`modelId`	string	▶︎	Embedding model, fixed at create time. Default `moss-minilm`; `custom` is not supported.

Example request

curl -X POST "https://service.usemoss.dev/v1/manage" \
  -H "Content-Type: application/json" \
  -H "x-service-version: v1" \
  -H "x-project-key: moss_access_key_xxxxx" \
  -d '{
    "action": "createWebSource",
    "projectId": "project_123",
    "rootUrl": "https://docs.yoursite.com",
    "indexName": "docs-site",
    "maxPages": 500,
    "refreshCadence": "weekly"
  }'

Responses

{
  "id": "7f2a9c04-3b1e-4c8a-9d2f-1e5b6a7c8d90",
  "indexName": "docs-site",
  "rootUrl": "https://docs.yoursite.com",
  "maxPages": 500,
  "maxDocuments": 50000,
  "maxDepth": 3,
  "respectRobots": true,
  "parseDocuments": true,
  "refreshCadence": "weekly",
  "nextRefreshAt": "2026-09-08T10:00:00.000+00:00",
  "lastCrawledAt": null,
  "lastPageCount": null,
  "lastDocCount": null,
  "status": "crawling",
  "lastErrorCode": null,
  "createdAt": "2026-09-01T10:00:00.000+00:00",
  "jobId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}

Field	Type	Notes
`id`	string	Web source ID. Use it with `getWebSource`, `resyncWebSource`, `updateWebSource`, and `deleteWebSource`.
`jobId`	string	Poll with `getJobStatus`. Crawl jobs move through the phases `crawling`, `parsing_documents` (when linked files are found), and `building_index`.
`status`	string	`crawling` while a run is active; otherwise `idle` or `failed`.
`refreshCadence`	string	`daily`, `weekly`, or `manual`.
`nextRefreshAt`	string \| null	When the next scheduled re-crawl becomes due, or `null` for manual sources.
`lastCrawledAt`, `lastPageCount`, `lastDocCount`	mixed	Stats from the most recent completed run.
`lastErrorCode`	string \| null	Error code from the last failed run, e.g. `CRAWL_FAILED`, `CRAWL_LIMIT_EXCEEDED`, `CRAWL_ROBOTS_DISALLOWED`, `CRAWL_UNREACHABLE_HOST`.

Linked documents When parseDocuments is true, PDF and DOCX links discovered during the crawl are parsed into the index. Limits: 50 MB per file and at most 20 linked documents per crawl run. Oversized, unreachable, or unsupported files (e.g. .doc, .xlsx, .pptx) are skipped and the crawl still completes. Errors

400 when rootUrl or indexName is missing, a limit is out of range, rootUrl is not a public website address, or modelId is not supported for website crawling.
403 when the model is not enabled for your organization.
409 when a build is already in progress for indexName.
503 when website crawling is temporarily unavailable.

Getting Started

Index Management

Web Sources

Document Operations

Create Web Source