> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moss.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Web Source

> Crawl a website and build an index from its pages.

Register a website as a source and start a crawl that builds an index from its pages. The
crawler stays on the site, uses its sitemap in addition to following links, respects
robots.txt by default, and extracts the main content of each page (navigation and footer
boilerplate are dropped). Linked PDF and DOCX files found during the crawl are parsed into
the same index.

Returns `202 Accepted` with a `jobId`; poll [`getJobStatus`](../index-management/getJobStatus)
until the job completes. Set `refreshCadence` to re-crawl on a schedule, or omit it and
trigger re-crawls manually with [`resyncWebSource`](./resyncWebSource).

**Required fields**: `rootUrl`, `indexName`

| Field            | Type      | Required | Notes                                                                                                       |
| ---------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `rootUrl`        | string    | ✅        | Publicly reachable `http`/`https` URL to start crawling from. Private and internal addresses are rejected.  |
| `indexName`      | string    | ✅        | Name of the index to build. One web source per index name.                                                  |
| `maxPages`       | integer   | ▶︎       | Page cap for the crawl. Default `500`, max `5000`.                                                          |
| `maxDepth`       | integer   | ▶︎       | Link depth from `rootUrl`. Default `3`, max `10`.                                                           |
| `maxDocuments`   | integer   | ▶︎       | Cap on indexed chunks. Default and max `50000`.                                                             |
| `includePaths`   | string\[] | ▶︎       | Only crawl matching path globs (e.g. `["/blog/*"]`). Up to 50 entries. Omitted from responses when not set. |
| `excludePaths`   | string\[] | ▶︎       | Skip matching path globs. Up to 50 entries. Omitted from responses when not set.                            |
| `respectRobots`  | boolean   | ▶︎       | Honor robots.txt. Default `true`.                                                                           |
| `parseDocuments` | boolean   | ▶︎       | Parse linked PDF/DOCX files into the index. Default `true`.                                                 |
| `refreshCadence` | string    | ▶︎       | `daily` or `weekly`. Omit for manual-only re-crawls.                                                        |
| `modelId`        | string    | ▶︎       | Embedding model, fixed at create time. Default `moss-minilm`; `custom` is not supported.                    |

**Example request**

```bash theme={null}
curl -X POST "https://service.usemoss.dev/v1/manage" \
  -H "Content-Type: application/json" \
  -H "x-service-version: v1" \
  -H "x-project-key: moss_access_key_xxxxx" \
  -d '{
    "action": "createWebSource",
    "projectId": "project_123",
    "rootUrl": "https://docs.yoursite.com",
    "indexName": "docs-site",
    "maxPages": 500,
    "refreshCadence": "weekly"
  }'
```

**Responses**

<CodeGroup>
  ```json 202 - Accepted theme={null}
  {
    "id": "7f2a9c04-3b1e-4c8a-9d2f-1e5b6a7c8d90",
    "indexName": "docs-site",
    "rootUrl": "https://docs.yoursite.com",
    "maxPages": 500,
    "maxDocuments": 50000,
    "maxDepth": 3,
    "respectRobots": true,
    "parseDocuments": true,
    "refreshCadence": "weekly",
    "nextRefreshAt": "2026-09-08T10:00:00.000+00:00",
    "lastCrawledAt": null,
    "lastPageCount": null,
    "lastDocCount": null,
    "status": "crawling",
    "lastErrorCode": null,
    "createdAt": "2026-09-01T10:00:00.000+00:00",
    "jobId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }
  ```
</CodeGroup>

| Field                                            | Type           | Notes                                                                                                                                                                                 |
| ------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                                             | string         | Web source ID. Use it with `getWebSource`, `resyncWebSource`, `updateWebSource`, and `deleteWebSource`.                                                                               |
| `jobId`                                          | string         | Poll with [`getJobStatus`](../index-management/getJobStatus). Crawl jobs move through the phases `crawling`, `parsing_documents` (when linked files are found), and `building_index`. |
| `status`                                         | string         | `crawling` while a run is active; otherwise `idle` or `failed`.                                                                                                                       |
| `refreshCadence`                                 | string         | `daily`, `weekly`, or `manual`.                                                                                                                                                       |
| `nextRefreshAt`                                  | string \| null | When the next scheduled re-crawl becomes due, or `null` for manual sources.                                                                                                           |
| `lastCrawledAt`, `lastPageCount`, `lastDocCount` | mixed          | Stats from the most recent completed run.                                                                                                                                             |
| `lastErrorCode`                                  | string \| null | Error code from the last failed run, e.g. `CRAWL_FAILED`, `CRAWL_LIMIT_EXCEEDED`, `CRAWL_ROBOTS_DISALLOWED`, `CRAWL_UNREACHABLE_HOST`.                                                |

**Linked documents**

When `parseDocuments` is `true`, PDF and DOCX links discovered during the crawl are parsed
into the index. Limits: 50 MB per file and at most 20 linked documents per crawl run.
Oversized, unreachable, or unsupported files (e.g. `.doc`, `.xlsx`, `.pptx`) are skipped and
the crawl still completes.

**Errors**

* `400` when `rootUrl` or `indexName` is missing, a limit is out of range, `rootUrl` is not
  a public website address, or `modelId` is not supported for website crawling.
* `403` when the model is not enabled for your organization.
* `409` when a build is already in progress for `indexName`.
* `503` when website crawling is temporarily unavailable.