GuidesIngesting documents

Ingesting documents

Give the assistant something to ground its answers on.

Audience: operators and developers managing the knowledge base. What you will accomplish: ingest a document (by URL or local upload) and confirm it is searchable.

Supported formats

PDF, TXT, Markdown (.md / .markdown), DOCX, and HTML (.html / .htm). The format is inferred from the file or URL extension.

PDF parser selection

By default, PDFs are parsed with OpenDataLoader when Java 11+ and the opendataloader_pdf package are present — no configuration required. ODL preserves Markdown structure (tables, headings, lists) for better RAG quality. When ODL is unavailable, the system falls back to PyPDF automatically.

  • Force ODL: PDF_PARSER=opendataloader in .env or "parser": "opendataloader" in the request
  • Force PyPDF: PDF_PARSER=pypdf in .env or "parser": "pypdf" in the request
  • Disable fallback: PDF_PARSER_FALLBACK=false makes ODL failure a hard error

See PDF parsing with OpenDataLoader for the three-track feature set (Markdown content, hierarchical chunking, and hybrid OCR/enrichment).

How ingestion works

Ingestion on /api/v1 is asynchronous: the request returns 202 Accepted with a Location header and status: "queued", then processes in the background. You poll the status endpoint for progress.

  1. Step 1: Submit the document

    Either point the server at a public HTTPS URL, or upload a local file directly (no public URL needed — privacy-friendly).

  2. Step 2: Poll the status

    Call the status endpoint until the document reaches a terminal state.

  3. Step 3: Confirm it is searchable

    Once status is done, ask a question it should answer and check the citations point at your document.

Ingest by URL

curl
curl -X POST http://127.0.0.1:8000/api/v1/ingest \
-H "Content-Type: application/json" \
-d '{"file_name":"returns","s3_url":"https://example.com/returns.pdf"}'

If it fails: The URL must be a public https link ending in a supported extension. A 401 means a key is required (see auth below).

Upload a local file

No public URL required — the file never has to be hosted anywhere.

curl
curl -X POST http://127.0.0.1:8000/api/v1/ingest/upload \
-F "file=@/path/to/returns.docx"
# optional explicit id: -F "file_name=returns"

If it fails: The file must be a supported format (validated by extension; PDFs also get a %PDF header check) and within the server’s MAX_FILE_SIZE_MB.

Per-request parser overrides

When ingesting a PDF, you can override the parser and hybrid mode for that specific document:

curl
curl -X POST http://127.0.0.1:8000/api/v1/ingest \
-H "Content-Type: application/json" \
-d '{
  "file_name":"scan",
  "s3_url":"https://example.com/scan.pdf",
  "parser": "opendataloader",
  "hybrid_mode": "full",
  "pages": "1-10"
}'
FieldValuesDescription
parserpypdf / opendataloaderForce a specific parser for this document.
hybrid_modeauto / fullOverride the deployment default for this document.
pagesstringProcess only specific pages, e.g. "1-10" or "1,3,5-7".

These overrides are threaded through the full pipeline, including the durable queue worker.

Poll status, list, and delete

curl
# Poll status
curl http://127.0.0.1:8000/api/v1/ingest/status/returns

# List (paginated)
curl "http://127.0.0.1:8000/api/v1/ingest/docs?limit=50&cursor=0"

# Delete (always needs X-API-Key)
curl -X DELETE http://127.0.0.1:8000/api/v1/ingest/returns -H "X-API-Key: <key>"

Status values

StatusMeaning
queuedAccepted; processing in the background.
doneEmbedded and searchable.
skippedUnchanged or duplicate content (the same file under a different name is caught by content hash).
failedProcessing failed; resubmit or check the document.

When using OpenDataLoader, the status response includes additional fields:

FieldValuesMeaning
parseropendataloader / pypdfWhich parser was used.
fallback_usedtrue / falseWhether ODL failed and PyPDF was used instead.
parser_modelocal / hybridLocal Java processing or hybrid AI backend.
page_countintegerNumber of pages in the document.
element_countintegerNumber of structural elements (ODL) or chunks (PyPDF).

Authentication

EndpointKey required?
DELETE /api/v1/ingest/{doc_id}Always (when API_KEY is set).
POST /api/v1/ingest, /ingest/upload, GET /ingest/docsOnly when REQUIRE_AUTH_FOR_INGEST=true.

Pass the key in the X-API-Key header. When API_KEY is empty, auth is skipped (dev mode).

Verify your result

  • Verify: The ingest call returns 202 with a Location header and status: "queued".
  • Verify: Polling the status endpoint eventually returns done (or skipped for duplicates).
  • Verify: A question the document answers now returns it in sources[].
  • Verify: DELETE with a valid X-API-Key removes the document.

Common mistakes and fixes

  • 401 on DELETE → DELETE always requires X-API-Key when a key is configured.
  • skipped instead of done → the content is unchanged or a duplicate; this is expected and not an error.
  • Invalid file_name → remove dots/slashes and keep it ≤128 chars.

Related next steps

Trust & citationsPDF parsing with ODL