PDF parsing with OpenDataLoader

Upgrade PDF ingestion from flat text to structured document understanding.

Audience: operators who want better answers from PDF documents — especially tables, structured data, and multi-page documents. What you will accomplish: enable ODL, understand the three value layers, and diagnose issues.

What OpenDataLoader does

The default PDF parser (PyPDFLoader) extracts raw text — tables become garbled columns, headings lose their hierarchy, and lists become loose words. OpenDataLoader (ODL) solves this in three layers:

1. 1

   Step 1: Layer 1: Content

   ODL outputs Markdown — tables become pipe tables, headings become # markers, lists become bullet points. The LLM receives structured text instead of noise.

2. 2

   Step 2: Layer 2: Structure

   ODL’s JSON output exposes heading hierarchy, element types, and bounding boxes. The system builds two chunk levels: L1 (section-level) and L2 (element-level), with parent links for context expansion.

3. 3

   Step 3: Layer 3: Intelligence

   Element-aware retrieval boosts table chunks for table-like queries and section chunks for overview queries. Context expansion pulls the full section when a paragraph-level match is found.

Prerequisites

• Java 11+ must be on the server’s $PATH (java -version).
• The opendataloader_pdf Python package (pip install opendataloader-pdf).
• Both are optional — when absent, the system falls back to PyPDF automatically.

Quick start

ODL is auto-detected. When Java 11+ and the package are present, PDFs automatically use ODL without any configuration change:

curl -X POST http://127.0.0.1:8000/api/v1/ingest/upload \
-F "file=@annual_report.pdf"

Check the parser used:

curl http://127.0.0.1:8000/api/v1/ingest/status/annual_report

Look for "parser": "opendataloader" in the response. If you see "fallback_used": true, ODL failed and PyPDF was used instead.

Three tracks (value layers)

Track 1: Markdown content (default, no config needed)

When ODL is active, PDF chunks store Markdown fragments instead of raw text. The existing RecursiveCharacterTextSplitter splits on \n\n and \n, which Markdown headings and section breaks naturally provide.

What improves:

• Tables appear as pipe tables (| Column | Value |)
• Headings preserve hierarchy (# Title, ## Section)
• Lists stay as bullet points (- item)

Track 2: Hierarchical chunking + retrieval (opt-in)

Enable by setting RETRIEVAL_STRATEGY=hierarchical.

Two chunk levels are indexed:

L2 chunks carry a parent_chunk_id pointing to their L1 section. The retrieval strategy:

1. Fetches both L1 and L2 candidates
2. Boosts table chunks when the query contains table-like terms (table, row, column, compare, vs)
3. Boosts L1 section chunks when the query contains overview terms (overview, summary, what is, about)
4. Expands L2 results to their L1 parent when the result set has room, giving the LLM full section context

Track 3: Hybrid OCR + enrichment (optional sidecar)

For scanned PDFs, formulas, and chart descriptions, run the hybrid sidecar:

docker compose --profile hybrid up

This starts an odl-hybrid service on port 5002. Enable in .env:

ODL_HYBRID=docling-fast
ODL_HYBRID_MODE=auto        # auto = triage (simple pages stay local, complex pages go to AI)
# or ODL_HYBRID_MODE=full   # all pages processed by the AI backend

Additional enrichment options:

ODL_ENRICH_FORMULA=true      # Extract LaTeX formulas (requires full mode)
ODL_ENRICH_PICTURES=true     # Describe charts and images (requires full mode)

Both enrichment flags require ODL_HYBRID_MODE=full — the AI backend must process every page to find formulas and pictures.

Configuration reference

Per-request overrides

When ingesting, you can override the parser and hybrid mode per request:

curl -X POST http://127.0.0.1:8000/api/v1/ingest \
-H "Content-Type: application/json" \
-d '{
  "file_name": "scan",
  "s3_url": "https://example.com/scan.pdf",
  "parser": "opendataloader",
  "hybrid_mode": "full",
  "pages": "1-10"
}'

Citations with ODL

When ODL fields are present, citations include additional context:

These fields are optional — old chunks without them simply show null, and older clients are unaffected.

Troubleshooting

Java not found

java -version
# → "java: command not found"

Install Java 11+ (see the operator guide). When PDF_PARSER_FALLBACK=true (default), the system silently falls back to PyPDF. Set PDF_PARSER_FALLBACK=false to make Java absence a hard error.

Hybrid server not reachable

If ODL_HYBRID is set but the sidecar is down:

• With ODL_HYBRID_FALLBACK=true → falls back to local Java (no OCR/enrichment)
• With ODL_HYBRID_FALLBACK=false → ingest fails with a clear error

Enrichment produces empty chunks

Formula and picture enrichment require ODL_HYBRID_MODE=full. In auto mode, the triage router may skip complex pages. The config validator raises a startup error if enrichment is enabled with auto mode.

Manually forcing PyPDF

Set PDF_PARSER=pypdf in .env or pass "parser": "pypdf" in the ingest request.

Hierarchical retrieval not activating

RETRIEVAL_STRATEGY=hierarchical is required. The default is mmr. Without this setting, L1/L2 chunks are retrieved but without element-type boosting or context expansion.

Related next steps

• See how retrieval strategies work in Retrieval.
• Configure all ODL variables in Configuration.
• Learn about document ingestion in Ingesting documents.
• Read the operator troubleshooting guide in the project repo: docs/odl-operator-guide.md.