Local Document-AI Extraction Pipeline
An on-premises pipeline for structured extraction from scanned documents, with canonical item matching and a comparison dashboard — running fully locally on a GPU workstation.
Problem
The documents worth extracting structure from — quotes, invoices, procurement paperwork — tend to be exactly the ones an organisation will not send to a hosted API. They also tend to arrive as scans, so there is no text layer to parse, and the same item appears under a different description in every document. The pipeline had to solve all three at once: stay on-premises, handle image-only input, and reconcile descriptions across documents well enough to compare them line by line.
Approach
- ›Intake classifies each page by whether it carries a usable text layer, routing to pdfplumber or Tesseract accordingly rather than OCR-ing everything
- ›Both paths converge on the same representation — words with bounding boxes — so downstream stages do not care where the text came from
- ›A local LLM served via Ollama turns layout-aware text into schema-constrained JSON: line items, totals, dates, and parties
- ›Canonical matching normalises free-text item descriptions against a catalog, using an alias table for known variants and fuzzy matching for the rest
- ›A comparison dashboard diffs two documents item by item and flags missing entries, price drift, and quantity mismatches
Architecture
# Local Document-AI Extraction Pipeline
┌──────────────────────────────────────────────────┐
│ Document Intake │
│ │
│ scanned PDF ──► page split ──► classify │
│ (text layer? / image only?) │
└───────────┬──────────────────────┬───────────────┘
│ has text layer │ image only
▼ ▼
┌───────────────────┐ ┌──────────────────────┐
│ pdfplumber │ │ Tesseract OCR │
│ │ │ │
│ words + boxes │ │ words + boxes │
│ table regions │ │ confidence scores │
└───────────┬───────┘ └──────────┬───────────┘
└──────────┬───────────┘
▼
┌──────────────────────────────────────────────────┐
│ Local LLM Extraction (Ollama) │
│ │
│ layout text ──► schema-constrained JSON │
│ line items · totals · dates · parties │
└────────────────────────┬─────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Canonical Item Matching │
│ │
│ raw description ──► normalise ──► catalog id │
│ fuzzy + alias table for known variants │
└────────────────────────┬─────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Comparison Dashboard │
│ │
│ doc A vs doc B ──► per-item deltas │
│ flag: missing · price drift · qty mismatch │
└──────────────────────────────────────────────────┘
── everything above runs on one GPU workstation ──
no document leaves the machine
Tech Stack
pdfplumber
Text-layer & table extraction
Tesseract
OCR for scanned pages
Ollama
Local LLM serving
Python
Pipeline orchestration
GPU workstation
Inference host
Dashboard
Document comparison UI
Challenges
OCR noise reaching the model
A misread digit in a quantity column is indistinguishable from a correct one downstream. Confidence scores are carried through to extraction so low-confidence fields can be flagged rather than silently trusted.
Getting structured output from a local model
Smaller local models drift from a requested JSON shape more readily than hosted ones. Schema-constrained decoding plus a validation-and-retry step keeps output parseable without a larger model.
Same item, different words
Item descriptions vary by vendor, abbreviation, and typo. An alias table handles the recurring cases and fuzzy matching covers the tail, with unmatched items surfaced for review instead of dropped.
Staying inside the machine
Every stage — OCR, inference, storage — had to run locally, which meant sizing the model to the available GPU rather than to the task, and accepting the accuracy trade that comes with it.