What Is Intelligent Document Processing?
Intelligent Document Processing (IDP) is a software category that automatically classifies, extracts, validates, and routes structured data from business documents into downstream systems. At scale, manual extraction introduces errors, delays, and gaps in audit coverage. Sensible is a document extraction platform that uses SenseML, a declarative configuration language you version-control, to combine deterministic layout-based extraction with LLM parsing. The result is typed, schema-validated output through a single API endpoint regardless of document format or issuer.
IDP moves information from unstructured sources (PDFs, scanned images, email attachments, spreadsheets) into structured formats that business systems can consume: ERP platforms like SAP and NetSuite, CRM systems like Salesforce, loan origination platforms, and downstream APIs. The four-stage pipeline below covers how that happens in production.
How IDP Works: Four Stages
A complete IDP pipeline covers four stages. A gap in any stage compounds forward.
1. Ingest
Documents arrive from many sources: email attachments, file uploads, API calls, scanning hardware, shared drives, Slack. The ingest stage normalizes this input: OCR converts image-based pages to searchable text, multi-page PDFs are split or segmented by document type, and spreadsheets undergo format normalization before extraction runs.
OCR quality is the first reliability problem in IDP. Poor scan resolution, skewed pages, and low-contrast printing degrade text before extraction begins. Sensible's OCR pipeline pre-processes images and flags scans below a quality threshold rather than silently returning degraded output.
2. Classify
Documents must route to the right extraction logic before extraction runs. A W-2 and a 1099 contain similar fields in different positions. Running a W-2 through a 1099 config returns incorrect values without raising an error. Classification errors are silent: the system appears to be working until a downstream process fails.
Classification runs on layout signatures (fixed-position elements unique to a document type), text fingerprints (specific phrases that reliably identify a format), or LLM reasoning for document types without consistent identifiers. Accurate classification is a prerequisite for accurate extraction.
3. Extract
Extraction is where most of the complexity lives. The extraction stage runs a config against the classified document and returns a structured output object.
Document variability is the central engineering challenge. Documents vary on two dimensions: structure (how consistently fields appear at predictable positions) and variability (how many format revisions or issuer variants exist in production).
A W-2 sits near one extreme: the IRS standardizes the layout, field positions are predictable across all employers, and deterministic extraction handles the full volume without LLM inference. A legal contract sits near the other: no two look alike, clauses appear in different orders, and LLM-based methods handle the variability where deterministic rules cannot. Most business documents (invoices, bank statements, insurance policies) fall between these extremes.
4. Validate and Monitor
Extraction output that looks plausible is not always correct. The validation stage enforces schema rules: type checking (is this field a valid date? a parseable currency value?), cross-field logic (does the invoice total equal the sum of line items?), and range checks.
In Sensible, extractions that fall below a confidence threshold route to a human review queue rather than propagating wrong values downstream. Production IDP pipelines should also track extraction metrics over time to catch configurations that degrade as document formats evolve.
The four stages look straightforward in a demo environment. The difficulty surfaces in production, where document variability in the long tail, model updates, and silent extraction failures compound over months of operation. That gap is where most IDP evaluations break down.
IDP vs. OCR vs. RPA
OCR, RPA, and IDP address adjacent problems in document workflows and are frequently conflated, including by vendors selling all three.
OCR (Optical Character Recognition) converts a document image to raw machine-readable text. It does not interpret content. OCR output is a stream of characters with no field labels, no types, and no structure. OCR is a preprocessing step inside most IDP systems, not a substitute for them.
RPA (Robotic Process Automation) automates repetitive interactions with software interfaces: clicking buttons, copying values across screens, filling out forms. RPA moves data between systems once that data exists in structured form, but it cannot extract named fields from unstructured documents on its own. The common architecture pairs both: IDP extracts the data, RPA routes it into Workday, SAP, Epic, or other destination systems.
IDP combines OCR, classification, extraction logic, and validation into a single pipeline. The output is a typed, schema-validated JSON object ready for downstream consumption.
Note: Some RPA platforms maintain process audit logs; extraction provenance is a different capability: mapping each extracted field value back to its source coordinates in the original document.
The 90% Wall: Why Production IDP Is Harder Than It Looks
Most IDP systems reach 80–90% accuracy on a heterogeneous document set without significant tuning. Initial extractions work, demos look compelling, and early results justify investment.
Production environments require higher standards. A 90% accurate AP automation pipeline fails on 1 in 10 invoices. In a high-volume workflow, that translates directly to manual intervention costs that erode the automation ROI. In healthcare claims processing, a meaningful error rate triggers compliance exposure and denied reimbursements. In mortgage underwriting, an incorrect field extraction on a closing disclosure introduces liability at the loan level.
The gap between 80–90% and production-grade accuracy comes from four sources that vendors rarely name:
Vendors who claim 99% accuracy often measure on a curated document set in a controlled evaluation. Production accuracy on the full document mix, including edge cases, poor scans, and issuer variants not seen at evaluation time, tends to be lower.
Closing the gap requires two things working together: deterministic extraction logic for fields where the layout is predictable, and a validation layer that catches errors before they propagate rather than after.
For a deeper analysis of what it takes to reach production-grade accuracy, download Sensible's 2026 Buyer's Guide to Intelligent Document Processing.
How Sensible Addresses the Production Gap
Sensible is a document extraction platform that uses SenseML, a declarative config language, to combine layout-based extraction (boosted with ML) with LLM parsing. The architecture makes extraction behavior predictable and governable, not just accurate on average.
Layout-based extraction where the document structure allows it. For fields with predictable positions across document variants (the account number in a bank statement header, the total on an invoice, the named insured on a declaration page), Sensible uses layout-based methods that return the same value on every run. No prompt to maintain. No model to retrain.
LLM reasoning where the layout doesn't. For variable fields that require interpretation, payment terms that appear in different formats across vendors, special handling instructions that shift position by carrier, Sensible applies LLM-based extraction with schema enforcement on the output. The LLM handles the reasoning; the schema guarantees the shape and type of what comes back.
Model lifecycle managed by Sensible. When a foundation model updates, extraction behavior can change. Sensible manages model versioning so that configs continue working as underlying models update or deprecate. Prompt maintenance cycles are not part of the developer workflow.
Extraction provenance on every output. Each extracted value maps back to its source coordinates in the original document: page number, bounding box, anchor text. When a downstream system flags a discrepancy (a loan amount that doesn't match what a processor entered, a claim field that doesn't reconcile with the policy), provenance is what lets you trace the error back to the source document in seconds rather than re-reviewing the file manually. For teams in regulated workflows, it is the difference between an auditable system and one that cannot explain its own outputs.
SenseML configs are version-controlled code. They live in Git, go through code review, run in CI/CD pipelines, and deploy through a REST API or Python and Node SDKs. That means extraction logic gets the same engineering rigor as any other production system: reviewable, reversible, and auditable. No configuration UI that one person manages manually; no extraction behavior that cannot be explained or reproduced on demand. The full SenseML reference covers every available extraction method.
IDP Use Cases by Industry
Financial Services
Banks, lenders, and fintechs use IDP to extract data from bank statements, tax forms (W-2s, 1040s, 1099s), pay stubs, and loan application packets. Common workflows include underwriting data collection, income verification, and tax document processing at filing volume.
Insurance
Carriers, MGAs, and brokers extract from ACORD forms, loss runs, and policy declarations. Carrier-specific document variants from AIG, Travelers, Hartford, and Zurich route through a single extraction pipeline rather than requiring per-carrier configuration.
Healthcare
Healthcare organizations use IDP to process CMS-1500 forms, explanations of benefits (EOBs), medical records, and prior authorization requests. Extracted data routes to Epic, Cerner, or clearinghouse systems with validated billing code formats.
Real Estate and Mortgage
Lenders and servicers process closing disclosures, appraisal reports, and income verification documents. IDP validates field calculations against CFPB disclosure requirements and accelerates underwriting timelines for workflows feeding Fannie Mae and Freddie Mac delivery systems.
Logistics
Freight brokers, 3PLs, and carriers extract from bills of lading, rate confirmations, and invoices. Structured data routes into TMS platforms like McLeod, TMW, and Oracle Transportation Management.
Frequently Asked Questions
What is the difference between IDP and OCR?
OCR (Optical Character Recognition) converts document images to raw machine-readable text without interpreting field structure. IDP applies classification, named-field extraction, and output validation on top of that text to return typed, structured data. OCR is a preprocessing step inside most IDP systems, not a replacement for them.
What is the difference between IDP and RPA?
RPA (Robotic Process Automation) automates repetitive software interactions: clicking, copying, and pasting between screens. IDP extracts structured data from unstructured documents. The two are frequently paired: IDP extracts the fields, RPA routes them into destination systems like Workday, SAP, or Salesforce. RPA alone cannot extract named fields from unstructured documents.
How accurate is modern IDP?
Most IDP systems reach 80–90% accuracy on heterogeneous document sets without significant configuration work. Production accuracy on the full document mix, including edge cases, poor scans, and issuer variants not in the training set, tends to be lower. Sensible closes that gap by applying layout-based extraction to fixed-position fields and LLM reasoning only where variability requires it, with schema enforcement and human review routing on all output.
What is the difference between IDP and document AI?
Document AI is a broad term for AI-powered tools that process or understand documents, including summarization, Q&A, and classification. IDP specifically refers to structured data extraction workflows that produce typed fields for downstream systems. Document AI tools may support IDP use cases, but not all document AI is production-grade IDP.
Does IDP require model training or prompt maintenance?
Production IDP systems built on LLM extraction do not require custom model training, but they do require ongoing prompt maintenance as underlying models update, unless the platform manages model lifecycle on behalf of the developer. Sensible manages model versioning so SenseML configs continue working when foundation models change. Retraining and prompt maintenance cycles are not part of the developer workflow.
How does IDP handle documents from multiple issuers or vendors?
A generalized extraction config handles document types with many issuer variants by applying LLM methods to accommodate label and position variation across the long tail. For high-volume issuers where accuracy is the priority, Sensible supports layout-specific configs that run deterministic extraction against fixed positions. Both run through the same API endpoint, and Sensible routes each document to the appropriate config automatically.
How much does it cost to implement IDP?
IDP implementation cost depends on document volume, the number of document types, and whether you use a prebuilt config or build custom extraction logic. Sensible is priced per document with no per-seat fees, which makes costs predictable as volume scales. For teams with complex document workflows or high-volume processing, managed services covers configuration, testing, and ongoing maintenance.
How does IDP handle sensitive document data?
Documents processed through IDP often contain sensitive information: loan applications, medical records, tax forms, insurance policies. Sensible is SOC 2 Type II certified, HIPAA compliant, and GDPR compliant. Documents are processed via API over HTTPS. Data retention policies are available on request, and custom region deployment supports data residency requirements. Extraction provenance provides a full audit trail of what was extracted and from where, without requiring access to the original document after the fact.
Start Extracting
Sensible's open-source configuration library includes prebuilt extraction configs for dozens of document types across financial services, healthcare, insurance, logistics, and real estate. Each config is a working starting point developers can deploy immediately or extend for their specific document variants.
Sign up for a free 2-week trial and run a first extraction against your own documents. For high-volume or complex document workflows, talk to our team about architecture, configuration, and managed services. For a detailed breakdown of the IDP evaluation criteria that matter in production, download the 2026 Buyer's Guide to Intelligent Document Processing.
.png)