> ## Documentation Index
> Fetch the complete documentation index at: https://claritylabs.inc/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# CL SDK Seven-Phase Insurance PDF Extraction Pipeline

> Understand the 7-phase extraction pipeline that converts insurance PDFs into structured, source-backed operational profiles and compatibility documents.

The CL SDK extraction pipeline transforms raw insurance PDF content into a structured, evidence-backed representation of policy facts. It runs as a coordinated sequence of seven phases — from normalizing raw text spans to projecting compatibility output — so every extracted fact traces back to a specific location in the source document.

## Pipeline Phases

**1. Normalize Source Spans**

Merge Docling-derived and caller-provided spans, then normalize their text. This phase reconciles overlapping or duplicate spans from different parsers into a single, deduplicated span set.

**2. Persist Source Evidence**

If a `sourceStore` is configured on the extractor, this phase saves the normalized spans and chunks before any extraction begins. Persisting evidence early means you retain raw source data even if a later phase fails.

**3. Build the Source Tree**

`buildDocumentSourceTree()` converts the flat `SourceSpan[]` into a hierarchy of `DocumentSourceNode[]`. Each node carries one of the following kinds:

| Kind | Description |
| - | - |
| `document` | Root of the tree |
| `page_group` | Logical grouping of pages |
| `page` | Individual PDF page |
| `form` | Named policy form |
| `endorsement` | Endorsement form |
| `section` | Labeled section within a form |
| `schedule` | Schedule table or list |
| `clause` | Individual clause or condition |
| `table` | Tabular structure |
| `table_row` | Row within a table |
| `table_cell` | Cell within a table row |
| `text` | Freeform text block |

**4. Group Source Structure**

Source nodes are grouped into declarations, policy forms, endorsements, sections, and schedules based on source text and parser-derived titles. Three hard constraints apply during grouping:

* **Don't invent IDs** — every node ID must derive from the source
* **Keep separately-numbered endorsements separate** — don't merge endorsements that carry distinct form numbers
* **Keep titles terse** — use the source heading text verbatim; don't rephrase

**5. Extract the Operational Profile**

A single `generateObject` call with bounded evidence extracts the full `PolicyOperationalProfile`. Every extracted fact must cite either `sourceNodeIds` or `sourceSpanIds`. This phase produces:

* `policyTypes` as `linesOfBusiness` ACORD codes
* Policy number, named insured, insurer, and broker
* Effective and expiration dates
* Total premium
* Coverage units with structured limits
* Endorsement inventory and support flags

**6. Clean Up Coverage Units**

An optional bounded cleanup pass reconciles coverage units — merging duplicate lines, filling missing fields, and resolving ambiguous references. You can control this pass via `modelCapabilitiesByTaskKind`.

**7. Project Compatibility Output**

The final phase materializes the `document`, `documentMetadata`, and `documentOutline` views from the operational profile and source tree. These are backward-compatible projections that conform to the `InsuranceDocument` schema.

## Quick Start

The snippet below shows the minimal setup to run the extraction pipeline against a base64-encoded PDF:

```typescript theme={"system"}
import { buildPageSourceSpans, createExtractor } from "@claritylabs/cl-sdk";

const sourceSpans = buildPageSourceSpans([
  { documentId: "doc-123", sourceKind: "policy_pdf", pageNumber: 1, text: pageOneText },
]);

const extractor = createExtractor({
  generateObject: myGenerateObject,
  onProgress: (msg) => console.log(msg),
});

const result = await extractor.extract(pdfBase64, "doc-123", { sourceSpans });

console.log(result.sourceTree?.length);            // canonical source hierarchy
console.log(result.operationalProfile?.coverages); // source-backed coverage units
console.log(result.document);                      // compatibility InsuranceDocument
console.log(result.chunks.length);                 // 0 on v3 source-tree paths
```

> [!NOTE]
>
> PDF inputs require `ExtractOptions.sourceSpans`. You must parse the PDF with LiteParse, Docling, PDF.js, OCR, or another parser before calling `extractor.extract`. The pipeline does not perform its own PDF rendering.

### Docling Input

If you parsed your document with Docling, pass the JSON document object directly instead of a base64 string:

```typescript theme={"system"}
const result = await extractor.extract(
  {
    kind: "docling_document",
    document: doclingDocumentJson,
    sourceKind: "policy_pdf",
  },
  "doc-123"
);
```

## Extractor Configuration

Create an extractor with `createExtractor`, passing an `ExtractorConfig` object. Only `generateObject` is required; all other fields are optional.

```typescript theme={"system"}
interface ExtractorConfig {
  generateObject: GenerateObject;   // Required — your AI provider wrapper
  onTokenUsage?: (usage: TokenUsage) => void;
  onProgress?: (message: string) => void;
  log?: LogFn;
  providerOptions?: Record<string, unknown>;
  sourceStore?: SourceStore;
  qualityGate?: "off" | "warn" | "strict";
  modelCapabilities?: ModelCapabilities;
  modelCapabilitiesByTaskKind?: Partial<Record<ModelTaskKind, ModelCapabilities>>;
  modelBudgetConstraints?: Partial<Record<ModelTaskKind, ModelBudgetConstraint>>;
}
```

**generateObject** `GenerateObject` *required*

Your AI provider wrapper. The pipeline calls this function for each structured generation step.

**onTokenUsage** `(usage: TokenUsage) =\> void`

Called after each model call with incremental token counts. Use this for real-time cost tracking.

**onProgress** `(message: string) =\> void`

Receives human-readable status messages as the pipeline advances through phases. Useful for streaming progress to a UI.

**sourceStore** `SourceStore`

If provided, the pipeline persists source spans and chunks in phase 2 before extraction begins.

**qualityGate** `"off" | "warn" | "strict"`

Controls how the pipeline responds to low-confidence extractions. `"strict"` throws on quality failures; `"warn"` logs them; `"off"` suppresses all quality checks.

**modelCapabilitiesByTaskKind** `Partial\<Record\<ModelTaskKind, ModelCapabilities\>\>`

Override model capabilities per pipeline task. Use this to allocate larger output budgets to specific phases.

### Model Routing Example

Route different token budgets to the operational profile extraction and the optional coverage cleanup pass:

```typescript theme={"system"}
const extractor = createExtractor({
  generateObject,
  modelCapabilitiesByTaskKind: {
    extraction_operational_profile: { maxOutputTokens: 16000 },
    extraction_coverage_cleanup:    { maxOutputTokens: 8000 },
  },
});
```

## Progress Messages

When you supply `onProgress`, the pipeline emits the following messages at key phase boundaries:

| Message | Phase |
| - | - |
| `"Building source-native document tree..."` | Phase 3 — source tree construction begins |
| `"Source-tree extraction complete."` | Phase 7 — compatibility projection finished |

> [!TIP]
>
> Forward `onProgress` messages to your application's job-status API so users see live feedback during long extractions on complex multi-form policies.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.