AI · Internal demonstration

LedgerlineContract intelligence with evaluated retrieval

A reference architecture for answering questions over thousands of contracts, with mandatory citation of the source clause and explicit refusal when evidence is weak.

Discipline: AI & Machine Learning
Technologies: Python · PostgreSQL · pgvector · LLM API · TypeScript
Demonstrates: Hybrid retrieval
Answers with verified citations
Continuous RAG evaluation
Privacy before the model

The challenge

01
Contracts carry clause numbers, party names and exact amounts. Purely semantic search fails precisely on those terms.
02
An answer without a source is useless to a legal team: every statement must point to the exact passage.
03
Changing models or chunking strategies can't be a blind decision. Retrieval quality must be measured before generation.

Constraints

Documents containing personal data
Auditable answers
Predictable cost per query

Architecture

Ingestion flow (top) and query flow (bottom), with the evaluation set acting as a quality gate.Fig. 01 · Ledgerline

Contracts: PDFs and DOCX, including scans.
Parse + OCR: Extracts text while preserving clause hierarchy.
PII redaction: Masks personal data before any external call.
Clause chunker: One chunk per clause, with the hierarchical path as metadata.
Hybrid index: PostgreSQL with pgvector for embeddings plus full-text search for exact terms.
Question: The user's natural-language question.
Retrieve + RRF: Vector and lexical search in parallel, merged with Reciprocal Rank Fusion.
Rerank: Reorders candidates and applies a minimum relevance threshold.
Cited answer: The model may only state what the cited passages support. No evidence, no answer.
Eval set: Reference questions with their correct clauses: measures recall@k and faithfulness.

Engineering decisions

D1Hybrid search instead of vectors only

Choice: Vector + full-text search, merged with Reciprocal Rank Fusion.
Rejected alternative: Embeddings only.
Why: “Clause 14.2” and “2% penalty” are exact terms. Embeddings approximate meaning but blur numbers and identifiers.

D2Chunk along the document's structure

Choice: One chunk per clause, with its path (Section 4 › 4.3) as metadata.
Rejected alternative: Fixed N-token windows with overlap.
Why: Fixed windows split clauses and mix different obligations in one chunk, which produces misleading citations.

D3Refusal is a valid result

Choice: Below the relevance threshold, the system says it found no basis in the documents.
Rejected alternative: Always generate an answer.
Why: For a legal team, a confident wrong answer costs more than no answer.

D4Evaluate before swapping any component

Choice: A reference question set runs in CI on every model, prompt or chunking change.
Rejected alternative: Occasional manual review.
Why: Without measurement, prompt improvements become silent retrieval regressions.

Implementation details

Ingestion runs as an asynchronous job: each document goes through extraction, personal-data masking and clause segmentation before indexing. Reprocessing is idempotent because chunk IDs derive from the document and the clause path.

At query time, vector and lexical searches run in parallel and are merged by rank (RRF), which avoids calibrating incompatible score scales. The reranker applies the threshold that decides between answering and refusing.

The model receives only approved passages, each with an identifier. The answer is validated: every citation must exist among the passages sent, otherwise the answer is discarded.

retrieval/fusion.pyRepresentative excerpt

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Merge ranked lists by position, so incompatible score scales never matter."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for position, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (k + position)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)


fused = reciprocal_rank_fusion([vector_hits, lexical_hits])
evidence = [c for c, _ in rerank(question, fused[:40]) if c.relevance >= MIN_RELEVANCE]
if not evidence:
    return Answer.refusal("No supporting clause found in the indexed contracts.")

Outcomes

Not applicable. As an internal demonstration there are no verified production metrics to report, and we don't publish estimates dressed up as results.

Facing a similar problem?

Tell us the context. We discuss architecture before any commercial proposal.

Next studyQuorum IndexerA reorg-safe EVM event indexer