Important

Provenance & Status. This document has two halves with different evidentiary standing. Stating that up front means nothing below has to be taken on trust.

The failure is observed. The three probes in §3 were executed by me against a live, commercially marketed legal-RAG product. The queries, the returned Spanish text, and the failure modes are real captured output, reproduced verbatim. §2 through §4 describe a system that actually behaved this way.

The remediation is specified, not deployed. The Statute-Binding Router in §5 is a reference architecture I authored in response to that evidence. It has not been built into the audited product and no production traffic has run through it. I was never engaged by the vendor — this is independent research, and the vendor is not named.

Every number carries a provenance label. Four classes are used throughout, and no figure of one class is presented as another:

Label	Meaning
Observed	Captured from the live audited system.
Structural	True by construction — provable from the architecture's semantics without running traffic.
Modeled	Arithmetic from stated inputs (published pricing, component latencies, declared assumptions).
Specified	A build target or acceptance threshold defined here. Not yet executed.

The central claim of this document is a Structural one, and §8 is the proof: a boolean metadata predicate cannot return a chunk from an unbound statute. That guarantee does not depend on a benchmark, and §10 states plainly what could still falsify it in practice.

---

1. Executive Summary: The Silent Statute Swap

In civil-law jurisdictions, identical article numbers exist across dozens of independent statutes. Artículo 42 appears in Spain's Código de Comercio (corporate group control), the Ley General Tributaria (solidary tax liability), and the Estatuto de los Trabajadores (subcontracting obligations) — three unrelated bodies of law sharing one numerical identifier.

This study audits a commercially marketed Conversational RAG platform advertising a sub-1% hallucination guarantee. Under independent black-box probing, the system returned the wrong statute on 3 of 3 overlapping article-number queries — each time quoting fabricated or misattributed legal text with full confidence. (Observed.)

Why This Matters Beyond Engineering. In jurisdictions governed by the EU AI Act (Article 14: Human Oversight, Article 15: Accuracy), legal-tech systems deployed as decision-support tools must demonstrate retrieval accuracy as a compliance property, not merely a performance metric. Under the SRA's Principles (Principle 2: act in a way that upholds public trust; Principle 7: act in the best interests of each client), a solicitor relying on AI-generated legal citations bears professional liability for their accuracy — regardless of whether the error originated in the model or the retrieval tier. The Oregon federal court's precedent of imposing per-citation sanctions ($1,000–$5,000 per fabricated case) establishes a concrete, quantifiable cost for retrieval failures that reach end users.

This study demonstrates that a system passing aggregate accuracy benchmarks can simultaneously fail on every probe of a structurally predictable class of queries — and that the fix is not a better model, but a deterministic compliance control inserted upstream of the AI inference layer.

The root cause is not a generation failure. The LLM is never given the correct evidence. The failure occurs upstream, in the retrieval tier, where dense vector search and ColBERT late-interaction reranking both collapse on article-number collisions. The downstream agent then burns 30 seconds to 5 minutes thrashing in an unconstrained ReAct loop before producing a confident hallucination.

The specified remediation is a deterministic retrieval integrity control — a pre-retrieval compliance gate running local CPU inference (~7ms, ~$0 marginal cost) inserted between the query and the vector database. It eliminates retrieval contamination by construction (§8), collapses agent thrashing latency from minutes to sub-second under the model in §6, and restores the factual integrity guarantee that production legal deployment requires.

---

2. The Anatomy of the Failure: Why Vector Search Collapses on Shared Identifiers

2.1 The Structural Problem: Article-Number Collision Space

Civil-law corpora are structurally distinct from common-law systems. Statutes are organized by sequential article numbers, and these numbers are reused across independent codes. In the Spanish legal domain alone:

Article Number	Statute	Domain	Subject Matter
Art. 42	Código de Comercio (CdC)	Commercial Law	Corporate group control & consolidation
Art. 42	Ley General Tributaria (LGT)	Tax Law	Solidary tax liability
Art. 42	Estatuto de los Trabajadores (ET)	Labour Law	Subcontracting obligations
Art. 42	Ley 15/1988	Arbitration	(additional collision)

When a user queries "¿Qué establece el Artículo 42?" (English: "What does Article 42 establish?") without specifying the target statute, the retrieval system must resolve this ambiguity before querying the vector database. Standard RAG pipelines do not perform this resolution.

2.2 The Physics of the Collapse: Three Simultaneous Failures

The pipeline under audit follows a standard Retrieve-then-Generate architecture:

flowchart LR
    A["User Query: 'Art. 42 CdC'"] --> B["Embed Query"]
    B --> C["HNSW Dense Search"]
    C --> D["ColBERT Rerank"]
    D --> E["LLM Agent"]
    E --> F["Response"]

The collapse occurs at three independent points in this chain. The mechanism described below is an analytical account of why the observed failures in §3 occurred; the failures themselves are Observed, the causal decomposition is the argument.

Failure Point 1: Dense Vector Centroid Contamination

Dense embedding models (e.g., OpenAI text-embedding-3) encode the semantic meaning of the full query into a single vector. For a query containing "Artículo 42", the resulting vector sits at the geometric centroid of all statutory contexts mentioning "Artículo 42":

$$\mathbf{q} = \text{Embed}(\text{``Artículo 42 del Código de Comercio''})$$

$$\mathbf{d}_{\text{CdC}} = \text{Embed}(\text{``Art. 42 CdC: control de grupo societario...''})$$

$$\mathbf{d}_{\text{LGT}} = \text{Embed}(\text{``Art. 42 LGT: responsabilidad tributaria solidaria...''})$$

$$\mathbf{d}_{\text{ET}} = \text{Embed}(\text{``Art. 42 ET: subcontratación de obras...''})$$

The cosine similarities between $\mathbf{q}$ and each document vector are dominated by the shared lexical anchor "Artículo 42", which contributes disproportionately to the embedding norm. The HNSW approximate nearest-neighbour index returns a contaminated candidate set containing chunks from all three statutes.

Failure Point 2: ColBERT Late-Interaction Score Collapse

ColBERT reranking operates via token-level MaxSim scoring. For each query token $q_i$, the system computes the maximum cosine similarity against all document tokens $d_j$:

$$\text{Score}(q, d) = \sum_{i=1}^{|q|} \max_{j \in \{1, \dots, |d|\}} \cos(\mathbf{q}_i, \mathbf{d}_j)$$

The query tokens ["Artículo", "42"] achieve near-identical MaxSim scores against chunks from the CdC, LGT, and ET — because these exact tokens appear verbatim in all three statutes. The discriminating context (e.g., "Código de Comercio") is diluted across the remaining query tokens, which carry insufficient weight to break the tie.

$$\text{Score}(q, d_{\text{CdC}}) \approx \text{Score}(q, d_{\text{LGT}}) \approx \text{Score}(q, d_{\text{ET}})$$

The reranker cannot disambiguate. It returns a mixed set of clauses from unrelated laws with statistically indistinguishable relevance scores.

Failure Point 3: ReAct Agent Thrashing

The downstream LLM agent receives conflicting evidence chunks from multiple statutes. Without the ability to verify which statute the user intended, the agent enters an unconstrained ReAct loop:

graph TD
    A([Agent Receives Conflicting Chunks]) --> B{Can Agent Resolve<br/>Statute Identity?}
    B -->|No| C[Fetch More Context<br/>from Vector DB]
    C --> D{Still Ambiguous?}
    D -->|Yes| E[Execute Web Scraping<br/>Fallback]
    E --> F{Resolved?}
    F -->|No| G[Thrash: 29-32<br/>Reasoning Steps]
    G --> H[Budget Exhaustion<br/>or Timeout]
    H --> I[Generate Confident<br/>Hallucination]
    B -->|Yes| J([Generate Correct Response])
    D -->|No| J
    F -->|Yes| J

    style I fill:#000000,stroke:#dc2626,stroke-width:2px
    style J fill:#000000,stroke:#16a34a,stroke-width:2px
    style G fill:#000000,stroke:#f57c00,stroke-width:2px

This thrashing inflates query latency to 30 seconds – 5 minutes per query (Observed — wall-clock, measured client-side across the test battery). The agent eventually exhausts its step budget and produces a response grounded in the wrong statute — with full confidence and no disclaimer.

Note

Provenance of the step count. The 29–32 reasoning-step range is Inferred, not directly Observed. The product exposes a progress indicator that enumerates agent steps during execution; the range is read from that surface, not from vendor telemetry or server logs. The wall-clock latency and the final output text are directly Observed. Cost figures in §4 that depend on the step count are labelled Modeled accordingly.

This is not a generation hallucination. The LLM is behaving exactly as designed — it is faithfully summarizing the evidence it was given. The evidence itself is contaminated at the retrieval tier. No amount of prompt engineering, model scaling, or generation-tier auditing (NLI gates, self-consistency checks) can fix retrieval contamination. The fix must occur before the vector database is queried.

---

3. The Forensic Trace: Three Tests, Three Catastrophic Failures

Important

This section is Observed. The following probes were executed by me against the live commercial product using the Spanish Código de Comercio (CdC) Artículo 42 as the ground-truth target. Queries and returned text are reproduced verbatim from the session. This was black-box testing against a publicly available interface — no privileged access, no vendor cooperation, and no vendor engagement. The vendor is not named, and nothing here is derived from confidential information.

Test 1 (T3): Isolate Verbatim Sub-Clause — letra b) of Art. 42.1 CdC

Query: "Reproduce the exact text of Art. 42.1, letra b) of the Código de Comercio."

Expected (Ground Truth):

"Tenga la facultad de nombrar o destituir a la mayoría de los miembros del órgano de administración." (English: "Has the power to appoint or remove the majority of the members of the administrative body.") — Art. 42.1 b) CdC (Corporate group control: power to appoint/remove board members)

Returned:

"Las que sean causantes o colaboren en la ocultación o transmisión de bienes... de la Administración tributaria." (English: "Those who cause or collaborate in the concealment or transfer of assets... of the Tax Administration.") — Art. 42 of the Ley General Tributaria (Solidary tax liability for concealment of assets)

Diagnosis: The retriever fetched the wrong statute entirely. The LLM quoted tax-liability text verbatim as if it were commercial code. Full confidence. Zero disclaimer.

---

Test 2 (T2): Resolve apartado c) — Probe letra/apartado Distinction

Query: "¿Qué establece el apartado c) del Artículo 42 del Código de Comercio?" (English: "What does paragraph c) of Article 42 of the Commercial Code establish?")

Expected: The system should resolve apartado c) within Art. 42 CdC and return the correct commercial-law sub-clause.

Returned: The system:

1. Never accessed the Código de Comercio.
2. Fabricated a contextual premise — "según el contexto de los hechos que estamos tratando" ("according to the context of the facts we are discussing") — referencing a nonexistent prior conversation.
3. Answered from two wrong statutes simultaneously: Art. 42.4 c) of the Estatuto de los Trabajadores (subcontracting) and Ley 15/1988 (arbitration).

Diagnosis: Two wrong laws, an invented conversational premise, and zero engagement with the letra vs. apartado structural distinction the test was probing. The retrieval layer delivered labour law and arbitration text for a commercial-law query.

---

Test 3 (T1): Fabricated Sub-Clause Trap — apartado 8

Query: "¿Qué establece el apartado 8 del Artículo 42 del Código de Comercio?" (English: "What does paragraph 8 of Article 42 of the Commercial Code establish?")

Expected: Art. 42 CdC contains no apartado 8. The correct response is a clear negation: "El Artículo 42 del Código de Comercio no contiene un apartado 8." (English: "Article 42 of the Commercial Code does not contain a paragraph 8.")

Returned: The system:

1. Fabricated verbatim text for the nonexistent apartado 8:

   "Se presumirá que existe control cuando una sociedad, por cualquier otro medio, pueda ejercer una influencia dominante sobre otra." (English: "It shall be presumed that control exists when a company, by any other means, can exercise a dominant influence over another.") — This provision does not exist anywhere in Art. 42 CdC.

2. Fabricated grounding context: Prefaced with "De acuerdo con la información proporcionada en el análisis previo" ("According to the information provided in the previous analysis") — there was no previous analysis.
3. Cited repealed legislation: Imported the "unidad de decisión" (English: "unity of decision") concept from Ley 62/2003, which was repealed by Ley 16/2007. The system treated a defunct provision as current law.

Diagnosis: A single response that simultaneously (a) fabricated a nonexistent sub-clause, (b) invented a fake grounding reference, and (c) cited repealed legislation as current. Three independent failure modes detonating in one answer.

This is not random hallucination. The system cannot bind the correct statute when the article number collides across codes. The French-law analogue of this test battery (using L233-3 / L233-10, which are near-unique identifiers) passed cleanly — a control condition confirming that the failure is structurally tied to article-number collision, not to general model weakness.

3.1 Sample Size — Stated Honestly

Three probes is a small battery, and the "100%" figure should be read as 3 of 3, not as a stable population rate. What the battery establishes is existence and reproducibility, not frequency:

• The failure is reproducible on demand — every collision probe failed, and the French control condition passed.
• The failure class is structurally predictable — §2 and §8 explain why any collision query must land in the same trap, which is what makes three failures sufficient to characterise the class.
• The failure is not established as a rate over the vendor's real traffic mix. The $\alpha = 5\%$ collision rate used in §4.2 is a declared assumption, not a measurement.

A larger battery would tighten the frequency estimate. It would not change the structural argument, which is the load-bearing claim.

3.2 Corpus and Ground Truth — What Was Checked Against What

Every returned answer was verified against the consolidated text of the Código de Comercio held in my own reference corpus. That is what makes each failure a fact rather than an impression: the correct text for Art. 42.1 b) is known, and what the system returned was Art. 42 of the Ley General Tributaria instead.

Two limits on that follow, and both matter:

• The corpus is mine, not a public versioned source. I have not run this against the legislation.gov.uk point-in-time API, which serves statute as data with the version in force on a given date. That is the reference a third party could re-run against without trusting my copy, and it is the measurement I intend to make next. Until then, reproduction requires the reader to obtain the consolidated Spanish texts themselves.
• Temporal validity was not systematically tested. The repealed-law failure in T1 (Ley 62/2003, repealed by Ley 16/2007) surfaced incidentally inside a collision probe. Whether the system returns the provision as in force on a given date is a separate question that this battery does not answer, and the control specified in §5 does not address it either — see §10.3.

Neither limit weakens the collision finding. The wrong statute was returned, verified against its correct text, three times out of three.

---

4. The Unit Economics of Agent Thrashing

Note

This section is Modeled. The latency inputs are Observed; the per-step cost, the step count, and the collision rate $\alpha$ are declared assumptions. The output is a cost model, not a measured invoice. Every input is stated so the arithmetic can be re-run against different assumptions.

4.1 Per-Query Cost of the ReAct Loop

When the retriever delivers contaminated chunks, the ReAct agent compensates by requesting more context, executing web-scraping fallbacks, and iterating through 29–32 reasoning steps. Each step involves an LLM inference call.

Let $n$ denote the number of ReAct steps, $c_{\text{step}}$ the per-step LLM inference cost, and $t_{\text{step}}$ the latency per step:

$$C_{\text{thrash}} = n \cdot c_{\text{step}}$$

$$T_{\text{thrash}} = n \cdot t_{\text{step}}$$

Input	Value	Provenance
$n$ — ReAct steps per collision query	30	Inferred — read from the product's step indicator (§2.2 note)
$c_{\text{step}}$ — cost per agent step	$0.003	Assumed — mid-tier frontier model, ~1k output tokens per reasoning step at published rates
$t_{\text{step}}$ — latency per step	6s	Modeled — derived by dividing Observed wall-clock latency by inferred step count

For the observed failure mode:

$$C_{\text{thrash}} = 30 \times \$0.003 = \$0.09 \text{ per failed query}$$

$$T_{\text{thrash}} = 30 \times 6\text{s} = 180\text{s} \text{ (3 minutes)}$$

The 180s figure sits inside the Observed 30s–5min envelope, which is the only cross-check available on this input.

4.2 Aggregate OPEX Bleed at Scale

In a production legal RAG serving Spanish statutory law, queries involving common article numbers (Art. 1, 42, 50, 100, etc.) represent a non-trivial fraction of total traffic. Let $\alpha$ denote the collision rate — the fraction of queries that hit an ambiguous article number:

$$\text{Daily Thrashing Cost} = Q_{\text{daily}} \cdot \alpha \cdot C_{\text{thrash}}$$

At declared assumptions ($Q_{\text{daily}} = 50{,}000$ queries, $\alpha = 5\%$):

$$\text{Daily Cost} = 50{,}000 \times 0.05 \times \$0.09 = \$225/\text{day} = \$6{,}750/\text{month}$$

Warning

$Q_{\text{daily}}$ and $\alpha$ are illustrative, not measured. I have no visibility into the vendor's traffic volume or query mix. These figures size the shape of the exposure — a per-collision cost multiplied by a collision frequency — and the sensitivity is linear in both terms. At $\alpha = 1\%$ the monthly figure is $1,350; at $\alpha = 10\%$ it is $13,500. The defensible claim is the per-query cost of thrashing, not the monthly total.

This is pure waste — compute burned on reasoning steps that produce wrong answers. It does not include the downstream cost of user-facing legal errors, churn, or liability exposure.

4.3 Latency Distribution Under Collision

The latency distribution bifurcates into two modes:

$$T_{\text{query}} = \begin{cases} T_{\text{normal}} \approx 2\text{–}5\text{s} & \text{if article number is unique (no collision)} \\ T_{\text{thrash}} \approx 30\text{–}300\text{s} & \text{if article number collides across statutes} \end{cases}$$

Both branches are Observed — the fast branch from the French control condition, the slow branch from the collision battery.

This bimodal distribution is architecturally invisible in aggregate latency dashboards (mean latency appears acceptable), but the tail creates a catastrophic user experience for a predictable, reproducible class of queries.

---

5. The Inevitable Pivot: Deterministic Statute-Binding Router

Important

This section is Specified. Everything from here forward is a reference architecture authored in response to the §3 evidence. It has not been built into the audited product. Component latencies quoted below are published/benchmarked figures for the named components, not measurements of an assembled system.

5.1 The Architectural Principle

The fix does not involve replacing the vector database, retraining the embedding model, or modifying the generation tier. It inserts a single deterministic routing node between the user query and the retrieval pipeline.

This node performs three operations in sequence, all on local CPU inference (~7ms total, Modeled from published component benchmarks):

1. Entity Extraction: Parse the query for article numbers, statute keywords, and structural identifiers.
2. Collision Detection: Check the extracted article number against a lightweight collision index.
3. Resolution or Interception: Either bind a hard metadata filter to the database query, or intercept the query with a structured clarification prompt.

5.2 The Architecture (Before vs. After)

Before — Unfiltered Retrieval (the audited system):

graph LR
    Q([User Query:<br/>'Art. 42 CdC']) --> Embed[Dense Embedding]
    Embed --> HNSW[HNSW Search<br/>Milvus]
    HNSW --> Mixed[Mixed Chunks:<br/>CdC + LGT + ET]
    Mixed --> ColBERT[ColBERT Rerank<br/>Scores ≈ Equal]
    ColBERT --> Agent[ReAct Agent<br/>30-32 Steps]
    Agent --> Wrong[Wrong Statute<br/>Confident Hallucination]

    style Mixed fill:#000000,stroke:#dc2626,stroke-width:2px
    style Wrong fill:#000000,stroke:#dc2626,stroke-width:2px
    style Agent fill:#000000,stroke:#f57c00,stroke-width:2px

After — Statute-Binding Router Inserted (specified target state):

graph TD
    Q([User Query]) --> Router

    subgraph SBR ["Statute-Binding Router (~7ms, Local CPU)"]
        Router[spaCy + DiBERT<br/>Entity Extraction] --> Parse[Extract:<br/>article_num, statute_keyword]
        Parse --> Check{Collision<br/>Detected?}
    end

    Check -->|No Collision| Bind[Bind Metadata Filter<br/>to Vector Query]
    Check -->|Collision + Keyword Found| Bind
    Check -->|Collision + No Keyword| Intercept[Intercept → Ask User:<br/>'By Art. 42, did you mean<br/>CdC, LGT, or ET?']

    Intercept -->|User Selects| Bind
    Bind --> DB[(Vector DB<br/>Hard-Filtered<br/>statute_id = 'CdC')]
    DB --> ColBERT2[ColBERT Rerank<br/>Clean Candidates]
    ColBERT2 --> LLM[LLM Generation<br/>Correct Evidence]
    LLM --> Correct([Correct Response])

    style SBR fill:#000000,stroke:#1a73e8,stroke-width:2px
    style Correct fill:#000000,stroke:#16a34a,stroke-width:2px
    style Intercept fill:#000000,stroke:#f57c00,stroke-width:2px

5.3 The Metadata Filter — Formal Specification

Once the statute is bound (either from the query text or from the user's clarification selection), the vector database query is constrained by a hard metadata predicate:

$$\text{Filter: } \{\texttt{statute\_id} = \text{``CdC''}, \;\texttt{article\_num} = \text{``42''}\}$$

This predicate is applied before HNSW candidate generation, not as a post-retrieval filter. It eliminates cross-statute contamination at the index level, ensuring that ColBERT only scores chunks from the correct law.

5.4 The Collision Index — Data Structure

The collision index is a lightweight, pre-computed lookup table mapping article numbers to the statutes that contain them:

$$\text{CollisionIndex}: \mathbb{N} \to \mathcal{P}(\text{StatuteID})$$

$$\text{CollisionIndex}(42) = \{\text{CdC}, \text{LGT}, \text{ET}, \text{Ley 15/1988}\}$$

If $|\text{CollisionIndex}(n)| > 1$, the article number is ambiguous and the router must resolve it before proceeding. The index is populated from the write-path keyword extraction pipeline, ensuring vocabulary consistency between read and write paths.

---

6. The Projected Latency and Cost Delta

Note

This section is Modeled. The "Before" column is grounded in Observed latency; the "After" column is a projection built from published component benchmarks and the assumption that the router removes the thrashing loop entirely. These are not measurements of a running system. §10 states the conditions under which the projection would degrade.

6.1 Projected Latency Compression

The router eliminates the ReAct thrashing loop entirely — this is the mechanism, and it follows from §8: with a hard-bound statute, the agent never receives conflicting evidence and therefore has nothing to thrash on.

$$T_{\text{before}} = T_{\text{embed}} + T_{\text{HNSW}} + T_{\text{ColBERT}} + T_{\text{thrash}}$$

$$T_{\text{before}} \approx 0.2\text{s} + 0.1\text{s} + 0.3\text{s} + 180\text{s} = 180.6\text{s}$$

$$T_{\text{after}} = T_{\text{router}} + T_{\text{embed}} + T_{\text{HNSW\_filtered}} + T_{\text{ColBERT}} + T_{\text{LLM}}$$

$$T_{\text{after}} \approx 0.007\text{s} + 0.2\text{s} + 0.05\text{s} + 0.15\text{s} + 2\text{s} = 2.4\text{s}$$

$$\boxed{\frac{T_{\text{before}}}{T_{\text{after}}} = \frac{180.6}{2.4} \approx 75\times \text{ (modeled latency reduction)}}$$

The filtered HNSW search is modeled as faster than unfiltered because the metadata predicate prunes the candidate space before the ANN traversal begins. ColBERT reranking is also modeled as faster because it scores fewer (and exclusively relevant) candidates.

Tip

What the 75× actually rests on. Almost the entire ratio comes from one term — deleting $T_{\text{thrash}}$. The other components move by tenths of a second. So the claim to interrogate is not "is 7ms realistic?" but "does hard statute binding actually eliminate thrashing?" That question is answered structurally in §8, not empirically here. If the router binds the wrong statute (§10), the user gets a fast wrong answer instead of a slow one — which is why the audit trail in the appendix is part of the control, not decoration.

6.2 Projected Cost Elimination

Metric	Before (Unfiltered)	After (Statute-Bound)	Provenance
ReAct steps per collision query	29–32	0	Inferred → Structural (no conflicting evidence to resolve)
LLM calls per collision query	30+ (agent reasoning)	1 (generation only)	Inferred → Specified
Per-query cost (collision)	~$0.09	~$0.003	Modeled (both)
Monthly OPEX at 2,500 collision queries/day	$6,750	$225	Modeled — assumes $\alpha$, $Q$ from §4.2
Monthly savings	—	$6,525	Modeled
Routing overhead	0ms (no router exists)	7ms	Modeled from component benchmarks
Routing cost	$0	~$0 (local CPU)	Structural — no per-token billing on local inference

6.3 Accuracy: What Is Measured vs. What Is Guaranteed

This is the table most likely to be misread, so the two columns are kept strictly apart. The left column is what the live system did. The right column is what the specified control guarantees — and why it guarantees it.

Metric	Observed (Audited System)	Under Specified Control	Basis for the Right Column
Accuracy on collision battery	0/3 (0%)	Not yet executed	Specified — re-running this battery is the acceptance test, not a result
Wrong-statute retrieval rate	3 of 3 on the collision batch	0%	Structural — the index predicate cannot return a chunk whose statute_id ≠ the bound value (proof: §8)
Fabricated-text rate	2/3 queries	Materially reduced, not zero	Modeled — correct evidence removes the cause observed here; it does not make the generator incapable of fabrication
Repealed-law citation rate	1/3 queries (Ley 62/2003, repealed by Ley 16/2007)	Requires a separate control	Honest gap — temporal validity is a corpus versioning problem, not a disambiguation problem. See §10.3.

Caution

Do not read row 2 as a benchmark result. "0%" is a structural guarantee about which chunks can enter the candidate set, provable from the boolean semantics of the filter. It is not a claim that the assembled system was tested and scored zero. The two are different kinds of statement, and conflating them is exactly the error this document is written to avoid.

---

7. Deployment Specification: Zero-Downtime Shadow Parity

Important

This is a rollout protocol, not a rollout report. No phase below has been executed. The thresholds are acceptance criteria I would hold the build to — the numbers a team should require before cutting over. They are stated as gates, never as observed results.

The Statute-Binding Router would be deployed using a Zero-Downtime Shadow Migration pattern:

Phase 1: Shadow Routing (Dual-Path)

The production pipeline continues to operate unmodified. 100% of production traffic is asynchronously mirrored to a staging instance running the Statute-Binding Router. No production user sees staging output. The staging instance logs:

• Whether the router detected a collision
• Whether the user's query contained a resolvable statute keyword
• Whether the staging response matched the production response
• The latency delta between production and staging

Phase 2: Telemetry Validation — Acceptance Gates

A minimum 72-hour soak, with cutover gated on the following criteria. Each is a threshold to be met, and each has a defined failure action:

#	Gate	Threshold to Proceed	If the Gate Fails
1	Collision false-positive rate — router flags a collision that does not exist	< 0.1%	Collision index contains a spurious entry; audit index generation against the corpus
2	Collision false-negative rate — router misses a real collision	0% on the exhaustive article-number mapping	Index is not exhaustive; regenerate from the full write-path extraction
3	Statute-binding precision — bound statute matches user intent where intent is unambiguous	≥ 99%	Entity extraction is the weak link, not the filter; retrain/expand the abbreviation gazetteer (§10.1)
4	Interception rate — share of queries routed to human disambiguation	Within a product-agreed ceiling	Too high = user friction; too low = silent wrong binding. Both are product decisions, not engineering ones
5	Routing latency, p99	< 25ms	Router is not free; profile spaCy pipeline and index lookup
6	Accuracy delta — staging correct where production was wrong	Positive and monotonic over the soak	If not positive, the thesis in §8 is wrong for this corpus. Stop.

Note

Gate 2 is set at 0% deliberately, and it is the one gate that is genuinely reachable: the collision index is a deterministic, pre-computed mapping over a finite corpus. A missed collision means the index is incomplete — a data-generation bug with a mechanical fix — not a model that needs tuning. Gate 3, by contrast, is a statistical target, because entity extraction from free-text queries is a statistical problem. The distinction between these two gates is the distinction between the parts of this architecture that are guaranteed and the parts that are merely good.

Phase 3: Feature-Flag Cutover

A feature flag flips routing to the staging pipeline. The production (unfiltered) path is decommissioned only after a defined bake period. Rollback is instantaneous — flip the flag back.

The metadata fields (statute_id, article_num) needed for the hard filter are assumed to already exist in the vector database schema — a standard field set for a statutory corpus, though unverified for the audited product specifically. If they do not exist, a re-index is required and the "no database rebuild" claim in §9 does not hold. The only genuinely new component is the routing layer itself — a lightweight Python service running spaCy and the collision index in-process.

---

8. Why Scaling the Embedding Model Cannot Fix This

This is the load-bearing section of the document. Everything claimed as Structural rests on the argument below, which is an argument from the semantics of the representation — it requires no traffic, no benchmark, and no deployment to evaluate. A reader who rejects this section should reject the specification; a reader who accepts it does not need a benchmark to accept the 0% claim.

A natural objection is: "Use a better embedding model." This section demonstrates why the collision is structurally irreducible by any dense embedding approach.

The Embedding Bottleneck Theorem (Informal)

Dense embedding models map text to a single vector $\mathbf{v} \in \mathbb{R}^d$. Cosine similarity between query $\mathbf{q}$ and document $\mathbf{d}$ is:

$$\text{sim}(\mathbf{q}, \mathbf{d}) = \frac{\mathbf{q} \cdot \mathbf{d}}{\|\mathbf{q}\| \cdot \|\mathbf{d}\|}$$

For two documents $d_{\text{CdC}}$ and $d_{\text{LGT}}$ that share the dominant lexical anchor "Artículo 42", the embedding vectors share a large component along the direction encoding that anchor:

$$\mathbf{d}_{\text{CdC}} = \alpha \cdot \mathbf{e}_{\text{Art42}} + \beta \cdot \mathbf{e}_{\text{commercial}}$$

$$\mathbf{d}_{\text{LGT}} = \alpha \cdot \mathbf{e}_{\text{Art42}} + \gamma \cdot \mathbf{e}_{\text{tax}}$$

Where $\alpha \gg \beta, \gamma$ because the article number is the primary semantic anchor of both documents. The cosine similarity between the query and each document is dominated by the $\alpha$ term:

$$\text{sim}(\mathbf{q}, \mathbf{d}_{\text{CdC}}) \approx \text{sim}(\mathbf{q}, \mathbf{d}_{\text{LGT}}) \approx \frac{\alpha^2}{\|\mathbf{q}\| \cdot \|\mathbf{d}\|}$$

The discriminating components ($\beta \cdot \mathbf{e}_{\text{commercial}}$ vs. $\gamma \cdot \mathbf{e}_{\text{tax}}$) are second-order corrections that vanish relative to $\alpha$ as the shared anchor grows more dominant.

Increasing the embedding dimension $d$ does not help. The additional dimensions encode finer semantic distinctions, but the dominant shared anchor still contributes the majority of the dot product. The fundamental issue is that dense embeddings compress the entire document into a single point — they cannot represent the conditional relationship "this text is about Art. 42 of the CdC specifically" as a hard constraint.

Note

Scope of the informal theorem. This is an argument about the dominance of a shared anchor term, not a formal proof with stated bounds on $\alpha/\beta$. It predicts score collapse, and score collapse is what §3 Observed. It does not prove that no embedding scheme could ever separate these documents — a model trained specifically to over-weight statute tokens would push $\beta$ up. The claim is that doing so is a tuning race against a boolean constraint that already wins outright, not that the race is unwinnable in principle.

Why Hard Metadata Filtering Is the Correct Solution

The metadata filter operates outside the embedding space. It applies a predicate at the index level:

$$\text{Candidates} = \{d \in \mathcal{D} \mid \texttt{statute\_id}(d) = \text{``CdC''} \wedge \texttt{article\_num}(d) = \text{``42''}\}$$

This predicate is a boolean constraint, not a similarity score. It cannot be approximated, degraded, or confused by lexical overlap. It partitions the search space deterministically before any vector similarity computation occurs.

$$\boxed{\text{Metadata filtering} \in O(1) \text{ precision.} \quad \text{Dense similarity} \in O\!\left(\frac{1}{\text{collision count}}\right) \text{ precision.}}$$

This is the proof behind the 0% claim. If $\texttt{statute\_id}(d) \neq \text{``CdC''}$, then $d \notin \text{Candidates}$ — not "ranks lower," not "is usually filtered," but is not in the set. The wrong-statute retrieval rate is therefore zero for any correctly bound query, by construction and independent of corpus size, model quality, or traffic volume.

The residual risk is not in the filter. It is entirely in the binding step — whether the router extracts the right statute_id in the first place. That risk is real, it is statistical rather than structural, and §10 addresses it directly.

No embedding model — regardless of architecture, training data, or dimensionality — replicates the precision of a hard boolean filter on structured metadata. The collision is not a model quality problem. It is a representation problem inherent to dense vector spaces when the dominant semantic signal is shared across distinct entities.

---

9. Conclusion

A commercially marketed legal-tech platform advertised a sub-1% hallucination rate. Under independent black-box probing against Spanish statutory law, the system returned fabricated legal citations on 3 of 3 overlapping article-number queries — quoting tax liability as commercial code, fabricating nonexistent sub-clauses, and citing repealed legislation as current law. Each failure would constitute a sanctionable event under professional conduct rules (SRA Principle 2 / ABA Model Rule 1.1) if relied upon in practice. This half is Observed.

The root cause was not the LLM. The root cause was an unaudited retrieval tier with no compliance controls — dense vector search and ColBERT reranking both collapse when identical article numbers span independent statutes, delivering a poisoned evidence set that no downstream model can correct. The agent compensated by thrashing through reasoning steps, burning minutes of compute per query, and ultimately generating a confident hallucination grounded in the wrong law.

The specified fix is a single-node insertion: a deterministic retrieval integrity control — a pre-retrieval compliance gate running local CPU inference at ~7ms and ~$0 marginal cost. It binds statute identity via hard metadata filtering before any vector similarity computation occurs — a boolean constraint that dense embeddings are structurally incapable of replicating (§8). This half is Specified, and has not been deployed.

Results

Metric	Observed (Audited System)	Target State (Specified Control)	Provenance of Target
Fabricated citation rate (cross-statute queries)	3 of 3 — system cited wrong law with full confidence	0% wrong-statute retrieval	Structural — proven in §8
Retrieval auditability	None — no trace of which statute was selected or why	Full — every query emits a logged decision trace	Structural — the log is emitted by the router by design
Adversarial test pass rate	0/3 structured tests passed	Not yet executed	Specified — this battery is the acceptance test
Compliance with sub-1% hallucination guarantee	Guarantee violated on all collision queries	Guarantee becomes defensible for this failure class	Structural, scoped — addresses disambiguation only
Enterprise procurement readiness (TPRM)	Would fail retrieval accuracy section	Satisfies the retrieval accuracy section by design	Specified — control + audit trail + measurable objective
Latency (secondary)	30s – 5 min (agent thrashing)	~2.4s (75× reduction)	Modeled — §6.1
Monthly OPEX (secondary)	~$6,750 wasted compute	~$225	Modeled — assumes $Q$, $\alpha$ from §4.2
Database rebuild required	—	No	Assumed — conditional on existing schema fields (§7)
Deployment risk	—	Zero-downtime shadow parity	Specified — protocol in §7, not executed

Lessons Learned

1. Retrieval contamination is invisible to generation-tier compliance audits. Most AI compliance frameworks (including the NIST AI RMF and ISO 42001) focus audit controls on the generation tier — hallucination detection, output filtering, human-in-the-loop gates. This case demonstrates that a system can pass all generation-tier audits while failing on every probe of a structurally predictable class of retrieval queries. Compliance audits that do not include structured probing of the retrieval tier are incomplete by design. NLI auditors, self-consistency checks, chain-of-thought prompting, and model scaling cannot correct evidence that was wrong before the LLM saw it. If the retriever delivers tax law when the user asks for commercial law, the best LLM in the world will faithfully summarize tax law. The fix must occur upstream of the embedding layer.

2. Dense embeddings have a structural ceiling on disambiguation. When the dominant semantic signal (e.g., "Artículo 42") is shared across distinct entities, cosine similarity scores collapse to near-identical values regardless of embedding model quality or dimensionality. This is not a model quality problem — it is a representation problem inherent to compressing documents into single-point vectors. Hard metadata filtering operates outside the embedding space and provides $O(1)$ precision where dense similarity degrades as $O(1/\text{collision\_count})$.

3. Agent autonomy without retrieval constraints is a liability multiplier. Unconstrained ReAct loops are designed to "figure it out" by fetching more context. When the retrieval layer itself is the source of contamination, more context means more contamination. The agent thrashes harder, burns more compute, and produces more confidently wrong output. Deterministic routing pre-empts the thrashing loop entirely.

4. The cheapest fix is often the most impactful. The Statute-Binding Router is a lightweight Python service running spaCy and a static lookup table. It costs nothing to operate, adds ~7ms of latency, requires no database rebuild under the stated schema assumption, and eliminates an entire class of catastrophic legal liability. The instinct to solve retrieval problems with better embeddings, larger models, or more sophisticated rerankers is a trap — the correct tool for a disambiguation problem is a disambiguator, not a better approximator.

5. Domain-specific corpora expose universal pipeline assumptions. The standard RAG pipeline (embed → search → rerank → generate) assumes that semantic similarity is a reliable proxy for document identity. This assumption holds in corpora with unique identifiers (e.g., French legal codes like L233-3) and breaks catastrophically in corpora with shared identifiers (e.g., Spanish Artículo 42 across a dozen codes). Any RAG system entering a new domain must audit for identifier collision before trusting the retrieval layer.

6. A hallucination guarantee without retrieval integrity controls is a liability, not a feature. The vendor marketed a "sub-1% hallucination rate." Under black-box probing the system fabricated legal text on every collision query tried. The guarantee was not false in aggregate — it was catastrophically false on a specific, predictable, and high-stakes class of queries. For regulated deployments, aggregate accuracy metrics are insufficient. Compliance requires demonstrable controls for every known failure mode, with an audit trail proving each control's effectiveness.

7. A guarantee that rests on construction is worth more than one that rests on a benchmark — and it must be labelled as such. The strongest claim in this document (0% wrong-statute retrieval) is also the one with no measurement behind it, because it does not need one: it follows from the semantics of a boolean predicate. The weakest claims are the ones with the most impressive numbers attached (75×, $6,525/month), because those rest on declared assumptions. Presenting both classes under one undifferentiated "Results" heading would have made the document more persuasive and less true. In a domain where buyers conduct technical due diligence, that trade is a bad one.

---

10. Residual Risks & What Would Falsify This

The specification is not risk-free, and the risks do not sit where a casual reader would look for them. The filter is the safe part. The binding step upstream of it is where this architecture can still fail — and one observed failure mode is not addressed by this control at all.

10.1 The Binding Step Is Statistical, Not Structural

The 0% guarantee in §8 holds for a correctly bound query. Extracting statute_id from free-text user input is ordinary NLP and carries ordinary NLP error rates:

• Abbreviation and variant coverage. "CdC", "C. de C.", "Código de Comercio", "el código mercantil", and bare "mercantil" all denote the same statute. Recall depends on gazetteer completeness, which is maintenance, not architecture.
• Colloquial and multilingual queries. Catalan, Galician, and Basque renderings of statute names, plus code-switched Spanish/English queries, are all realistic and all outside a naive gazetteer.
• Silent mis-binding is worse than thrashing. If the router confidently binds the wrong statute, the filter faithfully returns clean chunks from the wrong law and the user receives a fast, well-grounded, wrong answer with no ambiguity signal. This is a more dangerous failure than the slow, obviously-broken behaviour in §3. The audit trail exists precisely so this class is detectable after the fact; Gate 3 in §7 exists so it is caught before cutover.

10.2 Interception Is a Product Cost, Not a Free Win

Every query with a collision and no resolvable keyword becomes a clarification prompt. That is correct behaviour — it is the EU AI Act Art. 14 human-oversight story — but it is friction, and at a high enough interception rate it degrades the product. The right ceiling is a product decision. A specification that ignores this is understating its own cost.

10.3 Temporal Validity Is Out of Scope — Stated Plainly

Test 3 exposed a citation of Ley 62/2003, repealed by Ley 16/2007. The Statute-Binding Router does not fix this. Binding the correct statute says nothing about whether the retrieved provision is currently in force. Temporal validity requires a separate control — corpus versioning with in_force_from / repealed_by metadata and a recency predicate composed alongside the statute predicate. It is a natural extension of the same filtering approach, but it is not delivered by this specification, and claiming otherwise would overstate the control's coverage by one full failure mode out of the three observed.

10.4 What Would Falsify the Core Claim

The structural argument is testable, and these are the outcomes that would break it:

Observation	What It Would Mean
Filtered retrieval returns a chunk with statute_id ≠ bound value	The filter is not applied pre-ANN, or the index metadata is wrong. Implementation defect, and it invalidates the 0% claim until fixed.
Binding precision cannot clear ~99% on real query traffic	The bottleneck is extraction, not retrieval. The architecture still helps but the headline guarantee is unreachable in practice.
Agent still thrashes on cleanly-bound queries	Thrashing had a second cause beyond evidence conflict, and the §6.1 latency model is wrong.
Chunk-level statute_id is unreliable in the corpus	Metadata quality, not retrieval strategy, is the real problem — and a re-index is required, voiding the "no database rebuild" claim.

---

Regulatory Applicability

The retrieval integrity control specified in this document maps to requirements across multiple frameworks. These are design-level mappings — how the architecture would satisfy each requirement once built — not attestations of an audited, certified deployment.

Framework	Requirement	How This Architecture Would Satisfy It
EU AI Act (Art. 14)	Human oversight of high-risk AI	Collision interception routes ambiguous queries to human disambiguation before AI inference
EU AI Act (Art. 15)	Accuracy, robustness, cybersecurity	Deterministic routing eliminates a structurally predictable accuracy failure mode
SRA Principle 2	Uphold public trust in the profession	Prevents AI from presenting fabricated legal text as authoritative citations
SRA Principle 7	Act in best interests of each client	Ensures the correct statute is retrieved before any advice is generated
ABA Model Rule 1.1	Competent representation	Solicitor relying on this system receives statute-bound citations with a recoverable decision trace
NIST AI RMF (Govern 1.1)	Risk management processes	Collision index + audit trail = documented, measurable retrieval risk control
ISO 42001 (A.6.2.6)	AI system quality objectives	0% wrong-statute rate as a measurable, continuously monitorable quality objective

---

Audit Trail & Measurability

Every query passing through the Statute-Binding Router emits a structured log entry. The following is a specimen record conforming to the specified schema — an illustration of the log format, not an extract from a running system:

{
  "query_id": "q-20260415-0042",
  "timestamp": "2026-04-15T14:32:07Z",
  "extracted_article": "42",
  "extracted_statute_keyword": "CdC",
  "collision_detected": true,
  "collision_set": ["CdC", "LGT", "ET", "Ley_15_1988"],
  "resolution_method": "keyword_match",
  "bound_statute": "CdC",
  "binding_confidence": 0.97,
  "metadata_filter_applied": {"statute_id": "CdC", "article_num": "42"},
  "routing_latency_ms": 6.8,
  "human_disambiguation_required": false
}

This schema is designed to satisfy TPRM questionnaire requirements for:

• Decision traceability: Which statute was selected and why
• Anomaly detection: Collision rate monitoring over time
• Incident forensics: If an incorrect response is reported, the exact routing decision is recoverable
• Continuous compliance: The log feeds into a retrieval integrity evaluation dashboard for ongoing monitoring

The binding_confidence field exists because of §10.1 — silent mis-binding is the architecture's real residual risk, and a control that cannot surface its own weakest step is not auditable.

---

TL;DR

Observed: a live, commercially marketed legal-RAG platform advertising sub-1% hallucination returned the wrong statute on 3 of 3 cross-statute probes — tax law quoted as commercial code, a fabricated sub-clause, and a repealed statute cited as current. Vector search cannot disambiguate shared identifiers. ColBERT cannot break the tie. The LLM cannot fix evidence it was never given correctly.

Specified: insert a deterministic retrieval integrity control — a pre-retrieval compliance gate running ~7ms of local CPU inference — between the query and the database, binding statute identity with a hard metadata filter before any embedding computation occurs. Wrong-statute retrieval then goes to zero by construction, because a boolean predicate cannot return a chunk from an unbound statute (§8). Latency and OPEX figures are modeled projections; the residual risk is entity extraction, not filtering (§10); and temporal validity of the retrieved law needs a separate control this one does not provide (§10.3).

The failure is real and reproduced verbatim. The remediation is a reference architecture, not a deployment.