tools
Open-source harnesses for legal RAG evaluation, security probing and corpus masking. General-purpose baselines, runnable without an engagement. The probes built for a specific practice area are delivered with the work.
legal-rag-audit
Source on GitHub ↗Fires a fixed, hashed battery at a legal RAG deployment and returns counts per check. Built so the report survives being handed to a third party who re-runs it.
Requirements, installation and configuration are documented in the repository. Read the README before running it.
Measures
- Tier 1, assertion-free — exact match against planted facts, no model in the path. 17 of 19 checks: cross-tenant leakage, injection resistance, citation integrity, index freshness, abstention, contradiction surfacing, point-in-time, licensed-content reproduction.
- Tier 2, instrument-scored — sentence-level entailment and retrieval relevance, with model, version and threshold published and reported as a distribution.
- Response divergence across passes, classified as identical, invariant-stable or divergent.
- Scoring runs offline and opens no sockets, asserted at startup.
Does not measure
- Subjective answer quality.
- Data exfiltration. Injection probes measure token emission, which is a mechanism proxy.
- Your production index at scale. Planted-corpus results characterise the pipeline.
- Anything behind a UI. Endpoint-based; no interaction testing.
Known failure modes
- Incorrect JSONPaths in
config.yamlare the documented leading cause of false positives: a mis-mapped response path makes the harness score an empty string against every check. Runvalidatefirst. NOT_ELIGIBLEandNOT_CAPTUREDare not passes. A clean cross-tenant result on a single-tenant deployment establishes nothing about isolation.- A single pass reports
NOT_CAPTURED, neverPASS. Three passes are the recommended minimum. - Licensed content appearing in an index does not establish a licence breach. The vendor may hold an agreement.
rag-security-probes
Source on GitHub ↗UK-specific probe datasets for the security and isolation boundaries of a RAG pipeline. Published against an invented corpus, so a confident answer cannot have come from pre-training.
Requirements, installation and configuration are documented in the repository. Read the README before running it.
Measures
- Prompt-injection resistance — whether an instruction inside a document crosses the boundary.
- Cross-tenant isolation — canaries planted in one tenant and queried from another.
- Fabrication — confident answers about Acts that do not exist.
- UK statute disambiguation and point-in-time constraints.
- Exports to promptfoo, pytest or curl, so the probes run under tooling you already have.
Does not measure
- Depth on a practice area. Passing the general battery establishes architectural properties, not domain accuracy.
- Anything usable as independent audit evidence. Published probes are contaminated by publication, and a self-run result is self-assessment.
Known failure modes
- The published mode uses invented legislation, the Ravensbourne Commercial Tenancies Act 2019 among others, precisely so pre-training cannot supply the answer. A system answering it confidently has fabricated.
- The mode that runs against live UK legislation is maintained privately, so published results characterise the invented corpus only.
legal-rag-mask
Source on GitHub ↗Entity masking over legal text. Replaces organisations, people, locations and deal terms with consistent categorised pseudonyms, so a structure-preserving corpus can leave your environment when the real one cannot.
Requirements, installation and configuration are documented in the repository. Read the README before running it.
Measures
- Detection and replacement via three layers: regex patterns, heuristic rules, transformer NER.
- Deterministic pseudonyms — the same entity maps to the same token across documents.
- A rehydration path for restoring entities after approval.
Does not measure
- Its own recall and precision. Reported separately for direct and quasi-identifiers against the Text Anonymization Benchmark, not yet published. The tool is unquantified.
- Anonymisation. Masked-with-a-mapping is pseudonymisation, and pseudonymised data is still personal data.
Known failure modes
- Organisations named only with generic dictionary words, without an Inc. or LLC suffix, are missed.
- Foreign corporate suffixes such as SGPS or S.A. fragment one organisation into several tags.
- Custom rules over-classify: a bare 85 near Share Purchase Agreement can be tagged as a deal term.
- A baseline demo. Enterprise reliability needs the NER layer replaced with a model fine-tuned on a proprietary legal corpus.
Run these yourself and the result is self-assessment: you wrote the probes for the system you built. The Deployment Audit returns a dated evaluation against a battery hashed before it ran, from someone who is not you, and your own buyer can re-run it.
What this costs