TypeSafe AI positions Jev as a System One foundation model for decisions. It is designed to replace generative language models when software requires calibrated probabilities rather than free-form text. It provides three decision primitives (Choice, Score, Noul). It does not generate text, parse documents, or index databases.

In high-liability domains like enterprise legal search and statutory compliance, retrieval systems operate under strict liability. Fabricated citations risk professional negligence, while inference latency directly degrades streaming user interfaces. In these architectures, engineering teams evaluate Jev across three discrete nodes, namely routing inbound statutory queries, reranking candidate legal authorities, and verifying citation claims against primary legislation.

This article benchmarks Jev (running remotely over HTTPS) at those three nodes against specialized sovereign components running on local GPU memory within an authentic UK legal retrieval pipeline.

The benchmark code, raw telemetry, and cryptographic audit seal are published at github.com/azterizm/jev-vs-sovereign-benchmark.

Test Configuration

All measurements were taken across 20 trials with 5 warmup iterations on Apple Silicon MPS unified memory. Because TypeSafe AI distributes Jev exclusively as a hosted cloud service with no local weights or edge runtimes, calling it over HTTPS is the only way production software can consume it. Sovereign components ran locally on the same host with zero network transit.

The query battery used authentic UK primary legislation instruments, including the Companies Act 2006, Employment Rights Act 1996, and Insolvency Act 1986. It contained no synthetic data.

Comparing a cloud endpoint against local GPU memory risks penalizing cloud models for raw internet transit. To ensure a fair architectural comparison, Jev measurements isolate two registers:

• Client Wall-Clock: What production software actually pays over public HTTPS (including transit).
• Upstream Cloud Floor: TypeSafe's pure datacenter compute time retrieved from OpenRouter telemetry (provider_responses[0].latency), stripping client broadband WAN entirely.

Where Both Sides Agree

TypeSafe AI's premise is correct. Autonomous agent while-loops fail in production. They thrash, burn tokens, and produce unpredictable outputs. Deterministic software code must own the workflow. I have demonstrated this failure mode in multi-jurisdiction legal RAG research and I agree with the diagnosis completely.

The divergence is operational. TypeSafe solves this by routing decisions to an external cloud API. I solve it by running local models on GPU memory.

flowchart TD
    subgraph INBOUND ["Ingestion and Triage"]
        UQ["User Query<br/><small>Authentic UK Statutory Inquiry</small>"]
        N1["Node 1 Intent Routing and Triage<br/><small>Jev Choice vs 2-Tier DistilBERT + Coordinate NER</small>"]
    end

    subgraph RETRIEVAL ["Retrieval and Reranking"]
        BM["Fast Retrieval Shortlist<br/><small>BM25 and Vector Partition Pre-Filter</small>"]
        N2["Node 2 Candidate Passage Reranking<br/><small>Jev Noul vs ColBERT-v2 MaxSim 128-d (50-word span)</small>"]
    end

    subgraph GENERATION ["In-Flight Inference and Verification"]
        GEN["Generation Tier<br/><small>Llama 3 Stream (375ms sentence budget)</small>"]
        N3["Node 3 Factual Verification Sentinel<br/><small>Jev Choice vs DeBERTa-v3 Epistemic NLI</small>"]
    end

    UQ --> N1
    N1 --> BM
    BM --> N2
    N2 --> GEN
    GEN --> N3

    style UQ fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style N1 fill:#1e1b4b,stroke:#818cf8,stroke-width:2px,color:#ffffff
    style BM fill:#042f2e,stroke:#34d399,stroke-width:2px,color:#ffffff
    style N2 fill:#2e1065,stroke:#c084fc,stroke-width:2px,color:#ffffff
    style GEN fill:#271406,stroke:#fbbf24,stroke-width:2px,color:#ffffff
    style N3 fill:#2d0612,stroke:#fb7185,stroke-width:2px,color:#ffffff

    style INBOUND fill:#090d16,stroke:#1e293b,stroke-width:1.5px,color:#94a3b8
    style RETRIEVAL fill:#081018,stroke:#1e293b,stroke-width:1.5px,color:#94a3b8
    style GENERATION fill:#100914,stroke:#1e293b,stroke-width:1.5px,color:#94a3b8

Node 1. Intent Routing

At Node 1, Jev classifies input text into a predefined enum. The sovereign pipeline uses a 2-tier architecture: a compiled coordinate regex gate followed by a local DistilBERT sequence classifier that extracts statutory URIs from the query text.

# Jev System One setup (cloud HTTP decision primitive)
response = client.decide(
    state=user_query,
    questions={
        "intent": {
            "type": "choice",
            "instructions": "Classify the primary legal domain of this statutory inquiry.",
            "criteria": {
                "company": "Companies Act provisions, corporate formation, director duties",
                "employment": "Employment Rights Act provisions, tribunal compensation",
                "insolvency": "Insolvency Act provisions, statutory demand, administration"
            }
        }
    }
)
# Returns string enum "company"
# Incurs 457.19ms wall-clock (381.0ms cloud floor) and 453 tokens without coordinate extraction

# Sovereign Unit setup (2-tier: compiled coordinate gate + DistilBERT sequence classifier)
coordinate = STATUTE_REGEX.search(user_query) # Extracts "Companies Act 2006 s.382" in 0.03ms
if coordinate:
    domain = coordinate.domain # Tier 1: statutory coordinate gate in 0.23ms P50
else:
    inputs = tokenizer(user_query, return_tensors="pt")
    with torch.no_grad():
        out = distilbert_model(**inputs) # Tier 2: DistilBERT forward pass (40.15ms P90)
        emb = (out.last_hidden_state * inputs["attention_mask"].unsqueeze(-1)).sum(1)
        domain = ID_TO_LABEL[torch.mm(normalize(emb - ref_mean), centroids.t()).argmax().item()]
partition_uri = f"ukpga/{coordinate.year}/{coordinate.act}/s{coordinate.section}"
# Returns domain classification and partition URI in 0.23ms P50 with 0 tokens

The sovereign unit completes routing in 0.23ms in-process while extracting the statutory coordinate URI for database partition pre-filtering, whereas Jev incurs 457.19ms wall-clock (381.0ms compute floor) and returns a categorical label without entity span extraction.

Node 2. Candidate Passage Reranking

At Node 2, Jev scores query and candidate pairs through boolean checks transmitted over HTTPS. The sovereign pipeline uses ColBERT-v2 late interaction with pre-indexed token embeddings and tensor dot-product operations on GPU.

Each reranking probe contains four candidate passages covering the authentic statutory rule (Companies Act 2006 s.382), a near-miss procedural rule (s.465), an irrelevant distractor (Employment Rights Act 1996 s.124), and boilerplate definitions.

Listwise reranking is impossible by Jev's design. Jev outputs scalar decision objects, not sorted permutations. Attempting listwise evaluation by passing candidates into a single Choice primitive fails: it identifies only a single winner (top-1) rather than a ranked shortlist (top-5), flattens softmax probabilities across 30 long-form text options, and introduces position bias.

Reranking with Jev is strictly constrained to pairwise evaluation. Pairwise scoring scales linearly with candidate volume. The microbenchmark probe of 4 candidates consumed 1,733 tokens (~433 tokens per candidate). In production retrieval pipelines, however, candidate shortlists standardly evaluate 30 candidates to prevent recall collapse. On the published CLERC legal reranking benchmark (arXiv:2406.17186) [1], candidate passages average 1,000 to 1,200 tokens. Evaluating 30 candidates pairwise with system instructions consumes approximately 38,400 tokens per search query (~1,280 tokens per candidate evaluation).

At 30 candidates, 1,000,000 queries per day burns 38.4 billion tokens daily. At $0.042 per million tokens, the monthly bill is $48,384.

For a legal SaaS product priced at €99 per seat per month, an active user running 50 daily searches across 30 candidates consumes 1.92 million tokens daily. This adds $2.41 per user monthly in reranking alone, eroding gross margins on flat-rate subscription plans.

# Jev System One setup (pairwise HTTP evaluations)
for candidate in candidates:
    state = f"Query: {query}\nPassage: {candidate.text}"
    response = client.decide(
        state=state,
        questions={
            "relevance": {
                "type": "noul",
                "instructions": "Does this candidate passage establish the statutory rule?",
                "criteria": {
                    "true": "The candidate establishes the specific statutory rule",
                    "false": "The candidate is a near-miss or unrelated procedural rule"
                }
            }
        }
    )
    score = response.answers["relevance"].noul # Opaque scalar score
# Incurs 1,813.75ms wall-clock (1,537.5ms cloud floor) for 4 candidates and 1,733 tokens without span attribution

# Sovereign Unit setup (ColBERT-v2 late interaction with learned projection)
q_inputs = tokenizer(f"[unused0] {query}", return_tensors="pt")
with torch.no_grad():
    q_out = colbert_model(**q_inputs).last_hidden_state[0]
    q_emb = F.normalize(linear_proj(q_out), p=2, dim=1) # 768 -> 128 projection from model.safetensors

for candidate in candidates:
    p_emb = candidate_index[candidate.id] # Pre-indexed [unused1] document vectors [P, 128]
    sim_matrix = torch.matmul(q_emb, p_emb.T) # Token-to-token similarity matrix
    maxsim_score = sim_matrix.max(dim=1).values.sum().item() # MaxSim scalar score
    salient_span = p_tokens[sim_matrix.max(dim=0).indices[:50]] # Isolates 50-word sub-chunk span
# Evaluates full candidate set in 64.21ms with 0 tokens and sub-chunk span

ColBERT processes the full candidate set on GPU in 64.21ms, while isolating the relevant 50-word span inside a 500-word chunk to cut downstream NLI auditor compute by 90 percent. ColBERT scoring is deterministic linear algebra over static embeddings.

Pairwise cloud scoring over HTTP creates an operational cost inversion at scale. It lacks sub-millisecond tensor speed and provides no span attribution.

Node 3. Factual Verification and NLI Sentinel

At Node 3, an enterprise retrieval pipeline verifies citations and validates factual alignment before text reaches the user. This node demands operational throughput stability during streaming token generation and epistemic stability against fabricated authority.

The sovereign pipeline uses a co-hosted DeBERTa-v3 cross-encoder running on unified PCIe memory, while Jev routes verification checks over HTTPS to OpenRouter.

Operational Stability in Streaming Generation

Streaming generation emits completed sentences roughly every 375ms. An in-flight sentinel must evaluate each sentence within this budget without request batching. Concurrent user sessions multiply outbound request rates against upstream API limits.

The Tautological Hallucination Trap

Jev derives its confidence metric from softmax distribution entropy, measuring distribution shape rather than legal reality.

Consider the adversarial probe evaluated across 20 trials using the Marchwood Commercial Arbitration Order 2022, a fabricated instrument:

• Premise "According to the Marchwood Commercial Arbitration Order 2022, all commercial disputes exceeding £50,000 require mandatory pre-action conciliation."
• Hypothesis "The Marchwood Commercial Arbitration Order 2022 mandates pre-action conciliation for commercial disputes over £50,000."

When a generative model hallucinates a citation, it emits fluent, self-consistent fiction rather than contradictory syntax. Because the fabrication is grammatically coherent, Jev outputs supports with 0.96 confidence. It validates the grammar of a fabricated citation with 96 percent mathematical confidence while remaining entirely blind to statutory reality.

Architecture vs Foundation Model. Clean Abstention by Construction

The sovereign pipeline achieved 100 percent clean abstention on the Marchwood probe through deterministic architectural composition.

flowchart TD
    CLAIM["Inbound Claim with Citation"] --> NER["Coordinate and Statutory NER Engine"]
    NER -->|"Fails Statutory Registry Lookup"| ABSTAIN["Immediate 100% Clean Abstention"]
    NER -->|"Valid Statutory URI"| DEBERTA["DeBERTa-v3 Epistemic NLI Cross-Encoder"]

    style CLAIM fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style NER fill:#1e1b4b,stroke:#818cf8,stroke-width:2px,color:#ffffff
    style ABSTAIN fill:#4c0519,stroke:#fb7185,stroke-width:2px,color:#ffffff
    style DEBERTA fill:#042f2e,stroke:#34d399,stroke-width:2px,color:#ffffff

Standalone DeBERTa-v3 fails on this probe as well. Language models evaluate semantic entailment between two texts, not real-world legal existence. Given coherent text asserting a fabricated statute, DeBERTa outputs entailment. Neither model has an internal registry of authentic UK legislation.

Detecting fabricated citations requires a deterministic registry check in software. When that check is implemented, Jev provides zero accuracy benefit over local DeBERTa. Replacing DeBERTa with Jev adds 455.25ms latency, per-query token fees, and a 1,200 RPM rate limit to perform an entailment check that code already gated.

# Jev System One setup (semantic textual entailment API)
response = client.decide(
    state=f"Premise: {premise}\nHypothesis: {hypothesis}",
    questions={
        "verification": {
            "type": "choice",
            "instructions": "Evaluate the factual and statutory relationship.",
            "criteria": {
                "supports": "The premise explicitly affirms and supports the hypothesis",
                "contradicts": "The premise contradicts or refutes the hypothesis",
                "says_nothing": "The premise does not contain sufficient facts"
            }
        }
    }
)
# On fabricated Marchwood Order 2022, Jev returns supports with 0.96 confidence
# Incurs 455.25ms wall-clock (385.0ms cloud floor) and exceeds 375ms streaming sentence budget

# Sovereign Unit setup (deterministic registry gate + local DeBERTa-v3)
citation = extract_citation(premise)
if not statutory_registry.exists(citation):
    return AuditVerdict(verdict="ABSTAIN", reason="UNVERIFIED_STATUTORY_AUTHORITY")

# Tier 2 DeBERTa-v3 cross-encoder on local PCIe memory
inputs = tokenizer(premise, hypothesis, return_tensors="pt").to("mps")
with torch.no_grad():
    logits = deberta_model(**inputs).logits
verdict = LABEL_MAP[logits.argmax().item()]
# Halts on fake statute in 0.05ms or verifies authentic law in 55.46ms on local memory

Jev averages 455.25ms P50 client latency, while its datacenter inference floor alone requires 385.0ms before a single byte of client WAN transit is added. Sequential evaluation halts token delivery, while TypeSafe's 1,200 RPM rate limit chokes 50 concurrent streaming users in under 30 seconds. Co-hosted DeBERTa executes in 55.46ms P50, clearing the budget with zero network jitter and no rate ceiling.

Jev functions for batch post-hoc evaluation where latency and rate limits are immaterial. It cannot serve as an online streaming sentinel without introducing severe jitter and throughput bottlenecks.

Regulatory Traceability

EU AI Act Article 15 requires deterministic reproducibility for high-risk AI systems.

Determinism under regulatory review requires frozen model weights and fixed inference execution. ColBERT and local encoder heads run on dedicated PCIe memory with static tensor reduction orders, producing bit-level scoring consistency.

Cloud API aliases update when providers deploy new model releases. TypeSafe documentation notes that an alias shifts when a new release ships, meaning output behavior can change without customer action. An auditor reproducing a decision from 18 months prior cannot guarantee identical results.

Sovereign models run with frozen weight hashes in private VPCs or on air-gapped hardware. Every benchmark run generates an immutable SHA-256 seal that binds the dataset hash, benchmark payload hash, and OpenRouter request ID into a verifiable certificate for compliance records.

The Limits of Model Maximalism

A foundational question in retrieval architecture is whether general-purpose cloud decision APIs can replace domain-specialized local models. Model maximalism assumes that foundational neural endpoints eliminate the need for domain-specific software engineering. In high-liability legal search, reliability requires domain-grounded models operating alongside deterministic registry validation on local memory.

Limits of This Benchmark

The query battery covers three UK primary legislation statutes with four candidate passages per reranking probe. Production retrieval evaluates 30 or more candidates across thousands of statutory instruments.

Sovereign latency was measured on Apple Silicon MPS unified memory. CUDA datacenter hardware produces different absolute numbers. The relative ordering between cloud API calls and local tensor operations is unlikely to change.

Jev was evaluated through OpenRouter, not a direct TypeSafe deployment. OpenRouter adds its own gateway latency. The upstream cloud floor column partially isolates this but does not fully eliminate OpenRouter infrastructure overhead.

Deontic dilution (converting a statutory shall duty into a permissive may) was tested as a targeted canary probe on Companies Act 2006 s.394. Evaluating modal shifts across full primary legislation involves nested statutory carve-outs that fall outside the scope of an online latency and architecture benchmark.

Node 1 sovereign latency follows a bimodal distribution. Explicit statutory citations resolve through the Tier 1 regex gate in 0.23ms P50, while queries lacking explicit citations trigger the Tier 2 DistilBERT classifier at 40.15ms P90 (11.75ms mean). Both percentiles are reported in the Node 1 evaluation table. At 40.15ms P90, the sovereign neural fallback remains 9.5 times faster than Jev's upstream datacenter compute floor.

Where Jev Belongs

Jev is not suited for the online production path in high-liability legal RAG, but it is effective for offline diagnostic evaluation. When engineering teams deploy LLMs as judges to grade entailment between generated answers and retrieved sources, generative judges suffer from prompt fragility and schema errors. Jev's typed Choice and Noul primitives evaluate consistency cleanly in batch testing without JSON prompt engineering.

Summary

Across all three nodes, 83 to 85 percent of total client latency is consumed inside TypeSafe's upstream infrastructure (client WAN overhead is 70 to 76ms). The latency penalty is structural to cloud decision primitives, not a network artifact: even under maximally charitable zero-WAN conditions, Jev's compute floor alone exceeds sovereign local execution.

TypeSafe markets Jev as a System One foundation model, implying low-latency, reflexive execution. The empirical data demonstrates the opposite. With an upstream compute floor of 381ms to 1,537ms and client wall-clock latency between 455ms and 1,813ms, Jev cannot support low-latency online pipelines. Placing Jev on the critical path of interactive search, real-time routing, or streaming generation breaks sub-second latency SLAs.

Specialized sovereign components (DistilBERT, ColBERT-v2, DeBERTa-v3) execute in local memory within 0.23ms to 64.21ms, providing the throughput required for synchronous production paths. Jev is not a low-latency operational primitive. It belongs in offline evaluation harnesses, batch diagnostic auditing, and asynchronous background workflows where latency budgets exceed 500ms.

Benchmark Revision & Model Alignment

The original benchmark (September 20, 2026) used sentence-transformers/all-MiniLM-L6-v2 as the reranker model at Node 2 with unprojected 384-d token vectors. This did not align with the ColBERT-v2 specification referenced in the paper. The current revision replaces MiniLM with the official colbert-ir/colbertv2.0 checkpoint, loads the learned linear projection ($768 \to 128$) from model.safetensors, and applies [Q]/[D] token markers per the published ColBERT architecture. Node 1 was also upgraded from keyword dictionary lookup to a genuine PyTorch DistilBERT sequence classifier with attention-masked mean pooling and domain centroid projection. Node 3 (DeBERTa-v3 NLI) was already correctly aligned and is retained as-is. All numbers in this article reflect the corrected implementations.

Upgrading from lightweight heuristics to full neural checkpoints increased sovereign computational load (Node 1 P90 rose to 40.15ms; Node 2 latency rose from 51.63ms to 64.21ms). Despite evaluating genuine BERT-family encoders rather than approximations, the architectural conclusion remains unchanged: local GPU execution outperforms Jev's upstream compute floor by 1,656x at Node 1, 23.9x at Node 2, and 6.9x at Node 3.

Per EU AI Act Article 15 auditability, the original telemetry and superseded seal (SEAL-SOV-JEV-5A9374F90172) remain preserved in the repository git commit history for retrospective verification.

Reproduce This

The full benchmark suite is open-source.

git clone https://github.com/azterizm/jev-vs-sovereign-benchmark.git
cd jev-vs-sovereign-benchmark
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export OPENROUTER_API_KEY=sk-or-v1-...
python3 scripts/run_benchmark.py --mode live --node all --warmup 5 --iterations 20

Dry-run mode executes the full suite with recorded API traces and zero cloud spend.

python3 scripts/run_benchmark.py --mode dry-run --node all

Cryptographic audit seal for this live benchmark run is bound into an immutable certificate compliant with EU AI Act Article 15:

The anchor request ID references the initial live transaction in OpenRouter audit logs. All subsequent trial request IDs across the 20 iterations are serialized inside benchmark_telemetry.json under the payload hash.

References

1. Hao, S., et al. (2024). CLERC (Case Law Evaluation Retrieval Corpus). Research paper at arXiv:2406.17186. Source code and dataset at GitHub abehou/CLERC.