What cheaper cross-referencing changes, and what it does not.
Large corpora change the work
Law, medicine, software, finance, and science all produce more relevant material than one person can hold in working memory. Experts cope by specializing, searching, sampling, and relying on summaries. Each method can omit a relationship that crosses its chosen boundary.
Large-context models and retrieval tools can compare more of that material in one analysis. They do not remove limits: attention degrades, sources conflict, retrieval misses documents, and longer inputs cost time and money. They change the amount of material that can be checked together.
For material that exceeds the context window, an agent can follow a citation or dependency, retrieve the next source, and continue. This remains sequential search with a lossy working record, not simultaneous knowledge of the corpus. The practical gain is lower-cost cross-referencing across a larger set of documents.
Give the model decision criteria
The quality of the result depends on the standard against which the model evaluates the material.
An instruction such as “summarize these contracts” asks for description. A criterion such as “indemnification must be reciprocal, and liability caps may not exclude gross negligence” supports evaluation.
Both are prompts to the model. The useful distinction is whether the human has stated the governing rule precisely enough to test. Turning “find risks” into explicit contract requirements is substantive legal and business work, not prompt decoration.
- Instruction: “Read these 50 vendor contracts and summarize the risks.”
- Criteria: “Our contracts require reciprocal indemnification, and liability caps may not exclude gross negligence. For each of these 50 contracts, quote any conflicting clause, identify cross-references that change its effect, and cite the source page.”
The first task produces a summary. The second produces evidence that a qualified reviewer can audit. The model compares text with a stated standard, while quotations and citations make the comparison inspectable.
With explicit criteria and accessible sources, a model can:
- compare each retrieved claim with the stated rule;
- follow cross-references and record where a chain ends or becomes ambiguous;
- group recurring exceptions and contradictions; and
- produce citations that let a reviewer reproduce the analysis.
Writing good criteria is difficult because rules are contextual, contradictory, and full of exceptions. A flawed rule applied consistently can create a systematic error that looks rigorous.
The reviewer must therefore test the criteria against representative cases, inspect false positives and false negatives, and revise the rule. The model applies the current rule; the accountable expert decides whether the rule is valid.
Check claims against sources
Documentation describes behavior. Metadata assigns categories. Indexes record relationships. Citations claim support. Each artifact can drift from the source it represents.
The verification problem appears in many domains:
- Law: an authority table can misstate which statute authorizes a regulation.
- Code: documentation and comments can drift from runtime behavior.
- Medicine and science: a review may depend on methods or conflicts reported by the underlying studies.
- Finance: an analysis may inherit an institution’s classification of its own exposure.
A model can compare many such claims with their cited sources and identify mismatches for review. Coverage is bounded by retrieval, context, and the quality of the parser; calling the process exhaustive does not make it so.
Entity resolution, versioning, ambiguity, conflicting sources, and non-local context remain difficult regardless of scale. Trust depends on corpus quality, precise criteria, measured coverage, and validation. The model makes a larger comparison feasible; it does not establish that the comparison is correct.
Division of responsibility
The expert defines the question, criteria, acceptable evidence, and consequence of error. The model searches, compares, and drafts an account of what it found. Deterministic code performs calculations and structural checks where possible. The expert reviews disputed cases and remains accountable for the conclusion. This allocation is less dramatic than assigning creativity to people and rigor to machines, but it can be tested.
The Failure Mode
The approach fails when a model replaces analysis with an implementation plan it cannot test.
Ask a model to solve an underspecified optimization problem and it may return a stochastic mixed-integer formulation, Pyomo code, a retraining schedule, and an integration design. The response can look complete while omitting evidence that the formulation fits the data, scales to the instance size, or can be operated by the available team.
Those omissions determine whether the proposal can be built. A named method and syntactically valid code do not answer them.
The failure need not involve a fabricated fact. The papers and solvers may be real while the implied claim remains unsupported: that this formulation will work for this problem, scale, budget, and team. Fluency supplies no evidence for that claim.
The user needs a tractable formulation, stated approximations, resource estimates, and a validation plan. An expert may use the model to produce and compare those elements, but the proposal must meet operational evidence before anyone calls it a solution.
An independent critique can expose an omission, but disagreement between two models does not decide the issue. The useful answer identifies a tractable version, its approximations, what those approximations sacrifice, and the test that would reject it.
Execution supplies part of that test. Sandboxed interpreters, compilers, solvers, and model checkers force a proposal to encounter syntax, memory, runtime, and convergence limits immediately. A MemoryError or solver timeout provides evidence for revision. Execution still cannot validate an incorrect objective, missing constraint, or biased dataset, so domain review remains necessary.
I wrote more about the failure mode in When a Model Proposes a System It Has Not Tested.
What This Means for Building
Keep models in the analytical loop by giving them governed data, inspectable tools, and opportunities to test their proposals. Do not replace analysis with a fixed workflow merely because the workflow is easier to implement.
Build:
- governed access to primary sources through MCP tools, databases, or graph queries;
- context caching where repeated analysis justifies its cost;
- iterative retrieval that records each query, result, and source;
- sandboxed interpreters, compilers, and solvers for executable proposals;
- verification that compares claims with quoted source material; and
- explicit evaluation criteria, including exceptions and escalation rules.
Avoid unless a stated requirement justifies them:
- summaries that prevent access to the underlying source;
- fixed reasoning flows that hide assumptions in application code;
- consensus rules that treat agreement as evidence;
- prompt templates that obscure the actual instruction; and
- retrieval that exposes only top-ranked fragments with no route to the full document. Top-K retrieval remains useful for security, governance, cost, and auditability, but its coverage limits must be explicit.
A stronger model can improve a system built around stable sources, criteria, and tools. It cannot repair inaccessible data, an invalid criterion, or a workflow that prevents it from examining contrary evidence.
The Test
For any system you’re building with AI, ask:
- What claim requires model reasoning? Use a database query or deterministic program when either can answer the question directly.
- Which criteria govern the answer? State the rule, exceptions, and consequence of error.
- What can the model inspect? Record source coverage and provide a route from retrieved fragments to full documents.
- How will the proposal be tested? Require execution, citations, or another deterministic check appropriate to the claim.
- Who decides? Name the person accountable for disputed assumptions and the final action.
The goal is a system whose claims and actions can be inspected and challenged.
What’s Missing From This Essay
This essay lacks an end-to-end case study showing the criterion, retrieval path, finding, and validation. Until such cases exist, its design recommendations remain hypotheses. Some will survive implementation and others should be revised or rejected.
This is Part 7 of an ongoing series. Previously:On Running a Startup of Claude Code Agents, A Postmortem on Failing a Power User, Dialogues with Claude Code, Zen of Unix Tools: Code is Context, The Disintermediation Principle, Proved by Its Own Negation, When a Model Proposes a System It Has Not Tested.