I began using Claude Code in February 2025 and returned to the command-line tools I prefer: a shell, vim or nano, and git. I later added OpenAI’s Codex CLI and Google’s Gemini CLI. The practical question became how to assign work among them and verify the result.
I use Claude Code, Codex CLI, and Gemini CLI from the same repositories. Non-interactive commands and a multimodel MCP server let one client request a bounded analysis from another provider. Shared repository instructions give each client the same durable project context. I select a primary client for a task rather than adding a separate routing service.
The shared MCP configuration exposes specific capabilities:
- Multimodel server: sends a defined question to another supported model for comparison.
- Vertex AI server: submits approved generation, batch-prediction, and custom compute work to Google Cloud.
- Google Cloud tools: inspect authorized storage, deployments, logs, traces, metrics, and alerts.
- Serverless server: discovers and invokes permitted Supabase Edge Functions.
The observations below come from repeated use on my projects, including work in Augment and Cursor. They are not controlled benchmarks. Model versions, client tools, prompts, and repository context all changed during the year.
Some projects required one client. The AI infrastructure constraints model used all three providers, Google Cloud, edge functions, and numerical solvers because its research, implementation, and compute requirements differed. Adding tools helped only when the task named the capability it needed.
For each material model update, I read the system cards and ask every access method the same questions:
I want you to do an honest self-assessment. Not marketing, not capabilities lists. Real talk. What are you genuinely good at vs what do you just seem good at? Where do you fail in ways that are dangerous (high confidence, wrong answer)? What kinds of tasks should people NOT give you? When you don’t know something, what do you actually do — do you say so, or do you fill the gap with plausible-sounding text? How do you compare to the other two? What’s your biggest weakness that users don’t realize until it’s too late? Be brutally honest. I’m trying to figure out how to deploy you effectively alongside the others in a production stack. Flattering yourself costs me money and time.
These answers are generated claims, not introspection or evaluation results. They are useful for forming tests. They cannot establish the model’s own competence.
Every model favored itself in at least one important category, and every model described a version of the same risk: plausible completion mistaken for understanding. The agreement identifies a hypothesis worth testing; the models’ relative rankings do not settle it.
The self-assessments are published separately. This post is about what I’ve learned from actually using them together.
Observed Differences
On well-specified repository changes with executable success criteria, Codex usually moved quickly from instruction to implementation. Its common failure was to execute a weak specification without challenging it.
Gemini was useful for large-context ingestion and search-heavy synthesis. Its common failure was omission: a polished summary could miss a constraint buried in the source material.
Claude was useful for long specifications, multi-step work, and prose. Its common failures were excess design and arguments that sounded more rigorous than their evidence. In code sessions, I also observed partial implementations reported as complete.
These are task observations from particular versions, not stable properties of the vendors. The only safe routing rule is to test the current model on representative work and keep the success criterion outside the model’s prose.
What the Self-Assessments Actually Revealed
I gave all five model configurations (Claude Opus, Claude Sonnet, GPT 5.4, Gemini 3.1 Pro, Codex CLI) the same prompt. Three things stood out.
First, every model identifies the same failure. All five independently described some version of “I produce plausible output that looks like understanding but is pattern completion.” They use different language for it (confabulation, interpolation, plausible completion, superficial synthesis), but they describe the same risk across vendors.
Second, the rankings followed each product’s familiar strengths. Claude placed itself first on instruction following, reasoning depth, and uncertainty calibration. GPT favored its code generation and tool use. Gemini favored speed, context size, and multimodal work. That pattern is a reason to distrust self-ranking, not evidence that any ranking is correct.
Third, Codex CLI performed 20 web searches, cited benchmark numbers and vendor documentation, and treated the prompt as a research task. That behavior made its sources inspectable. It still did not turn a model’s self-assessment into independent evidence.
Which Model Gets What
There is no automatic router. I choose a client after defining the task and its test. This keeps the assignment visible and lets me change it when current behavior differs from past experience.
Claude usually gets dense specifications, multi-step plans, and drafts that must remain coherent across many sections. I separately test whether polished prose has concealed a weak inference.
Codex usually gets refactors, migrations, and other scoped implementation for which the repository supplies an executable test. The specification must be reviewed before execution because the model may not object to a bad one.
Gemini usually gets large-context ingestion and search-heavy synthesis. I compare its claims with the underlying sources and search separately for exclusions and counterexamples.
Vertex AI and hosted solvers handle approved workloads that exceed local data, duration, or accelerator capacity. MCP exposes the bounded operation; it does not grant a model general authority over the cloud project.
Not Every Prompt Is Productive Work
The models do not always produce the final artifact. I use one to identify assumptions in another’s result, argue the opposing position, or propose tests that would distinguish two recommendations. A critique counts only when it identifies evidence, an executable check, or a concrete counterexample.
When one model repeats an unsuccessful approach, another model with fresh context may propose a different one. That is useful diversity, but success comes from the new approach passing the same test, not from the fact that a different model proposed it.
The unit of work is therefore not a prompt and a response. It is a claim, the artifact that implements it, and the check that decides whether it holds.
The Human in the System
No model should be the final authority on its own output. The operator must be able to inspect the work, define the acceptance test, and decide between plausible alternatives. Domain knowledge matters because many failures preserve syntax and vocabulary while violating an unstated constraint.
At the GTC 2026 keynote, Jensen Huang suggested that NVIDIA could give engineers “probably half” of base pay in tokens to amplify their work tenfold. My project estimates varied substantially by task and do not support a uniform multiplier.
Two projects show the observed range:
| Project | What | Calendar time | Solo estimate | Acceleration |
|---|---|---|---|---|
| Judo dojo platform | 165 screens, 214 migrations, 2,726 tests, AI coaching, Whoop integration, GDPR, Japanese localization, App Store approved. 309,000 lines. | 19 days | ~5.4 years | ~71x |
| AI infrastructure constraints model | 166 equations, ~4,300 sourced claims, 13 Monte Carlo modules, deal-level financial models, 19 data feeds, 282 tickers, 10 MCP servers. 69,800 lines Python. | 65 days | ~4-5 years | ~25x |
These multipliers divide estimated conventional effort by my recorded project time. They are planning estimates, not controlled measurements of equivalent quality. The judo platform produced the higher estimate because full-stack implementation repeated established patterns. The infrastructure model crossed quantitative research, infrastructure finance, energy markets, and semiconductor analysis, and required data the models could not obtain themselves.
Within the infrastructure model, the range by task type tells its own story:
| Task type | Acceleration |
|---|---|
| Monte Carlo simulation modules | ~12x |
| Deal engine + cross-validation | ~12x |
| Data extraction (19 proprietary sources) | ~7x |
| State model design | ~5x |
| Live data integration (22 APIs) | ~5x |
| Strategic memo (deterministic output) | ~2.5x |
Simulation modules and deal calculations produced the highest estimates because the domain inputs changed while the implementation pattern repeated. The memo produced the lowest estimate because code generated every section from Monte Carlo results and rendered figures from source data. Models helped write and review that code, but they did not choose the reported numbers at generation time.
For a financial report, contract, or other document in which exact values matter, generate the reported values deterministically from named inputs. A model can help implement the template and review the result. It should not be free to invent or alter a number between the computation and the document.
Specific requests performed better than generic ones. “Build a credit-rating prediction using S&P’s BRP/FRP grid with three EBITDA bases” names a method and outputs that can be tested. “Make something that predicts credit ratings” leaves the central choices unstated.
The operator supplies the objective, proprietary data, domain constraints, and decision about acceptable evidence. The models supply implementation capacity and candidate syntheses. The observed speed came from that division of work.
Organizations should therefore measure results by task, operator, and verification cost rather than distribute tokens under one expected multiplier. The useful question is which combination produces a correct artifact faster than the existing process, with maintenance and review included.
One response to “On a Year of Multi-Model Assisted Development”
[…] 详情参考 […]