Jason A. Hoffman, PhD
AI demand forecasts often pass through one hidden conversion. More adoption produces more queries; more queries require more centralized inference; more centralized inference requires more datacenter power. The first step can be true while the last two weaken.
A new Stanford paper, Intelligence per Watt: Measuring Intelligence Efficiency of Local AI, measures how much single-turn chat and reasoning work recent local models can handle. Its result should change forecasts that translate query growth directly into centralized megawatts. It does not justify an 88.7 percent discount to inference demand, much less to total datacenter growth. It establishes something more useful: workload placement is now a variable.
Frontier training can remain centralized and continue to scale while a growing share of routine inference moves onto devices. The datacenter creates the capability frontier. Endpoints absorb capabilities after models and hardware make them cheap enough to run locally.
What the paper establishes
The researchers evaluated more than twenty local language models on eight accelerators. Their main task set contains one million single-turn chat and reasoning queries, supplemented by knowledge and expert-reasoning benchmarks. They define a local model as one that activates no more than 20 billion parameters for each forward pass. That definition includes mixture-of-experts models with much larger total parameter counts.
The strongest individual local system was GPT-OSS-120B on an Apple M4 Max with 128 GB of unified memory. It handled 71.3 percent of the selected queries under the paper’s success criteria. The corresponding best system from 2023 handled 23.2 percent. Over those two years, intelligence per watt improved 5.3 times through gains in both models and hardware.
The widely quoted 88.7 percent result answers a different question. For each query, the researchers looked across more than twenty local models and counted the query as locally solvable when any one of them succeeded. That is an oracle result: the selector knows which answer is correct after the models have run. It measures the combined reach of the local model set, not the performance of a router that can be deployed today.
The definition of success also varies by task. Benchmark questions use their reference answers. Open-ended chat responses use an LLM judge to decide whether the local answer wins or ties against a much larger reference model. “Handled” is therefore more accurate than “answered correctly” across the whole study.
The paper’s scope matters. The main evaluation covers single-turn text with inputs capped at 32,000 characters. It does not measure multimodal use or production workloads with very long contexts. An appendix tests 245 multi-turn agentic queries on GAIA and TerminalBenchV2. The authors call that experiment a sanity check, not a definitive characterization. On identical models, the local system was 12 to 15 times slower and the cloud system delivered 2.4 to 3 times more intelligence per joule. Local execution remained competitive on answer quality, but the evidence for agents is much thinner than the million-query headline.
The installed fleet sets another limit. A 128 GB M4 Max shows what a high-end local system can do. It does not describe the memory, software, or thermal capacity of the ordinary personal-device fleet. A separate smartphone test ran three small models on an iPhone 16 Pro. Neither result supplies an adoption curve for capable devices.
Local placement is not a silicon-efficiency victory
The hardware comparison prevents the easiest misreading. For identical single-turn workloads, an NVIDIA B200 produced about 1.4 times more intelligence per watt and 1.6 to 2.3 times more intelligence per joule than the M4 Max. SambaNova’s SN40L reached as much as 7.4 times the local system’s intelligence per joule on the paper’s comparisons.
The units answer different questions. Watts measure the power the device must sustain. Joules include the time required to finish the query and therefore measure its energy. A low-power device can satisfy the first constraint while losing the second comparison because it runs longer.
Those tests used batch size one to isolate the model and accelerator. Cloud inference normally serves concurrent users. When the researchers raised the B200 batch size to 64, its intelligence per joule improved another 11 to 20 times. The paper’s routing simulation accounts for part of that advantage by running its cloud baseline at batch 16.
Local inference can still improve the economics of the whole system. The endpoint may already have been purchased for other work, although the electricity used by inference remains a cost. Running there can remove an API call, a network round trip, and a datacenter invocation while keeping private context on the device. These are placement advantages. They do not require a laptop chip to beat a datacenter accelerator on energy per completed query.
This distinction also clarifies the paper’s simulated savings. The authors send every query in the baseline to Qwen3-235B on an H200, then compare that baseline with a hybrid system of four smaller local models and cloud fallback. A simulated router with 80 percent prediction accuracy reduces compute by 61.8 percent, energy by 64.3 percent, and cost by 59.0 percent.
The paper identifies smaller model size as the dominant source of savings. The 61.8 percent result therefore combines two decisions: which model should answer and where that model should run. A demand forecast cannot treat all of the saving as endpoint displacement. Its comparison must first credit any smaller-model routing that the centralized system would have performed anyway.
Add placement to the demand model
The usual demand split distinguishes training from inference. That is no longer enough. Inference needs a placement stage after the workload has been classified.
One useful identity is:
Datacenter MW = base cloud MW + training and post-training MW + gross inference MW × (1 - edge share)
The edge share must be compute-weighted rather than query-weighted. A thousand short editing requests can represent less work than one long agent trajectory. Moving the first group says little about the megawatts required by the second.
The placement term can be decomposed further:
edge share = addressable workload × capable installed devices × quality and latency success × routing adoption × (1 - fallback rate)
Each factor answers a separate forecasting question. Is the task suitable for local execution? Does the user own a device that can run the required model? Does the local answer meet the quality and latency threshold? Will the application route the work locally? How often does the request return to the cloud after a failed attempt?
The paper provides evidence about part of the first and third factors for a selected task set and a few recent devices. It does not measure the installed base, production routing adoption, or fallback behavior. Those terms need their own evidence.
Consider a scenario in which inference accounts for half of incremental datacenter demand. Suppose endpoints eventually displace 20 to 40 percent of that inference compute after routing and fallback. Total incremental datacenter growth falls by 10 to 20 percent:
50% inference share × 20–40% compute displacement = 10–20% total growth adjustment
This is a forecast scenario, not a result from the paper. It produces a material correction, but one far smaller than a 71.3 or 88.7 percent reduction. The scenario leaves baseline cloud demand, frontier training, post-training, and the centralized portion of inference unchanged.
What stays in the datacenter
The placement boundary will move by workload, not by industry slogan. Frontier training remains a tightly coupled supercomputer job. Difficult reasoning, large shared contexts, enterprise retrieval, centralized tools, and workloads that benefit from batching retain strong datacenter advantages. Local models are best positioned to take private, repetitive, latency-sensitive work that fits within the device’s memory and quality limits.
Applications will use both. A local model can handle a session, retrieve private state, or decide that it lacks the capability to proceed. A larger centralized model can take the difficult request, use shared systems, and return a result. Better routing then expands the locally served set without pretending that one model or location should handle everything.
This architecture changes the relation between AI adoption and infrastructure demand. Query volume can grow faster than centralized inference compute when models become smaller, routers improve, and capable devices spread. Centralized inference can still grow in absolute terms if total use grows faster than the placement share. Training can continue its own scaling path at the same time.
The paper does not prove that most inference compute will migrate to endpoints. It supports, but does not establish, a narrower hypothesis: a majority of individual-user query count may eventually become locally addressable, while the centralized remainder can contain a disproportionate share of total compute.
That is enough to change the forecast. AI does not repeal the old movement between centralized and personal computing. It advances the capability frontier, then moves yesterday’s frontier outward as hardware, models, and software catch up. Datacenters will keep creating the frontier. They should no longer be assumed to serve every use of it.
A watch post in the On Scaling Effective Compute series, extending On the Demand Identity.
Sources and further reading
- Jon Saad-Falcon et al., Intelligence per Watt: Measuring Intelligence Efficiency of Local AI, version 5, August 7, 2026.
- Hazy Research, Intelligence per Watt code and profiling harness.
Leave a Reply