Jason A. Hoffman, PhD
Someone tells you a number: a hundred petaFLOP/s, a million tokens, ninety percent utilized. The first question is never whether the number is right. It is which quantity the number describes. Most disagreements about computing infrastructure are people quoting different quantities at each other, and the confusion survives because the units all sound alike.
The accounting runs through stages, and the unit alone does not tell you which stage you are looking at. First are resources supplied or consumed: peak FLOP/s, allocated accelerator-hours, and joules drawn. Next are gross workload outputs: generated tokens, returned images, and completed simulation steps. Last is goodput, the subset of that output that passes declared quality and deadline gates, reported per unit time. A rejected or late result remains observable output. It simply does not count toward goodput.
The difference is concrete. Suppose a service emits one hundred answers per second. Ninety pass the quality gate, and eighty also arrive before the deadline. Gross throughput is one hundred answers per second. Goodput is eighty answers per second. Both numbers are honest; they answer different questions.
Quoting one stage as if it were another is the standard error. “The chip delivers four petaFLOP/s.” At what number format and through which execution path? On an H100, the dense FP8 tensor peak is nearly thirty times its FP32 CUDA-core peak. Both numbers describe the same silicon, and neither is complete without those declarations. “We used 10²⁵ FLOPs to train it.” That is the resource consumed, not the capability produced; two teams can spend the same FLOPs and get different accepted results, and the spending is the easier number to report. “The cluster is ninety percent utilized.” Which stage? Allocated, executing kernels, sustaining model FLOPs, and producing accepted output are different measurements that travel under the same name.
Even gross observables are currencies without an exchange rate. A token count means something only inside a declared model, task, quality bar, and latency budget, and it is not fungible across unlike workloads. A million tokens of casual chat and a million tokens of verified legal analysis are the same count of different goods. Counts can be billable; they are not comparable until the declarations match.
The habit that cuts through all of it is one question, asked every time: which quantity is this, and what gates has it passed? A spec sheet may quote a resource measure. A press release may quote a gross observable. A customer ultimately experiences goodput. I wrote about the same disease in GPU failure rates: several real numbers collapsed into one phrase because each belonged to a different stage. Compute has it at industrial scale, and the cure is the same everywhere. Name the quantity.
A companion to On Scaling Effective Compute, one of a series of short posts breaking a long map into teachable pieces.
Leave a Reply