Skip to content

New research, once a week

The week's new whitepapers in one short email.

You can stop at any time on the Opt-Out page.

New Research Challenges GPU Utilization as Measure of Inference Infrastructure

From Supercomputing 2026: WEKA, the AI data and memory infrastructure company, in collaboration with market intelligence firm NAND Research, today introduced the Inference Economics Scorecard, a new common framework for…

New Research Challenges GPU Utilization as Measure of Inference Infrastructure

WEKA16 November 20267 min read

New NAND Research report underscores that busy GPUs aren't an accurate measure of productive output; introduces new Inference Economics Scorecard to measure inference infrastructure by the useful work it delivers, as applied in an eight-hour benchmark with Nebius on NVIDIA HGX B300 GPUs and WEKA NeuralMesh.

CHICAGO, Nov. 16, 2026 /PRNewswire/ -- From Supercomputing 2026: WEKA, the AI data and memory infrastructure company, in collaboration with market intelligence firm NAND Research, today introduced the Inference Economics Scorecard, a new common framework for measuring inference infrastructure productivity and economic value by the useful work it delivers rather than by how busy it keeps GPUs. The scorecard defines nine measures, including tokens per watt, cost per inference, and context retention. It sets four testing rules, including runs of at least eight hours and counting only successful output. Already applied in a benchmark with Nebius, the AI cloud company, it comes as inference is claiming a growing share of AI infrastructure spending and buyers hit power, memory, and rack-space limits.

Buyers Are Already Treating Inference Differently, New Research Finds
New research from 451 Research by S&P Global shows organizations are already adapting, projecting that inference will grow from 34% of AI infrastructure spending in 2026 to 46% by 2030 and finding that 41% of organizations running inference buy fundamentally different infrastructure for it than for the rest of their IT stack.

"Inference is where the value of AI is ultimately realized, and our research shows it's reshaping how organizations buy infrastructure," said Greg Macatee, Senior Analyst at 451 Research by S&P Global. "Agents are driving exponential growth in token volume; memory, power, and supply constraints greatly challenge the notion that organizations can keep up simply by adding more GPUs."

Why GPU Utilization Falls Short for Inference

For years, training was the dominant AI workload. Because training keeps GPUs busy with large, predictable jobs, GPU utilization became the default measure of AI infrastructure productivity. Inference, especially agentic inference, places very different demands on GPUs. An agent does not simply answer a prompt; it completes a task over many steps, and at each step it re-reads the context it built earlier. After the GPU evicts that context from memory to make room for other users and sessions, it must recompute it when the task returns, and the task makes no further progress until that work is done. The result is a GPU that is fully utilized but unproductive because it wastes cycles recomputing the same tokens multiple times.

The Inference Economics Scorecard measures inference systems at production scale: completed requests across many concurrent users, sustained over hours rather than minutes.

"The industry has been grading inference with a training-era report card. GPU utilization tells you the chip was busy. It cannot tell you whether the work was finished, repeated, or thrown away, and with agents, a lot of it is repeated," said Liran Zvibel, co-founder and CEO of WEKA. "Power, supply, and rack space are constraints now as much as capital. The new math required for inference infrastructure is useful work per watt, per rack, per dollar, at the response times your users expect. The Inference Economics Scorecard puts that math on paper, and WEKA will be measured by it like everyone else."

From Busy to Productive: How the Scorecard Works
The Inference Economics Scorecard defines nine measures of inference performance and economics: tokens per watt, tokens per rack, cost per inference, GPU goodput, time to first token and end-to-end latency, memory efficiency, operational density, agent concurrency, and context retention. It also sets the discipline for testing against them:

  • The same workload and service targets across every system tested
  • Measurement of the whole serving stack
  • Runs of at least eight hours
  • Only successful output counted

"The most expensive mistake in AI infrastructure right now is buying for peak throughput and paying for repeated work," said Steve McDowell, principal analyst and founder at NAND Research. "Agentic workloads make that mistake easy to miss, because every step of a long task re-reads context that a busy GPU may have to rebuild from scratch. We designed the scorecard so buyers can hold any system to the same standard: finished work, measured over hours, with every failure counted. The Nebius benchmark shows what that discipline reveals, because it ran for the full eight hours and counted every request that failed or timed out."

Putting It to the Test: Real Agent Load, Sustained for Eight Hours
WEKA and Nebius recently applied the scorecard's rules in a joint benchmark on NVIDIA HGX B300 systems and reported results on several measures. The test ran on one eight-GPU node serving DeepSeek-V4-Pro to 512 active sessions drawn from a corpus of more than 5,000 real agentic coding sessions over eight hours per configuration, counting every failed and timed-out request.

Using WEKA NeuralMesh with Augmented Memory Grid, which pools NVMe across the cluster as a shared context tier, the node produced 3.7x the output tokens per GPU-hour compared with HBM-only serving on the same GPUs, and 97% of successfully completed requests reached a first token within five seconds, up from 44%. On HBM-only serving, 60% of prefill work was recomputed from context the cluster had already processed earlier in the session; with Augmented Memory Grid, 6.8%. At the measured rate, the HBM-only node would need about 30 hours to produce what Augmented Memory Grid produced in eight. The full benchmark is published at weka.io and nebius.com blogs.

"Peak throughput shows what a system can do for a few minutes, but our customers run agents for hours, days, or longer, so what matters is sustained useful work," said Narek Tatevosyan, Vice President of Product at Nebius. "Over a full shift, the HBM-only configuration steadily lost ground as context fell out of memory and timeouts climbed, while WEKA NeuralMesh kept first-token latency flat the entire run. That changes how we plan capacity: we size for finished work over a working day, not a benchmark peak, and the Inference Economics Scorecard gives us a common way to show customers that number."

"Agentic inference is the workload that defines the AI factory, and it puts extraordinary demand on memory and context. The question at scale is no longer how much context fits on the GPU, but how far it can be extended across the cluster without being recomputed," said Jason Hardy, VP, Storage Technology at NVIDIA. "The WEKA and Nebius results on NVIDIA HGX B300 systems show what a shared context tier can do for sustained throughput and responsiveness, and as inference grows, buyers will need a clearer way to compare systems on the work they actually finish."

Reproducible by Design: Open Traces and Tooling
The traces and tooling behind the benchmark are open-sourced so anyone can run the same workload on their own systems and compare results on equal terms. Because the traces come from real agentic coding sessions rather than synthetic load, and include the varied timing and complexity of real work, they give any team a realistic stand-in for production. Others are already building on them: SemiAnalysis's AgentX benchmark is built on the traces, and NVIDIA AIPerf benchmarking tool ships a loader for them.

"A benchmark for agentic inference must reflect real-world production traffic, including long-running agents, growing context, complex multi-agent workflows with subagents, and the bursty, unpredictable traffic patterns that characterize real-world deployments," said Dylan Patel, founder and CEO at SemiAnalysis. "Too few benchmarks meet that bar, which is the gap we built AgentX to close, and it's why AgentX runs on real-world agentic traces using WEKA's open kvlab trace format, so anyone can replay the same multi-turn workflows and compare results on equal terms. As agents become the dominant inference workload, reproducible measurement like this is how the industry gets past peak numbers and sees what its infrastructure can sustain."

SemiAnalysis plans to share AgentX results on WEKA NeuralMesh as testing progresses. For more information, visit https://inferencex.semianalysis.com/agentx.

Additional Resources

  • The Inference Economics Scorecard and NAND Research companion report are available HERE.
  • Download a complimentary copy of the 451 Research by S&P Global report, "From inference to agentic AI" HERE.
  • Read the blog post from WEKA CEO Liran Zvibel. 
  • To learn about the engineering behind the capture tooling and how the measurements were defined, visit HERE.
  • Visit WEKA at SC26 in Chicago, November 16 to 20, booth #2519 to learn more about how to apply the Inference Economics Scorecard to your own inference workloads.

About WEKA
WEKA is the AI data and memory infrastructure company transforming the economics of agentic AI. The WEKA NeuralMesh platform unifies high-performance data storage with extended GPU memory, giving enterprises, AI cloud providers, and AI builders a single foundation for training, inference, and agentic workloads. With Augmented Memory Grid, NeuralMesh extends GPU memory capacity by 1000x, accelerates time to first token by up to 20x, and delivers 10x more concurrent users from the same GPU footprint, as proven in production benchmarks. WEKA enables organizations to scale production AI faster, get more useful work from every GPU, and reduce the cost of every token served. Learn more at www.weka.io or connect with us on LinkedIn and X.

WEKA and the WEKA logo are registered trademarks, and WEKA NeuralMesh and Augmented Memory Grid are trademarks of WekaIO, Inc. Other trademarks mentioned herein are the property of their respective owners.

WEKA: The Foundation for Enterprise AI

Cision View original content to download multimedia:https://www.prnewswire.co.uk/news-releases/new-research-challenges-gpu-utilization-as-measure-of-inference-infrastructure-302939620.html

Found this useful? Share it

New research, once a week

The week's new whitepapers in one short email.