Tom's Hardware Builds a CPU Benchmark for Agentic AI Without a GPU

Tom's Hardware has built a working prototype of what it calls the first repeatable CPU benchmark for agentic AI, and the central number it produces is sessions per second. The test strips the GPU out of the loop entirely, then floods a processor with concurrent agent sessions until contention reveals where peak throughput actually sits. No vendor roadmap, no marketing deck — just a concurrency sweep and a stopwatch.

The initiative, detailed by Senior CPU Analyst Jake Roach, addresses a gap that has widened over the past six months. Agentic AI has reshaped CPU vendor messaging, yet no solid, repeatable benchmark exists to evaluate how the latest hardware handles it. The prototype is settled in its central design, but the numbers are not yet ready to withstand scrutiny.

Why concurrency, not raw speed?

The team developed the test on an Nvidia DGX Spark, the GB10 system with 128 GB of unified memory, a Blackwell GPU, and what Roach dryly calls the relatively weak Arm-based Grace CPU. Early profiling with gemma4-26b-a4b showed nearly all agent time spent on the GPU. Even demanding CPU work, such as tool calls from a coding agent, registered as a small fraction of total runtime.

A single agent loop, in other words, barely touches a modern CPU. A document agent scanning PDFs spends almost all its time in pure GPU inference after a brief burst of CPU activity at the start. To create a CPU bottleneck with one agent, the team would have had to engineer a severe processor-to-GPU imbalance. That would tell nobody anything useful.

So the design pivots to concurrency. The test scales from a single worker process up to the CPU's maximum thread count, running three repeats per level. Each worker executes a list of 30 tasks in its own process, with new workers offset by one task — worker one starts at task one, worker two at task two — so uneven task lengths produce deliberately imbalanced contention.

How does the trace-replay system work?

Custom agents built with LangGraph run 30 sequential tasks against a llama.cpp server serving gemma4-26b-a4b. During recording, agents communicate through a proxy server that logs every LLM request and response to a master JSON file. Tool calls are logged separately, and each correct response is hashed and stored.

Replay then removes the live model entirely. Agents run normally, but the proxy hands back recorded responses matched by session label and hash. Any mismatch halts the run and marks it invalid. Critically, the test compares live tool calls to recorded results but always returns the recorded result, even when the call is correct. This prevents drift; genuine tool mismatches are logged and invalidate the level.

The architecture makes expansion straightforward. New personas can be written in Python via LangGraph, other frameworks such as CrewAI can be added, and traces can be re-recorded against different models. Anything speaking an OpenAI-compatible endpoint fits.

What does it measure?

The headline metric is sessions per second — a session being one task completed end to end. Two scores emerge from the sweep:

• Peak throughput: the highest sessions-per-second figure recorded, plus the concurrency level at which it occurred.
• Ideal throughput: the point where the slowest 5% of sessions take no longer than 1.5 times the duration of a single session at base concurrency. Beyond that threshold, task latency lengthens significantly and throughput can degrade depending on processor topology.

The frozen replay measures the CPU's ceiling, assuming all the GPU compute it could theoretically want. A live-inference variant also exists: responses still come from the recording, but the llama.cpp server generates and discards the same token count, capturing prefill, decode, cache behavior, and agents queuing for a GPU slot. Both numbers matter, and neither alone tells the full story.

Which agents run, and where are the gaps?

The benchmark defines six personas, each a LangGraph script with Python-function tools:

• Analyst: queries a fake retail company in DuckDB via SQL, writes a report.
• Coding: fixes bugs in four small projects (two C, two Python) from issue reports, with a reviewer step.
• Documents: matches invoices against purchase orders using PDF extraction and Tesseract OCR, with errors planted in the scans.
• Research: answers questions from a roughly 50,000-article English Wikipedia corpus.
• Support: answers tickets against a knowledge base for a fictional smart home company.
• Web: finds products in a headless Chromium session against a local outdoor-gear store site.

Five personas follow a plan-execute-report structure. The coder is the exception, running a bug-fix-test-review loop.

The limitations are stated plainly. Recordings exist only from two platforms: a 64-core Threadripper test bed with 128 GB of DDR5-6000 RDIMMs, and the DGX Spark. Testing other systems — Intel boxes, Mac Studio clusters, GPU-heavy workstations — may force re-recordings, a model swap, and a full re-gather of results. The hybrid cloud-API path, where tools still run on a weak local machine, remains unexplored, partly because throwing tokens at setup is expensive. Docker sandboxing adds under 2% overhead versus bare metal, but alternative sandboxing methods are unexamined. And all tasks are single-turn; multi-turn flows risk context-window spill and crashes, so they wait.

The next steps are the first real published numbers, framework diversity beyond LangGraph, and the one-click agents. A bulletproof benchmark that survives this market's pace is, in Roach's assessment, a fool's errand. A starting point that scales sessions until a chip buckles is the honest alternative — and as hardware built for agentic workflows arrives, those sessions-per-second curves will decide which CPU stories hold up.