A workflow-level benchmark for agent infrastructure

Token throughput measures one part of an agent workload. It does not, by itself, locate time spent in orchestration, host execution, tool calls, queueing, or transitions between stages.

A recent study characterizes agent execution as fragmented across inference, tool calls, and orchestration. Its authors report repeated CPU–GPU boundary crossings, host CPU work on the critical path, bursty load, uneven GPU use, and capacity stranded by conventional uniform servers. Those are reported study findings, not evidence that every agent system has the same bottleneck. The study also describes an Agora prototype using role-aware CPU pools, affinity-aware scheduling, and GPU-memory oversubscription with state prefetching. Read the study.

AMD separately describes an execution loop in which models select actions while CPUs handle tools, processes, file operations, compilation, and worker coordination. It argues that this host-side work affects total task-completion time even when inference runs on a GPU or NPU. Its accompanying benchmark is vendor-produced and hardware-specific, so it is evidence of AMD’s documented test rather than independent validation of a general performance advantage. Read AMD’s account.

Together, these accounts motivate an operator question: where does the complete workflow wait? The worksheet below is our proposed method for answering it. It is an engineering prescription derived from the operating tensions in the two sources, not a benchmark demonstrated by either source.

Start and stop at task completion

Define the benchmark around a complete, repeatable task. Start the clock when the system accepts the task and stop it when the final result or terminal failure is available. Record total elapsed time and the outcome.

Then divide that interval into stages meaningful to the tested system:

• admission and queueing
• model inference
• orchestration and state handling
• tool preparation and execution
• data or state movement
• result assembly
• retries, backoff, and recovery

The stage names are analytical choices. Their value comes from applying the same definitions across runs and configurations.

Capture the boundaries, not only the stages

For every transition, record the source stage, destination stage, timestamp, transferred state size when available, and time until useful work resumes. Count crossings separately from their duration. A large number of individually small transitions may merit different investigation from one long transfer.

Tool activity deserves its own event record. Capture invocation time, completion time, execution location, queue delay, retry count, and terminal status. This makes it possible to distinguish a slow tool from delayed dispatch or orchestration around that tool.

Put occupancy on the same timeline

Sample CPU occupancy, GPU occupancy, memory pressure, and runnable or queued work against a shared clock. Preserve per-stage attribution where the platform exposes it.

The purpose is not to maximize every utilization number. It is to identify overlap and waiting. For example, GPU idle time overlapping host execution is a candidate CPU or orchestration constraint, while host idle time overlapping an inference queue is a candidate GPU-scheduling constraint. Those interpretations are hypotheses to test against traces and controlled configuration changes; occupancy alone does not establish causation.

A compact benchmark worksheet

Field	Record	Diagnostic question
Task outcome	success, terminal failure, or partial result	Are faster runs completing the same work?
End-to-end time	admission through terminal outcome	What does the user-facing workflow experience?
Stage time	elapsed and active time per stage	Where does time accumulate?
CPU occupancy	per stage and over the full run	Does host work overlap the critical path?
GPU occupancy	per stage and over the full run	Is inference capacity active, queued, or idle?
Boundary crossings	count, duration, and state moved	Are transitions fragmenting execution?
Tool events	queue, execution, retry, and failure time	Is the tool slow, or is dispatch slow?
Queue depth	timestamped by resource or pool	Where does burst demand wait?
Idle-capacity overlap	idle resource beside active or queued work	Is capacity present but poorly matched to the stage?
Configuration	hardware, pools, affinity, concurrency, and scheduler settings	Can another run reproduce the result?

Compare interventions one at a time

Once a candidate constraint appears, change one relevant configuration while holding the task set and measurement definitions stable. Examples include CPU allocation, worker concurrency, affinity, GPU scheduling, or tool placement. Compare the full task-time distribution, stage timings, outcomes, and resource timelines—not just a single throughput figure.

This process cannot prove that a configuration will behave identically under every production workload. It can show whether a proposed change moves the measured constraint within the tested task set.

The practical buying question is therefore broader than tokens per second: which configuration completes the representative workflow, with its tools and orchestration intact, within the target operating envelope? Teams evaluating an agent system can use the same worksheet when scoping a build, then preserve the task definitions and traces as the baseline for later infrastructure changes.