Why the CPU Is the Bottleneck in the Agentic AI Era

The era of “just buy more GPUs” is ending.

Through 2024, the formula for AI infrastructure was simple. Secure GPUs. As many as possible. Every shortage conversation centered on GPUs and HBM. Then in March 2026, AMD CEO Lisa Su said agentic workloads are pushing work back onto traditional CPU tasks, and Intel’s CFO admitted server CPU supply is “absolutely constrained.” The bottleneck in AI infrastructure has started to move.

Why CPUs, and why now? This piece focuses on that one question.

1. Agents don’t generate answers. They do work.

The AI of the chatbot era worked like a calculator. Put a question in, get an answer out. The AI of the agent era is closer to a team of assistants. It searches the web, reads documents, executes code, calls external tools, and assembles intermediate results into a final output.

Even in traditional chatbot inference, CPUs handle tokenization, scheduling, and I/O. The bottleneck was almost always on the GPU side. A user sends a question, the CPU converts text to tokens, the GPU runs the model and generates a response, the CPU converts it back to text. A simple round trip. GPU compute time dominated total latency, so CPU performance barely registered as a concern.

Agentic AI changes that structure. Instead of ending at a single response, the workload expands into a multi-step workflow involving tool calls, external system lookups, code execution, and intermediate result composition. When a user asks the AI to analyze Q3 earnings and build an investment strategy, the agent searches the web for earnings data, parses PDFs, queries databases, runs numerical analysis in Python, and synthesizes all of it.

In this workflow, the GPU handles “thinking,” which is token generation. The coordination burden for search, API calls, code execution, file I/O, and result orchestration sits largely on the CPU and host system side. In certain agentic workloads, CPU-side tool processing and orchestration drive more of the total latency than the GPU does.

A November 2025 paper from Georgia Tech and Intel, “A CPU-Centric Perspective on Agentic AI,” put numbers on this. After profiling five representative agentic workloads, they found CPU time spent on tool processing accounted for 50% to as much as 90% of total latency in some cases. In specific workloads, the GPU was ready for the next batch while the CPU was still finishing a tool call.

The summary: in agentic workloads, the bottleneck increasingly lives outside the GPU, particularly in the CPU and memory hierarchy.

2. The GPU computes tokens. The CPU makes those tokens do something.

The GPU’s role is clear: it handles the matrix multiplications in Transformer models. It runs Q, K, and V matrix construction, attention score computation, and output generation across thousands of cores in parallel. If the CPU excels at complex branching logic, the GPU is built to apply identical operations across massive datasets simultaneously. In LLM inference, the GPU owns two phases: prefill (processing all input tokens at once, which is compute-bound) and decode (generating tokens one at a time while reading the previous cache, which is memory bandwidth-bound).

The CPU’s role in the agent era is to take what the GPU generated and turn it into actual actions.

Orchestration means managing the execution plan: which tool to call, when to call it, and how the output of one step feeds into the next. This is sequential, branch-heavy logic, the kind CPUs handle better than GPUs.

Tool execution covers Python code, web scraping, PDF parsing, and database queries. These are inherently sequential and I/O-bound tasks.

API gateway work involves external service calls, response parsing, error handling, and retries. A single agent task can generate dozens of external calls.

Memory management covers context sharing between agents, KV cache offload coordination, and storage and retrieval of prior step results.

In the chatbot era, the CPU assisted the GPU. In the agent era, the CPU handles a substantial portion of workflow operations. While the GPU is “thinking,” the CPU is doing much of the “doing.” The assumption that a faster GPU is sufficient no longer holds.

Why it’s getting worse now: the VRAM wall and long context

There is a compounding factor accelerating this shift: the rapid expansion of context windows.

Through 2024, most mainstream models supported 128K or 200K tokens, but the long-context race was already moving toward 1M-scale. Into 2025, Gemini 2.5 Pro and Flash, GPT-4.1, and Llama 4 Maverick all began supporting one million tokens. Claude Sonnet 4 upgraded from 200K to one million tokens as well.

The problem is that as context grows, KV cache grows faster than model weights. The KV cache stores Key and Value vectors from prior tokens so the model does not need to recompute them. It scales linearly with token count.

For reference, the per-token memory footprint is: 2 (K and V) multiplied by layer count, KV head count, head dimension, and precision in bytes.

For a Qwen2.5-14B model with GQA applied, that is roughly 0.2MB per token. At 200K tokens, the KV cache is around 40GB, which is already tight on an H100 with 80GB VRAM once the model weights are loaded. At 1 million tokens, it is around 200GB, which is physically impossible on a single GPU.

At million-token scale, GPU VRAM alone cannot hold the workload. The industry is converging on three approaches.

KV cache CPU offload pushes “cold” tokens with lower access frequency from GPU VRAM to system RAM and pulls them back only when needed. InfiniteHiP research demonstrated handling 3 million tokens on a single 48GB GPU in a research setting. The practical constraint is the roughly 150x bandwidth gap between HBM bandwidth (around 4.8TB/s) and PCIe bandwidth (around 32GB/s). Real-world serving deployments require further validation.

Prefill-decode disaggregation physically separates input processing and token generation into different clusters. Transferring the KV cache over the network increases the CPU’s I/O management responsibilities substantially.

Running lightweight inference directly on the CPU is becoming viable. 3B to 7B parameter small language models are showing competitive results on certain tool-use benchmarks relative to their parameter count, making a split of “heavy reasoning on GPU, lightweight judgment on CPU” increasingly practical.

All three paths point the same direction: away from VRAM-only configurations and toward a tiered memory architecture combining VRAM, system RAM, and NVMe. CPU memory capacity, memory bandwidth, and CPU-to-GPU interconnect have suddenly entered the critical path for system performance.

3. A good CPU for the agent era is not one with more cores. It is one with better memory and connectivity.

For the past two decades, server CPU design aimed at general-purpose coverage. A single CPU needed to handle databases, web servers, virtualization, and HPC. AMD EPYC connected multiple chiplets through Infinity Fabric; Intel Xeon scaled core counts through multi-tile designs. Chiplets make core scaling easier, but inter-die communication introduces NUMA latency.

Agentic workloads ask something different from CPUs.

Per-core performance matters first. Orchestration is a sequence of sequential decisions. A slow individual core stalls the GPU waiting for the next batch. Single-core speed comes before core count.

Memory bandwidth and capacity matter more than before. KV cache offload, RAG retrieval, and context management all involve heavy memory I/O. Conventional DDR5 with 8 channels delivers roughly 300GB/s. NVIDIA’s Vera achieves 1.2TB/s using LPDDR5X. Memory capacity on Vera reaches up to 1.5TB, three times the previous generation.

Single-die design carries weight here. Vera puts all 88 cores on a single die. For workloads like agent orchestration where inter-core coordination is frequent, eliminating chiplet-to-chiplet latency is meaningful.

CPU-to-GPU interconnect is now a differentiator. NVIDIA connects its CPU and GPU through NVLink C2C, providing higher bandwidth than PCIe and cache coherency. The CPU can directly reference GPU memory, which simplifies KV cache management considerably.

x86 vs. ARM: a shifting competitive axis

This transition is reshaping the ISA competition as well.

The x86 camp, AMD EPYC and Intel Xeon, holds the software ecosystem advantage. Most of the tools agents call, Python, Bash, database engines, and more, have been optimized for x86 over decades. That is the basis for Lisa Su’s comment that x86 has an edge in agentic workloads.

The ARM camp, NVIDIA Vera/Grace, AWS Graviton, and Ampere, argues for power efficiency. As gigawatt-scale data centers make power a physical constraint, performance per watt is gaining importance in the purchase decision.

The reason NVIDIA is building CPUs while running the GPU market is that maximizing GPU utilization requires solving the CPU-side bottleneck directly. Offering a CPU tightly integrated with its own GPU through NVLink C2C enables optimizations that are simply not achievable with another vendor’s processor in the host socket.

Whether x86 ecosystem inertia proves stronger, whether ARM’s power efficiency drives a real transition, and how much NVLink C2C integration actually moves agent throughput numbers in production are all still accumulating data. What is clear is that CPUs, once treated as commodity components, are becoming differentiated products where the choice of CPU measurably changes system performance.

4. The market is already pricing in this shift.

This is not theory. Demand signals are showing up in real markets. Three of the clearest ones.

AMD reported Q4 2025 data center revenue of $5.4 billion, up 39% year over year. Lisa Su directly cited surging demand for EPYC CPUs. Fifth-generation EPYC Turin accounted for more than half of server CPU revenue, and cloud instance deployments running EPYC grew more than 50% year over year. According to Mercury Research, AMD’s server CPU revenue share crossed 40% for the first time. Su explained the dynamic in terms of agents pushing work back onto traditional CPU tasks.

Intel’s CFO Dave Zinsner acknowledged that supply is “absolutely constrained.” The company said it is intentionally pulling back from low-margin client markets to concentrate manufacturing capacity on data center products. Lead times for Chinese customers stretched to up to six months, and server chip prices rose more than 10%. Intel admitted internally that it did not anticipate this acceleration in CPU demand, which is telling given how quickly it developed.

The AWS-OpenAI partnership announced in November 2025 was reported as a $38 billion deal. The public announcement specifically included language about accessing hundreds of thousands of GPUs while scaling to tens of millions of CPUs to rapidly expand agentic workloads. Most observers focused on the GPU number. The fact that the CPU figure made it into the official press release signals that frontier AI labs are taking CPU scale seriously as a planning variable.

Beyond those three: IT distributors reported server CPU ASPs rising roughly 30% in Q4 2025, AMD lead times expanded from eight to ten weeks, and TSMC’s 3nm lines are seeing GPU wafer demand crowd out CPU wafer allocations as a secondary effect.

CPU shortage is no longer a prediction. It is happening.

5. What each company is doing

NVIDIA began selling the Vera CPU as a standalone product. CoreWeave was the first customer. The Meta deal included what was described as the first large-scale Grace-only deployment, meaning CPUs deployed at scale without GPU pairings. This is vertical integration aimed at maximizing GPU utilization by internalizing the CPU layer and maximizing the integration advantages of NVLink C2C. If CPU TAM expands in the agent era, NVIDIA intends to capture it directly.

AMD‘s EPYC Turin pushed server CPU revenue share past 40%, and the company is now projected to overtake Intel in 2026. The x86 ecosystem’s inertia is its primary competitive asset. In February 2026, AMD announced a potential deal with Meta worth more than $100 billion, supplying MI450 GPUs and Venice EPYC CPUs while issuing up to 160 million shares, roughly 10% of the company, as performance-based warrants. The structure is aggressive. The gap relative to NVIDIA is the absence of a coherent CPU-to-GPU interconnect at the level of NVLink C2C.

Intel is in reset mode under Lip-Bu Tan. The supply constraint is, paradoxically, creating a short-term profitability improvement. Xeon 6 Granite Rapids and the 18A roadmap are the real tests of whether the company can execute. Intel is partnering with SambaNova on Xeon plus SN50 configurations to pitch GPU-free agentic inference as an alternative, and working with Cisco on edge AI positioning through Unified Edge. Execution is the open question.

6. Where this leaves us

The data center of the chatbot era was straightforward: hundreds of GPUs, a handful of CPUs, and the CPU functioning purely as a host.

The agent era changes the ratio. In some inference and agent systems, the CPU-to-GPU allocation is shifting from what was standard. The reasons Jensen Huang is selling Vera as a standalone product, the reason the Meta deal included a Grace-only deployment, and the reason the AWS-OpenAI announcement specified tens of millions of CPUs all point the same direction.

The GPU shortage years taught one lesson that applies here. When the bottleneck moves, so does where the value accrues.

In 2023, the bottleneck was the GPU, and NVIDIA captured most of the upside.

In 2025, the bottleneck was HBM, and SK Hynix captured most of the upside.

In 2026, the pressure is spreading across CPUs, system memory, networking, and power simultaneously.

TSMC sustains capacity premiums from simultaneous GPU and CPU demand. Memory companies sit in a dual-benefit structure from both HBM and LPDDR5X/DDR5 demand. Networking, Broadcom and Marvell in particular, sees growing demand as data movement becomes the new constraint.

The bottleneck is moving from compute to data movement, from VRAM to system memory, from a single GPU axis to the entire system stack. Reading that shift correctly is what positions someone to catch the next cycle’s opportunity.