Enabling inference at massive scale with hybrid storage for KV cahce offloading

The AI trial phase is over. As enterprises and cloud builders move pilots into production, inference workloads are scaling faster than the infrastructure designed to support them. Context windows are growing, driven by agentic workflows that involve multiturn exchanges. With each inference request, a Key-Value (KV) cache accumulates the model’s attention values from the tokens computed by the GPU. Efficiently storing KV cache values means the GPU can avoid recalculating prompts, reducing cycles and power consumption.

KV caches are typically stored in high-bandwidth memory (HBM) or dynamic random-access memory (DRAM), both of which are straining as cache sizes grow. Hyperscalers have already seen positive results implementing hybrid storage tiers of SSDs and hard drives for AI applications.1 This multi-tiered system balances capacity, cost, and latency. By following their lead, enterprises and AI infrastructure builders can turn storage into a strategic advantage that enables them to expand the number of users leveraging inference and agentic services and extend the life of their sessions.

SK hynix, a leader in flash and SSD storage, and Seagate, a leader in hard drive storage, partnered together to explore the use of SSD and hard drive storage tiers for KV cache offloading over remote direct memory access (RDMA)-accelerated networking. In the proposed joint solution:

• NVM Express® over Fabrics (NVMe-oF) composability creates a flexible, high-performance, and cost-effective storage pool with GPU direct memory placement performance.
• SSDs serve latency-sensitive operations for disaggregated LLM inference, while hard drives provide mass capacity.
• The Storage Performance Development Kit (SPDK) and Linux Logical Volume Manager (LVM) enable dynamic allocation of NVMe-oF targets from resource pools composed of both types of drives.
• NVIDIA Dynamo KV Block Manager uses composed logical volumes to efficiently utilize GPU resources for multi-turn inference. • Prefill workers build a distributed KV store, avoiding prompt reprocessing by decode workers, yielding faster Time to First Token (TTFT) and lower power consumption.

The paper concludes with performance projections, cost analyses, and drive type mix for infrastructure architects, platform teams, and CTO leadership as they build storage architectures