Press Releases

Tokenomics has a new name: Inferra by Lightbits™ — the world's first KV cache acceleration engine that dramatically improves GPU utilization, enabling long-context and multi-session AI workloads – makes its public debut at the AI Infra Summit this September. Built by Lightbits Labs®, inventors of the NVMe over TCP protocol, Inferra™ tackles one of AI inference's most pertinent challenges: the memory bottlenecks created by exploding KV caches with long context lengths in LLMs.

Inferra eliminates the “Utilization Gap,” idle GPU cycles or redundant context recomputation, enabling organizations to run multi-session clusters with strict SLA guarantees. By virtualizing GPU memory across memory tiers and transforming the KV cache into an intelligent, persistent data layer, Inferra fundamentally changes the economics of inference.

“We're reinventing inference by shipping the world's first KV cache acceleration engine that eliminates idle GPUs and powers context windows of up to 10M tokens on commodity hardware,” added Ramesh Chettuvetty, Senior Vice President of AI Product and Business at Lightbits Labs. “By delivering Smart Infinite HBM Virtualization across memory tiers, Inferra delivered >300x latency improvement and 16x more concurrent sessions on the same hardware - with zero modifications. It eliminates the stalls that traditionally limit long-context performance, so Neoclouds and enterprises can serve larger models and longer conversations at lower infrastructure cost. Essentially, Inferra pays for itself before you start running it.” 

Key AI inference workload breakthroughs on display:  

• Infinite HBM Virtualization & 16x Session Density: Extends and shares the KV cache beyond limited GPU memory, enabling organizations to run multi-session clusters with strict SLA guarantees. Serve up to 16x more concurrent inference sessions on existing GPU infrastructure - without costly hardware upgrades.
• >300x Latency Improvement & 10M-Token Context Windows: Drastically lowers Time-to-First-Token (TTFT) and Time Per Output Token (TPOT) by proactively and predictively pre-fetching attention states from storage rather than recomputing them, enabling next-gen AI assistants and multi-session agents that require context windows extending well beyond GPU memory limitations.
• AI-Native Security and Isolation: Delivers consistent SLAs with secure tenant isolation and intelligent KV cache management across shared inference infrastructure.

To learn more about the performance and efficiency improvements delivered by Inferra, see Lightbits' Director of AI Architecture, Arthur Rasmusson, present "NVMe Attention is All You Need: How Prefetching on Inferra Powers High-Throughput Inference at a Fraction of the GPU Spend and Power," in the Product Spotlight Theatre on the AI Infra Summit expo floor on September 15th.

AI infrastructure teams, Neoclouds, and GPU cloud providers are invited to visit Booth 219 at the AI Infra Summit for live demonstrations and to learn how Inferra delivers superior token economics and unbounded LLM scale. 

Join Lightbits Labs at the Hyatt Regency on September 15th, from 5:00 - 9:00 p.m., to network with organizations that have effectively plugged the GPU utilization gap and improved profitability with Inferra. Register your interest in attending this event. 

About Lightbits Labs

Lightbits Labs (Lightbits®) is the inventor of the NVMe over TCP storage protocol, which is natively built into its industry-leading block storage, and the first KV cache prefetch engine for AI acceleration. Lightbits data storage solutions are engineered to deliver unmatched high performance and maximum hardware efficiency for LLM inference, real-time analytics, and transactional workloads at scale. Lightbits is backed by enterprise technology leaders [Cisco Investments, Dell Technologies Capital, Intel Capital, Lenovo, and Micron] and is on a mission to deliver best-in-class, cost-efficient storage systems for performance-sensitive workloads at scale.

To learn more about Lightbits Labs, visit https://www.lightbitslabs.com/ and follow Lightbits Labs on Linkedin, X, Facebook, Instagram, and YouTube.

Lightbits, Lightbits Labs, LightOS, and Inferra are registered trademarks of Lightbits Labs, Ltd.