Press Releases

Production LLM serving still depends on hand-written GPU kernels. A large model needs hundreds of kernel shapes, each tuned per GPU, and existing AI-assisted approaches take weeks per model and card with a human in the loop. Emmy generates kernels automatically from a high-level description of the model and tunes them for specific shapes and hardware, covering the same optimizations found in hand-tuned libraries, including TMA transport and automatically derived GEMM and FlashAttention schedules.

"We built Emmy because kernel engineering is the bottleneck of the whole inference stack. When the compiler finds a schedule that beats a hand-tuned library, that win applies to every similar kernel automatically. That is the difference between optimizing one model and optimizing every model we serve," said Dmitry Trifonov, CEO of RiftStack.

Shipped as a drop-in vLLM plugin, Emmy now beats stock vLLM on time to first token at every long-context point measured: 16 percent faster single-stream, 11 percent faster on RAG-style long-input serving, and 10 percent faster under a saturated 64-request load.

The team will demonstrate its inference stack at AI Infra Summit in Santa Clara, September 15 to 17, including a live demo session, "From Rack to API: An Enterprise Engine for High-Performance Inference," presented by CEO Dmitry Trifonov on Wednesday, September 16.

The full benchmark methodology and results are available at https://riftstack.ai/research/optimizing-gemma-4-12b-rtx.

RiftStack builds the operating system for sovereign AI deployments: a control plane that turns datacenters into AI cloud offerings, with built-in LLM inference, GPU virtualization, and the Emmy ML compiler. RiftStack is SOC 2 certified and an NVIDIA Inception member.

Media contact: Heiko Polinski, heiko@cloudrift.ai