Beyond Utilization: Measuring Useful Work on Rented GPU Infrastructure

Organizations increasingly run AI training and inference on GPU capacity rented from neoclouds and hyperscalers, where the marginal cost of an idle or under-utilized accelerator is borne directly by the tenant. The utilization figures exposed by standard tooling — principally the DCGM GPU_UTIL field — are widely misread as a measure of productive work. In fact this counter reports only that a kernel was resident during a sampling interval; a device can register near-full utilization while performing little useful computation, stalled on memory, communication, or scheduling. This paper develops a layered model of GPU efficiency that separates allocation, occupancy, arithmetic activity, and delivered floating-point throughput, and argues that Model FLOPs Utilization (MFU), measured against paid GPU-hours and defined per workload class, is the correct headline metric for a paying tenant. We give the precise MFU derivation and its assumptions, the DCGM DCP profiling signals that explain the utilization–MFU gap, a roofline-based account of memory- versus compute-bound regimes, and a decomposition of the efficiency shortfall into actionable causes. We describe how the Caladrius platform implements this model — computing MFU with explicit measurement-confidence labeling, differentiating training from latency-bound inference, and attributing each increment of lost efficiency to a cause and a remediation. All numeric values herein are illustrative.