The Most Expensive Place to Do Your Thinking Is at Query Time

Token prices fell a thousandfold. Enterprise AI bills went up anyway. The arithmetic points to a fix that has nothing to do with buying more chips.

Three years ago, running a frontier-class language model cost roughly a thousand times what it costs today. It is one of the steepest cost collapses in the history of computing. By any normal logic, the bills should have followed.

They did not. OpenAI's spending on inference, the business of running models rather than building them, rose from $8.4 billion in 2025 toward a projected $14.1 billion this year. The average enterprise artificial-intelligence budget grew from $1.2 million in 2024 to about $7 million in 2026. Industry analysts estimate that between 55% and 80% of corporate spending on graphics processors now goes to running models, not training them.

The hardware doing that work sits idle most of the time. Typical enterprise deployments report utilization rates between 15% and 30%. The machines are not thinking. They are waiting: for memory, for the next request, for a user to finish typing. At the small batch sizes that interactive applications require, the binding constraint is memory bandwidth rather than raw computation. Companies are renting supercomputers to answer one question at a time.

Put those facts together and the picture is uncomfortable. The industry has become dramatically more efficient at a unit of work it performs far too often, in the least efficient way available, at the worst possible moment.

Nearly every corporate AI system built in the past three years shares one shape. A question arrives. The system searches a store of documents, retrieves passages of text, packs them into a prompt and asks a language model to produce an answer. The technique is called retrieval-augmented generation, and it is the default architecture of the era.

Consider what that design decides. It defers the expensive work, the searching and the reasoning, until the instant a person is waiting for a reply. Every question pays the full cost again. Ask the same thing twice and you pay twice, on hardware built for throughput while you use it for latency.

It also degrades as it succeeds. Add more documents and the search returns more candidates of lower average relevance. Context windows fill. Systems truncate. Truncation is among the most reliable ways to make a model invent an answer. The archive grows, responses slow, and accuracy quietly deteriorates. Anyone who has moved a working demonstration onto a real corpus has watched it happen.

Computing has an old answer for this situation. Do the expensive work once, in advance, and leave something cheap behind.

Interpreted programming languages analyze a program every time it runs. Compiled languages perform that analysis once and produce an artifact that executes directly. For production systems, nobody considers this a close call.

The alternative, and the reason we call our approach compile-time inference, is to move the thinking to the moment the data arrives. When a document, a broadcast frame or a transaction enters the system, that is when the facts get extracted, scored for quality, stripped of duplicates and checked against one another. The result is compiled into a live, cited structure. When a question finally arrives, the system selects from that structure instead of reasoning across the archive from scratch.

Work done at ingest is batchable. You control the queue, so it can run at high utilization rather than 15%. It is schedulable, able to wait for cheap capacity, off-peak power, spot instances or a quiet night. It is paid for once: a fact extracted today serves every question asked about it for the next decade. And it is not latency-critical. Nobody drums their fingers while a frame is indexed, which means the work can run on hardware that would be unacceptable in a request path, ordinary processors included.

Query time becomes a lookup against a compact compiled structure. Lookups do not slow down because the archive grew. That is the point of an index, and it is a property no amount of chip procurement will buy if the architecture insists on reasoning afresh for every request.

The bill stops scaling with questions asked and starts scaling with data ingested, a number that can be forecast, capped and scheduled.

We are not the first to argue that the model sits in the wrong place. Palantir has made a version of this case for years, and with more commercial success than anyone: put the ontology at the center and the model at the edge. The structured, governed representation of the business becomes the system of record. The language model is an interface to it, and a replaceable one. We think that is correct, and we have studied it closely.

Our disagreement is narrow, and worth stating plainly because it is where we are placing our bet.

In Palantir's Foundry, the ontology is largely modeled by hand. Human experts, usually supported by a substantial services engagement, design the objects and relationships that mirror the business. The result has real quality. It also carries the characteristics customers describe: seven-figure annual costs, long deployments, and a semantic layer whose shape is owned by a vendor. That last point is precisely the sovereignty concern enterprises now raise.

Our bet is that the ontology does not have to be built by hand. It can be compiled from the customer's own data by machine, scored, deduplicated and bound to the source artifact each fact came from, with the human role shifting from modeling to governing. If the bet is right, the same architectural benefit arrives without the services bill, and the semantic layer stays inside the customer's estate.

We may be wrong. Hand-modeling captures judgment that is genuinely hard to derive from data, and anyone who says otherwise has not tried it. But it is the assumption in the incumbent design most worth testing, and it can be tested, which is more than can be said for most claims in this field.

There is a second consequence, and for regulated buyers it is the one that closes deals.

When an answer is selected from a compiled structure rather than sampled from a model, identical facts produce identical outputs. Ask the same question twice and get the same answer twice, not because the temperature setting is zero but because the decision was never a sampling problem in the first place. The output changes when the data changes, and at no other time.

That turns reproducibility from an aspiration into a property of the system. Every claim can carry its receipt: the source, the timestamp, the artifact a reader can open and check. When the record does not cover the question, the system can say so and stop. A sampled system cannot structurally make that offer, because it will always produce something.

Engineers will recognize the difference as the one between a cache and a guess.

If your inference bill is climbing while token prices fall, cheaper tokens will not rescue you. Volume is the problem, and volume is a consequence of when you chose to do the work.

The question worth putting to any AI system in production is not which model it runs or how many chips it has. It is simpler and less comfortable.

How much of what it computes today did it already compute yesterday?

For most systems now in production, the honest answer is nearly all of it.

Verificate builds compile-time inference, a groundedness layer for AI systems that have to be right. Deterministic. Cited. Auditable.