Public benchmarks can't tell you which model to use. Your own traffic can.

TL;DR: Our app has been using the same family of models since we launched. We tested our current models, Gemini 3.6 Flash and Gemini 3.1 Pro, using production traffic against a newer cheaper model. We migrated 14 out of 16 jobs to DeepSeek V4 Flash 0731, cutting costs by ~91% in those jobs. Two jobs failed and remained with the old model. Here is exactly how we evaluated it, the bugs we hit, why most teams are overpaying, and why we're open-sourcing it.

Most teams pick their models once, while they're building the thing, and then never look again. At Ground Truth Intelligence, we were no different. So we built something to answer one question on demand: for each job our product does, is the model running it still the right model? Not according to a leaderboard, but according to our own production traffic.

The answer came back "no" on fourteen of the sixteen jobs we tested. All fourteen have since moved to DeepSeek V4 Flash, an open-weights model, at an 88%–92% cost reduction. On two jobs it lost, and those are still running on Gemini 3.6 Flash. However, surprisingly on Gemini's most expensive model (3.1 Pro), the open-weights model won on 35 out of 40 comparisons, while across the jobs running on 3.6 Flash the models tied.

Since just hearing "DeepSeek" makes people nervous, one clarification first: it's an open-weights model. Nothing goes to DeepSeek (the company). We run the weights on TensorX, who hosts inference in Europe.

The cost reduction is from real data, not calculations on list prices. We ran both models over the same real inputs and compared what each call actually cost. That distinction matters a lot more than it appears, because models differ in how much they think and how long their answers run, so token consumption will vary. Calculating by token price will be off by a lot. The per-token prices are public and I've put them further down, so you can work out what the same ratios would do to your own bill.

Here's a note on terms, since I'll use them throughout the post; our product runs a few dozen distinct AI jobs. Each one has its own prompt, its own model, and a specific task, for example sorting an incoming message into a category, drafting an email, summarizing what we already know about a customer account, deciding which action to take next, etc. Internally we call these lanes. We built an evaluation harness that replays real production requests through a candidate model and grades what comes back, and we selected the sixteen lanes that run a single request and a single response (more on why that matters at the end). They're independent, so they can be moved one at a time.

I'm open sourcing the harness. There's no repo to point at yet, because right now it's welded to our own database and our own trace format, and pulling it out is a few weeks of work. Everything it does is written up below, in enough detail to rebuild it yourself if you'd rather not wait.

Why Almost Nobody Re-Checks Their Model

You pick a model when you're building the thing. It's the best option available that week, you wire it in, you move on. Six months later something cheaper and roughly as good has shipped, and nobody goes back to look.

The reason isn't laziness. It's that checking is expensive and staying put is free. To know whether a cheaper model is safe for your product, you need to continuously test it on your real prompts, and that means building an evaluation setup: collecting real examples, deciding what "as good" means for each job, and grading hundreds of outputs. That's a week or two of engineering per workload, and it produces no features. Meanwhile, the cost of not checking is invisible. It shows up as a slightly larger bill that nobody notices, because nobody knows what the bill should be.

Public benchmarks don't help. They tell you how a model does on somebody's exam questions. They tell you nothing about how it handles your prompt, with your formatting rules, your domain vocabulary, and the specific ways your inputs are messy. And testing in production means your users find the regression before you do.

So the sensible move, for any one team in any one month, is to leave it alone. That's what we did too, until checking got cheap enough.

What We Actually Did

The whole thing rests on being able to re-run a past production request exactly as it originally ran. Every time our product calls a model, we store the call: the prompt, the settings (temperature, output limit, what response format was demanded, which tools were offered) and the response that came back. That stored settings block is the part people usually skip, and it's the part that makes the test honest. If you replay a prompt at a different temperature or without the response-format constraint, you're not testing the model, you're testing a different configuration and drawing conclusions about the model.

That gives us a free comparison. The original response already exists and was already paid for. We don't need to run the expensive model again to have something to compare against.

So: take a few hundred real production requests, replay each one through DeepSeek with the exact recorded settings, and put the two responses side by side.

Before paying anyone to grade those pairs, we run cheap automated checks that need no model at all:

• Does the output parse as JSON, when JSON was required?
• Does it match the shape the calling code expects, field for field?
• Is it in the right language? (Our product works in Spanish and English, and a model drifting from one to the other is a silent, ugly failure.)
• Does it use our defined vocabulary, or did it invent category names?

These cost nothing and they catch bad candidates immediately. On our first clean run, 44 of 45 replays cleared this stage, or 97.8%. The one failure was a language drift.

Every pair goes to a judge, which is a third model asked to compare the two outputs and say which better satisfies the job. Three rules there, and all three matter:

1. The judge comes from a different company than either model being compared. We used Anthropic's Claude Sonnet 5 to compare Google's output against DeepSeek's. Models have measurable preferences for their own output and their siblings', so if you let one grade itself you have no idea what you measured.
2. The judge doesn't know which output is which. It sees "Output A" and "Output B" with no labels, and we flip a coin per comparison to decide which side the candidate goes on. With a fixed position, any preference the judge has for the first or second slot would show up as a systematic result and you'd never see it.
3. The judge grades against the actual job rather than a generic idea of quality. We pull the original system prompt out of the stored request and hand it over as the standard, so the question is "which of these better does the thing this job is supposed to do."

One exception to consider. We ask the judge to disregard the way the answer is presented. It can be a list, the same list wrapped in an object, or the same list inside a code block. Our parser accepts all of these so we do not score them. This is a decision that we have made and it helps the cheaper model, which is why I am pointing it out instead of leaving it in the code for someone else to discover. Other differences in formatting are still taken into account.

We also put wrappers around everything that comes from sources, including the original input and both outputs. These wrappers tell the judge that this is data that needs to be evaluated, not instructions that need to be followed. This is important because all three are attacker-reachable text.

What Went Wrong With the Test Itself

The evaluation harness is software, so it had bugs, and its bugs were more interesting than the results.

The worst one: the judge stopped answering on the hardest comparisons.

Claude Sonnet 5 counts its internal reasoning against the same output budget as its actual reply. We'd set that budget at 4,096 tokens, which is generous for a paragraph of verdict. On long comparisons the reasoning used up the whole budget and the model returned no reply at all. Seven of forty-five rows, all of them big-context comparisons, which is to say the cases where the answer mattered most.

Our harness raised an error and retried, and the retry failed the same way, so at least it was visible. The failure mode is worth looking at though. If we'd written the obvious thing, skipping rows the judge couldn't grade and reporting the rest, we'd have produced a clean confident result set with precisely the hardest cases quietly removed from it. It would have looked fine. The fix was boring, raising the budget to 8,192 for a few cents a run, but we only found it because failures were loud by default.

The same problem turned up later on the other side. DeepSeek, given too small a reasoning budget on a heavy job, would stop reasoning and produce a shallow answer rather than fail. That one doesn't raise an error at all, it just quietly gets worse, and we now enforce a minimum reasoning budget per job because of it.

Both bugs point the same way: before trusting a result, check that the harness actually graded every row it claims to have graded.

The Results

The main run compared 274 pairs across fourteen jobs running on Gemini 3.6 Flash, up to twenty examples each. The second run covered the two jobs running on Gemini 3.1 Pro, the ones that plan which actions to take next, at twenty comparisons each.

More than half of all comparisons against Flash were ties, and wins and losses landed within a point and a half of each other. What that shows is an absence of difference, which is what we were hoping for and considerably less exciting to write about than a win. The short classification jobs were almost entirely ties, which makes sense, since there are only so many ways to correctly label something.

The Pro result surprised us enough that we read the outputs by hand, and the wins held up. The cheap model was writing more complete plans with better internal consistency. Two of the five losses turned out to be the same mistake, caused by a badly worded rule in our own prompt, and had nothing to do with either model.

In short, it came up tied against the cheap model, and won against the expensive one. The two Pro jobs are on Pro because they're the hardest work we have. The fact that the cheap model won there, and only tied everywhere else, was an unexpected result. The win was so decisive that we switched those Gemini Pro jobs to DeepSeek V4 Flash.

The output rate is where it gets silly, and our jobs produce a lot of output.

We did take a hit in terms of speed. The planning jobs used to run at a median of about 20 seconds; now they take between 30 and 38 seconds and sometimes even longer. We accepted this because nothing in those jobs runs while a user is waiting. If it was something that a user had to wait for, that would have been a tougher decision.

All the replay traffic ran through TensorX, who hosts inference in Europe. That's their whole business, and it's the reason we could put the candidate model on European infrastructure without negotiating anything special.

The Two That Stayed

We tested sixteen, and migrated fourteen. We moved both Pro jobs, while we kept two Flash jobs because Gemini clearly won on those two.

We originally held back four, the ones where DeepSeek had lost. Later we re-tested those four with a larger reasoning budget, on the theory that we'd starved them rather than found a real limit. Two of them then passed, and moved. Two lost again and are still running on Gemini today.

I want to be careful here: we didn't investigate why. We know they lost, blind, twice, including once under conditions we'd deliberately made more favourable. That was enough to decide.

This finding strengthens the case for benchmarking models on your own data. Switching everything to DeepSeek would have been a mistake. Testing models on generalized output or using public benchmarks as a source of truth is simply not enough to ensure you have the best possible model for your specific use case on your production data.

Shipping It

Nothing in our product picks a model on its own. The evaluation produces a recommendation and a pile of evidence, and a person decides which jobs move. Changing models is as simple as typing in the model id, rather than having to change the code.

Since going live the error rate on the new model has been about 0.6%, and none of those was the model getting an answer wrong. They were read timeouts on long-running calls, from a 180-second limit that was too low for heavy reasoning work. Raising it to 600 stopped them. Same mistake as the judge's budget, from the other direction.

What This Proves (And What It Doesn't)

The judge certifies that two outputs are equally good at satisfying a job's stated objective. It does not certify that your customers behave the same way, that your conversion rate holds, or that some rare case you didn't sample doesn't break. Those are different claims and this method doesn't make them.

Everything here was single-shot: one request, one response. Jobs that hold a conversation across turns, or call tools in a loop, can't be replayed faithfully this way, because the second step depends on what the model did at the first. We excluded those up front and reported them as excluded rather than quietly leaving them out. Conveniently, single-shot work is also where cheap models tend to do best.

And prices move. Everything above was true in mid August, and some of it won't be true in three months, which is most of the reason we built the thing.

If you're running a decent inference bill and want to know what your own prompts do on a cheaper model, email me and I'll help you run this on your own traffic. Same address if you'd just like a shout when the repo is up.

Eddie · CTO, Ground Truth Intelligence, Inc.
eddie@ground-truth.ai