New language models appear constantly, often promising better quality, lower cost or faster responses. If you run an automation that depends on a model, such as sorting emails, extracting data from documents or classifying support tickets, it is tempting to swap the model name in your configuration and move on. That is how quiet failures reach production: a missing field, a category applied differently, or a format that breaks a downstream step.
Switching models is a system change and deserves the same care as any other. This guide describes a practical, repeatable way to test a new model before it takes over a live workflow. The running example is a support-ticket classification workflow, but the steps apply to most model-based automations.
1. Define the output contract
Before you compare anything, write down what a correct output looks like. This is your output contract: the agreement between the model step and everything that depends on it.
A useful contract answers three questions.
Which fields are required? For a ticket classifier, that might be category, priority, language and a short summary. If any of these is missing, the next step in the workflow cannot run.
Which formats are allowed? Be specific. category must be one value from a fixed list (for example billing, technical, account, other). priority must be low, medium or high. The whole response must be valid JSON with no extra text before or after it. The summary must stay under a set length.
Which failures are unacceptable? Not every mistake is equally serious. Putting a billing question in “other” is annoying; marking an outage report as low priority may be a real problem. Write down the failures you will not tolerate, such as invented order numbers, a wrong language tag that sends a ticket to the wrong team, or personal data copied into a field that is logged elsewhere.
The contract turns a vague question (“is the new model better?”) into checkable criteria and tells you what to measure later.
2. Build an anonymised test set
Next, collect real examples that represent the work the automation actually does. Before you use them, remove or replace personal data: names, email addresses, phone numbers, account numbers, addresses and anything else that could identify a person. Use consistent placeholders so the text still reads naturally, and follow your organisation’s data protection rules about where test data may be stored and which providers may process it.
A good test set covers three groups.
Typical cases. The everyday tickets that make up most of your volume: a password reset request, a question about an invoice, a bug report with clear steps.
Missing data. Tickets with an empty subject line, a body that only says “see attachment”, or a message cut off halfway. Check what the model does when information is not there. Does it choose a sensible fallback, or does it make something up?
Ambiguous inputs. Tickets that could reasonably belong to more than one category, messages in mixed languages, sarcastic or angry complaints, and tickets that contain several unrelated requests. These are where models tend to differ most.
For each example, record the expected output according to your contract, and note where your team disagrees about the right answer. Add new examples whenever a real failure shows up in production.
3. Compare both models with consistent prompts, inputs and settings
A comparison is only meaningful if the model is the only thing that changes. Run the current model and the candidate model against the same test set, using:
- the same prompt, including the system instructions, examples and output format description;
- the same inputs, in the same order and with the same preprocessing;
- the same settings, such as temperature, maximum output length and any structured-output or JSON mode that both models support.
Some differences cannot be avoided, for example when only one model supports a structured output feature. Document them so you know what you actually compared.
If your settings allow randomness, run each input more than once. Inconsistent answers are a risk in an automation, even if the best answer looks good.
Store every raw output alongside the input, the model name and version, the prompt version and the settings. This makes the evaluation reproducible.
This is also the right moment to look at cost and speed. Record the number of input and output tokens for each run and the response time. A newer or cheaper model may change your costs in unexpected ways, for example if it produces longer outputs or needs a longer prompt to behave correctly. Before you switch, estimate the expected monthly cost from your real token volumes, for instance with an LLM API cost calculator, and check current prices on the provider’s own pricing page.
4. Evaluate validity, factual errors, task completion and review need
With outputs from both models in hand, evaluate them against your contract. Four dimensions cover most automations.
Output validity. Does every response parse? Are all required fields present? Are values inside the allowed lists and formats? This check can and should be automated. A model that is slightly more accurate but occasionally returns broken JSON may cause more trouble than it solves.
Factual errors. Does the output state anything that is not supported by the input? In a ticket summary, that might be an invented product name, a refund amount that was never mentioned, or a claim that the customer already tried a fix they did not mention. These errors are harder to detect automatically, so review a sample by hand, focusing on the missing-data and ambiguous cases.
Task completion. Did the model do the job? For classification, compare the predicted category and priority with your expected labels. Look beyond the overall share of correct answers: check where the errors fall. A model that is wrong mostly on low-impact categories may be acceptable; one that misses urgent tickets is not.
Human-review need. Estimate how many outputs a person would have to check or correct before they could be trusted. If your workflow already routes uncertain cases to a human, see how often each model triggers that route and whether it does so for the right tickets. A model that saves money per request but doubles the manual review load may not save anything overall.
Summarise the results side by side and list the specific examples where the candidate is worse; they are often more informative than any aggregate number.
5. Release gate and rollback plan
Testing only helps if it leads to a clear decision. Define a release gate before you look at the results, so the outcome does not influence the criteria.
For the support-ticket classifier, a release gate might look like this:
- every output in the test set is valid JSON that meets the contract;
- no unacceptable failures from step 1, such as an outage report classified as low priority or an invented order number;
- task completion on typical cases is at least as good as the current model;
- results on ambiguous and missing-data cases are no worse in the categories your team has marked as high impact;
- the human-review rate does not increase beyond a threshold your team agrees on;
- the estimated running cost and response time fit within your budget and service targets.
If the candidate passes, do not switch all traffic at once. Start with a limited rollout: send a small share of new tickets to the new model, or run it in shadow mode, where it classifies tickets in parallel but its results are only logged, not used. Compare its decisions with the current model and the final human-assigned categories for a defined period.
At the same time, prepare the rollback plan:
- keep the previous model and prompt version available and configurable, ideally switchable with a single setting rather than a code change;
- log the model and prompt version with every processed ticket, so you can identify which tickets were handled by which model;
- define the signals that trigger a rollback, such as a rise in invalid outputs, more tickets reassigned by agents, or complaints about misrouted requests;
- decide in advance who can trigger the rollback and how affected tickets will be reviewed afterwards.
Once the new model has run cleanly for the agreed period, increase its share step by step until it handles all traffic. Keep the test set and the results. The next comparison will then take a fraction of the time.
Conclusion
A model switch can improve an automation, but only if you know what “better” means for your workflow. Define the output contract, test with realistic anonymised data, compare under identical conditions, evaluate what actually matters downstream, and release behind a clear gate with a rollback ready. That discipline turns a risky configuration change into a routine, well-documented decision.