When this guide is useful
Product owners whose AI assistant produces plausible but unsupported business answers.
- The assistant invents policy details, references or customer information.
- Answers change materially when the same question is repeated.
- Citations are present but do not support the statement made.
- Staff spend as long correcting output as they previously spent writing it.
Classify errors before attempting a fix
Collect examples with the question, permitted evidence, generated answer and expected result. Separate factual errors from missing context, stale data, wrong record selection and presentation problems. A failure to find a current policy requires a different intervention from a model misreading a clearly retrieved table.
Use a short error log with severity and business impact. An invented address in a marketing draft and an invented bank detail in an invoice workflow need different controls. Do not combine them into one reassuring average accuracy score that hides the failures that matter most.
Give the model a bounded evidence task
Retrieve the specific records or document passages needed for the question. State which source governs when there are conflicting versions and ask the model to identify missing information. Validate dates, identifiers, currency codes and arithmetic using deterministic checks wherever possible.
For an answer about an order, fetch the order from the authorised system rather than asking the model to infer it from an old conversation. Preserve the distinction between source facts, calculations and suggestions in the interface. A customer should not have to guess which part of an answer came from a live record.
- Require source identifiers for material factual claims.
- Reject output that references records outside the retrieved set.
- Escalate conflicting or incomplete evidence.
Build an evaluation set that can disprove success
Include ordinary questions, ambiguous language, out-of-scope requests and cases with deliberately missing information. Have a knowledgeable reviewer record the expected answer and acceptable variations. Keep some cases separate from prompt development so they reveal overfitting to the examples used during tuning.
OpenAI's evaluation guidance describes the variability of generated output and the need for suitable evaluation. Repeat important cases, compare changes against the same baseline and review both false answers and unnecessary refusals. A system that refuses everything may avoid fabrication but still fail its business purpose.
Release improvements with proportionate review
Change one part of the workflow at a time: retrieval, prompt, validation or model. Otherwise a better score gives little information about what helped. Track the version of each component with the evaluation result so later regressions can be investigated.
For low-impact drafting, a visible review step may be sufficient. For commitments, payments or decisions affecting people, the workflow needs stronger checks and an authorised decision-maker. Monitor a representative sample after launch, and turn new failures into test cases instead of relying on users to repeatedly report the same problem.
From diagnosis to delivery
How Britixo can help
Britixo can review an existing AI workflow or help design a new one with an explicit accuracy standard. A scoped assessment can examine retrieval, source quality, prompts, structured output checks and the user's review experience. Deliverables can include an error taxonomy, evaluation dataset, measured comparison and a prioritised improvement plan. We can then implement agreed changes and define monitoring appropriate to the use case. Share a small set of sanitised examples showing the question, expected result and actual output when contacting Britixo; the most useful evidence includes failures as well as successful demonstrations.
Talk through your requirementsCommon questions
Will a larger model eliminate hallucinations?
No. It may improve some tasks, but missing evidence, wrong permissions or stale records still need to be fixed. Compare candidate models on your own representative cases.
Does telling the model to be accurate solve the problem?
It can help communicate intent, but it does not provide missing facts or enforce correctness. Combine instructions with evidence, validation, testing and an appropriate review process.
Which accuracy measure should we use?
Choose measures tied to the task, such as supported factual claims, correct record selection and successful escalation. Report serious errors separately from minor wording issues.
Sources and further reading
Use the original documentation to check current product terms, eligibility and technical details.
Published by Britixo on 29 October 2026. How these guides are prepared. Product availability and delivery scope are confirmed for your requirements.