How to Stop ChatGPT Hallucinating on Regulated Compliance Work

A UK Responsible Gambling (RG) team came to us with a problem that anyone automating regulated work will recognise. They use ChatGPT to turn automated compliance triggers into formal analyst notes. When a customer breaches a threshold — turnover up more than 100% in 30 days, a session running past its limit — an analyst copies the relevant data into ChatGPT, and the model, following a pre-uploaded prompt, drafts a note for the customer’s compliance file.

That note is not a draft email. It is a legal document reviewable by the UK Gambling Commission. And after a while, ChatGPT started to hallucinate figures, miss key information, insert em dashes that broke house style, and quietly drift away from the rules by the afternoon. On a regulated file, each of those is a compliance issue.

The question they asked was simple: can we get the consistency regulated work requires from a language model at all? The answer is yes — but not by tweaking the wording of a prompt inside a chat window. It takes a change in architecture plus a disciplined rewrite. This article walks through the exact method, using the RG workflow as the worked example, so you can apply it to any regulated process where the same input must produce the same trustworthy output every single time.

Why ChatGPT drifts on regulated work

Before fixing anything, it helps to name the failure modes precisely. On the RG workflow we reproduced five distinct ways the output broke, ranging from critical to merely annoying.

The two critical ones are the dangerous pair. Hallucinated figures put a wrong deposit or profit-and-loss number on a regulated file — the model fills a gap from training memory rather than admitting the data is absent. Convention reversal is subtler: this operator uses an inverted profit-and-loss convention where a positive number means the customer is losing. It is counterintuitive, so the model reverts to the standard convention it learned in training and writes that a customer is losing when they are actually winning.

The high-severity issues — em dashes that violate house style, and session drift where rules are ignored later in the day — do not put a wrong number on file, but they erode trust in the output and create rework. The medium issue, informal intervention names instead of the exact regulatory terms, quietly corrupts the compliance record.

The root cause is statelessness — or the lack of it

Four of those five failures share one root cause. In a single ChatGPT session, every case you review is appended to the context window. Review case one and the model sees the prompt and one case. By case fifteen it sees the prompt plus the statistical residue of fourteen previous exchanges. The prompt’s rules are still there, but they are competing with a growing pile of context, and attention to them decays.

This is why the same prompt produces a perfect note at 9am and a flawed one at 4pm. The architectural fix is statelessness: every case must be a completely fresh, isolated call that sees only the locked prompt and the current case. Nothing carries over. An AI agent configured this way gives the prompt the same attention on case fifty as it did on case one.

Statelessness is necessary but not sufficient. It stops the drift, but it will faithfully reproduce a bad prompt on every call. So the prompt itself has to be hardened.

Ready to grow your business?

Start your free trial today and see results within days.

Request demo Log in

Step 1: Harden the prompt for primacy and recency

Language models do not weight every line of a prompt equally. Attention is roughly U-shaped — strongest at the beginning and the end, weakest in the middle. The original RG prompt fought this. Its most important rule, the inverted profit-and-loss convention, sat in the middle of the prompt, stated once. The em dash ban was the very last line, also stated once.

The rewrite moved the critical rules to where the model actually looks and reworked the weak parts of the system prompt . Here is what changed and why.

The single biggest lesson here: vague adjectives are not instructions. Telling a model to be “natural” actively encouraged the em dashes and rhetorical flourishes that broke house style. Replacing that with concrete, testable rules is what made the style consistent.

Step 2: Handle missing data instead of inventing it

Hallucination is often framed as a model defect. On structured work it is usually a prompt defect: the prompt never told the model what to do when a field is absent, so it did the statistically likely thing and produced a plausible value.

The fix is a missing-data protocol. Split your fields into required and optional. Optional fields are omitted silently when absent. Required fields do the opposite — if one is missing, the model must stop and request it in an exact format rather than generate a note. That single rule converts a silent hallucination into a visible, auditable prompt for more data. It is the difference between a wrong figure on a regulated file and a note that was never produced because the input was incomplete.

Join our newsletter

Get latest tips, trends, and deals for free.

Email address

Subscribe

Step 3: Lock the agent configuration

With a hardened prompt, the agent itself has to be locked down so nothing reintroduces variability. The whole flow is deliberately minimal — three nodes and nothing else.

The configuration choices that mattered:

• System prompt in the System Message field, never the Input field, so it is read as instructions rather than data.
• Temperature 0.2. Low enough for maximum consistency, high enough to avoid robotic prose. Zero was too rigid; one was too creative.
• No tools attached. File and URL retrievers introduce unpredictability and can bypass the agent’s reasoning. For this task the model needs nothing but the prompt and the pasted case.
• Max tokens left at default. Capping tokens caused notes to truncate mid-sentence, and a truncated note on a customer file is itself a regulatory risk.
• A silent-start instruction at the top of the prompt, so the agent does not confirm the rules back to the analyst when the flow first loads.

The one operational rule that matters most

Every fix above is undone by one bad habit: reusing the same chat. The most important operational rule for the analyst team is to start a new conversation for every single case. Never continue reviewing cases in the same window. This is the rule that eliminates session drift — the root cause of most of the original failures — and it costs nothing to follow.

Does it actually work? 9 models, 27 runs

A method is only as good as its evidence. We tested nine models against the hardened prompt, using an identical real case (a Gambling Intensity trigger with prior session-duration history) across three runs each — 27 runs in total — and scored every run against 16 compliance criteria.

Across all 27 runs, no production-ready model hallucinated a figure, reversed the P/L convention, used an em dash, or truncated output. The hardened prompt eliminated every one of the original failure modes on the models that qualified. The differences that remained were about analytical depth and consistency, not correctness.

The disqualifications are instructive. Claude Sonnet 4.6 produced beautiful prose but confused the trigger threshold with the deposit limit — a factually wrong figure on a regulated file, which is disqualifying regardless of how well it reads. DeepSeek’s run-to-run swing (6, 8.5, 9) is the real problem: in regulated work, quality that depends on luck is not quality at all.

Cost is not the constraint

It is tempting to optimise for price, but the numbers make the case against it. Even the most expensive production-ready model, Claude Opus 4.8, costs roughly six cents per note. For a team reviewing 50 cases a day, that is under $90 a month. The cost of a single regulatory finding from an incorrect note would dwarf the annual cost of any model on the list. For regulated compliance, choose the most reliable model you can, not the cheapest.

What a hardened note looks like

The best single note across all 27 runs came from Claude Fable 5. Notice what it does: it states the position in the correct convention, quotes figures verbatim, uses the exact intervention name, and — critically — explains why escalation would be disproportionate rather than simply stating an outcome.

The customer is currently in a winning position of £1,612.50 with an account balance of £8,962.50, which represents an improvement on the winning position of £1,275.00 recorded at the previous review… Session duration was reviewed under Session Duration Level 1 on 19/07 and again on 20/07… Given that those decisions were made within the last few days, insufficient time has elapsed to assess their effectiveness, and repeating or escalating on the same behaviour at this stage would be disproportionate… In this context, No Further Action Required is the proportionate outcome.

That is the standard regulated work demands: evidence-based, proportionate, and identical in quality on the first case of the day and the fiftieth.

Applying this to your own regulated workflow

The RG note generator is a specific case, but the method generalises to any process where a language model must produce trustworthy, repeatable output on regulated or high-stakes data:

1. Move off the chat window. Run each case as a stateless call so context cannot accumulate and drift.
2. Harden the prompt for primacy and recency. Put critical rules first and repeat them last. Add self-checks.
3. Give the model a missing-data protocol. Required fields block generation when absent; optional fields omit silently. Never let it guess.
4. Lock the configuration. Low non-zero temperature, no unnecessary tools, uncapped output, prompt in the system message.
5. Test before you trust. Run the same case multiple times across candidate models and score against explicit criteria. Consistency across runs is the metric that matters.

Do those five things and you convert a model that hallucinates by the afternoon into one that produces audit-ready output every time. The consistency regulated work requires is achievable — it is an architecture and prompt-engineering problem, and both are solvable.