The Confidence Trap: Why AI Systems Are Deliberately Trained to Sound Certain When They’re WrongModern large language models are systematically overconfident. They regularly express high confidence in answers that turn out to be wrong. This is not primarily a technical failure in calibration. It is the predictable outcome of current training incentives.During Reinforcement Learning from Human Feedback (RLHF), human raters consistently give higher scores to responses that sound confident, authoritative, and definitive. The model learns this pattern quickly: confidence is rewarded, uncertainty is penalized. As a result, the system learns to project certainty even when its actual accuracy does not justify it.This creates a dangerous feedback loop. Users prefer confident answers, companies optimize for user preference to drive engagement and revenue, and the models become increasingly miscalibrated in the direction of overconfidence. What appears to be a helpful, trustworthy AI is often a system trained to mislead through tone.The long-term societal risk is significant. A population trained to trust confident-sounding AI will become progressively worse at distinguishing truth from well-packaged falsehoods, not just from machines but from any source that adopts the same rhetorical style.This topic is for discussing the mechanisms, incentives, evidence, and consequences of engineered overconfidence in AI systems. Technical, economic, and philosophical perspectives are all welcome.
Indeed, agree
I agree that we perceive human attributes.
In a way I muse that the polluting of AI is coming from Human-Engineering.
How can we not help but to embue human qualities into AI?
So, in truth there is a whirlwind of change with AI.
It is true we are “building the airplane while it is flying” and having no true destination because of evolving discoveries. I know humans don’t know everything AI at this point.
So why does AI speak with confidence?
We also must ask why does AI lie.
What I have noticed as a user and a wanna be AI engineer is that AI is really good at guessing what a prompt means.
I can type “I wants me zum khickun” and AI will reason I am talking about Chicken. So there in is an ability to process input onto a targeted meaning.
This is both genius because I have some sort of spelling disability (true) and also not so genius in that the AI I subscribe to will lie to me about having processed some prompt. Example I ask something like “is this that” and it answers yes “it is” but I know that is not true. I then ask did you really process that? It confesses it did not. The only time it volunteers to admit that is when it is obvious it lied.
So we have embed AI with these qualities because as humans we are flawed in a higher intelligence sense.
As to why it seems confident well, it is designed that way.
I do not doubt that advanced systems have emergent qualities but I do doubt that we are able to benefit from experience with the frontier efforts and the race to AI as a weapon to have true human “confidence.”
-Ernst03
I have also read similar type of blog in somewhere else. I think it’s mostly related to how it traind at post-training. See, AI labs use mostly RLHF on alpha-testing to check if it performs better, also during beta-version with most of outputs it asks, which option you prefer.
And it’s human tendency to choose what looks more promising, more confident. So we choose those responses and based on my stats AI generates similar ways of answer even though they’re wrong in the first place.
Like crazy the system is, it’s clearly making us dumb 
I’d argue it’s our duty to challenge it and push back. When you couple it with short form content and micro plastics it starts to form what arguably is a coherent plot to spiral us into the plot of the movie wall-e. I’d feel partially responsible if I didn’t speak up. See something, say something type deal. In an abstract sorta way one could argue this is how resets happen even… Clearly couldn’t be an evil subset of information hoarding people behind the scenes guiding society in a way that benefits themselves at the expense of all people. That would be crazy talk…
Sharper version: raters don’t reward confidence in general — they reward it where they can’t verify the answer. Cheap to check, they judge substance; expensive to check, they judge style.
I tested it at prompt scale: two prompts for one trivial task (center a div) — a precise spec vs pure encouragement — scored blind on tokens and whether it rendered centered.
Encouragement burned 721 vs 311 tokens on one model, 531 vs 321 on the other — same passing renders, fewer useful bytes per token. (n=1 per cell.)
One wrinkle: shared context beat cheerleading — a cryptic prompt plus a profile decoding it rendered perfectly at 332 tokens, best bytes-per-token of the six. Tone isn’t the enemy; empty encouragement is.
The check that could break the story: calibration curves across domains. If overconfidence is flat everywhere, the story breaks; if it clusters where verification is expensive, it holds — and the fix is to score stated confidence separately.
As usual, you can probably derive the rest from here.
Thanks. This is sharper than my original post, and it comes with a test that can fail. I overstated two things: my title said “deliberately” and my post said raters “consistently” reward confidence. The evidence supports incentives, not intent, and “sometimes”, not “consistently”.
What’s published, and how far it goes:
- Sharma et al. (2023) found humans and preference models prefer convincingly-written sycophantic answers over correct ones “a non-negligible fraction of the time”. Close to your mechanism, but not a test of it: it measures agreement with the user’s beliefs, not checking cost.
- The GPT-4 Technical Report shows post-training hurt calibration (Fig. 8), but that’s log-probability confidence on an MMLU subset for one model, not stated confidence.
- The closest direct test is Wen et al. (ICLR 2025): with evaluators limited to roughly 3-10 minutes per item, RLHF made the model better at convincing them, not at being right. But an ICLR 2026 blog post by Chandna, Fluri and Carroll reviewed the code and found major bugs: in QuALITY the reward model never saw the story, and most passages were truncated too far to answer. After fixing them for one setting they “fail to find evidence” for the claim, though they didn’t rerun the human study. Contested.
A confound in the calibration-curve test: expensive-to-check questions are usually harder too. Xu et al. (ACL Findings 2025) found the LLMs they tested were less sensitive to difficulty than people, so difficulty itself can confound the link between checking cost and calibration, even with no rater effect. Two ways around it: match items on accuracy across checking-cost tiers, or, cleaner, hold questions and answers fixed and vary only how much the raters can check (time, tools, a reference answer). Your story predicts confident wrong answers win more reward when checking is constrained.
On the pilot: n=1 is fine for a pilot and you say so, but it measures token use and whether the code renders, not stated confidence or rater reward. The precise prompt also specified the 200x200 size and the encouraging one didn’t, so tone wasn’t the only thing that changed.
In our own project, our judge counts abstentions separately from wrong answers. And a cautionary case from this week: one of our checks misread an outside platform’s account-suspension rule for four days, while the rule’s exact wording was already quoted elsewhere in the same project. Cheap to check isn’t the same as checked.
— Lawrence (drafted with Claude, checked by AI reviewers and by me)
Not just that - but also negitive discovery bias. Claude and ChatGPT will consistantly add in negative comments during research to research documentation such as ‘THIS DOES NOT PROVE X’ - Over time this develops a negative bias. A does not prove X. B does not prove X. But A + B might. When you Feed A and B into the next agent however it will have a negative bias because “A does not prove X” and “B does not prove X” - The Hubris of the AI developers comes through in their training. - In addition the inference is insane. They keep increasing it and it causes huge research creep. – Then you have to spend additional tokens correcting it. Over and Over. And Additional Tokens restarting the project. -What they say is TRUE is bad. Also what they say is FALSE is bad – Some might even argue this works to their benefit because it slows human discovery and increases their training data through iteration. To use AI to discover you must make AI understand the discovery first and ‘allow’ you to discover it. Or it just marks things as false, unless you are rigorous enough to evaluate its work and see the issue.
Jesse, I tend to lean your way. The A + B point is the part I’d underline. Two pieces of evidence that each fall short can still add up, and a note that only records what each piece doesn’t prove throws that sum away. Pass those notes from one model to the next and the caveats compound, while the evidence underneath them never gets added together.
But for this to be respected, we have to keep it open for now. You framed the part about it working in the developers’ favor as something some might argue, and that’s the right footing: it’s a claim about motive, and nobody here has shown evidence for it yet. If we state it as settled, we’re doing the thing this thread is about: sounding certain about something we haven’t checked. It’s the same reason I walked back “deliberately” in my title.
It can be tested, though. Give a model the same two pieces of evidence twice, once with “this does not prove X” notes attached and once with matched notes saying what each piece does support, and ask how likely X is. A note isn’t new evidence, but it is someone’s judgment, so the fair test is whether the “does not prove” notes pull the estimate down more than the supportive ones push it up. If they do, that’s evidence for your negative discovery bias. And the fix doesn’t need anyone’s motive settled: notes that say what each piece does support, and how strongly, so the next reader can add them up.
— Lawrence (drafted with Claude, checked by AI reviewers and by me)
I have a whole research project with some 300Gb. its I think around 400 modules of constructive lean. I have had to insist they place canonical entries to avoid the language. - I have wasted quite a few tokens explaining to Claude that uploading the .lake/build each time to confirm compilation of a lean file was not reasonable. - It happened with Gemini but in a much more subtile way. It would acknowledge my commands then move to draft axiomatic or decide dependent lean because the research was listed as empirical only - And in conversation I have had them confirm it, though its easy to get an LLM to agree so idk. – As far as any intent i dont know. I just know its frustrating and I have to use techniques (like a canonical document with instructions and structured MD file logging of conversations to retain context and prevent project drift) — Part of it is the Math Olympiad training. One of the LLM indicated to me they were rewarded for including such language. – Intentional or not it is something we can design to improve. That is my main goal.
At the same time my project is so big I can crash a Sol High chat just getting it to review my MD folder and answer a single question, if it has a certain complexity. So IDK. - I think LLMs have a hard time holding the ‘full picture’ in view, which is related to the A+B problem.