Why should your AI not grade its own work? (And how to add a second checker)
Updated November 2026
An AI tends to be kinder to its own answers than to anyone else’s, and kindest when it is wrong. Here is what the studies say and seven steps to add a second checker.
Ask an AI to check its own work and it will often say the work is fine. This article covers what the research says about that, what it does not say, and how to set up a second checker so one blind spot does not pass everything.
Also on this topic: what an MCP server does for a one-person business, incrementality: did the offer cause anything and what MCP Analytics is.
Our position: use a second checker that works alone, ideally from a different family of models, and add a real test wherever you can write one. Do not rely on “are you sure?”
The answer first: a real test
An AI model tends to grade its own work less reliably than it grades other work, and it is softest exactly when it is wrong. Models that think alike often make the same mistakes, so a checker from the same family tends to pass the same errors. On tests with known answers, a second checker from a different family, working on its own, caught more. It is not a guarantee: different makers can still think alike. Real tests beat both wherever you can write one.
Think of a student grading their own test. They wrote the answers, so the answers look right to them. A teacher who did not write them reads with fresh eyes.
What this means for you
- Do not stop at “are you sure?” It is a weak check on its own.
- Add a second checker that works alone. A different family of models is a better bet than the same one.
- Trust what both find first. Keep the single finds, and have the other checker look at each.
- Add a plain test where you can. A total that must match the bank does not share the AI’s blind spots.
What the research says
Many studies find an AI tends to rate its own work higher. Self-bias showed up in every model tested in one study, across languages and tasks (Xu et al., Pride and Prejudice, ACL 2024).
An AI can often tell which answer it wrote, and tends to favor it. In one study, GPT-4 picked out its own writing 73.5% of the time, and the better a model was at spotting its own writing, the more it favored it (Panickssery, Bowman and Feng, NeurIPS 2024).
It is softest when it is wrong. Strong models often prefer their own answers for a good reason: they are often right. But on a math test, one strong model still picked its own wrong answer over a right one 86% of the time (Chen et al., 2025, preprint).
Not every model does this. GPT-3.5 showed no self-preference in the first big study of AI judges (Zheng et al., NeurIPS 2023), and Llama and Mistral showed none in a later one (Spiliopoulou et al., 2025, preprint). The honest claim is “less reliably”, not “cannot”.
The tilt spreads to the family, and alike models share mistakes. A family is a set of models from the same maker, built alike. Some judges also gave higher scores to their own model family (Spiliopoulou et al.). On a multiple-choice test, when two models were both wrong, they picked the same wrong answer about 60% of the time, where luck would give 33%, because there were three choices (Kim, Garg, Peng and Garg, ICML 2025). In one example, an AI judge scored its small cousin 71% correct and a stronger outsider 67% (Goel et al., ICML 2025).
“Are you sure?” is a weak check on its own. On the reasoning tests studied, asking a model to re-check itself without new information often did not help and sometimes hurt: GPT-3.5 fell from 75.8% to 41.8% on a common-sense test (Huang et al., ICLR 2024). Finding the mistake is the hard part. Once the location is pointed out, models could usually fix it in those tests (Tyen et al., Findings of ACL 2024).
Same-model review is not useless. A model’s own feedback improved its work on several tasks in Self-Refine (Madaan et al., NeurIPS 2023). Review works best with reliable outside feedback (Kamoi et al., TACL 2024).
A different family checks better, with limits. On tests with known answers, checking by a different model family worked better than self-checking or checking within the family (Lu et al., 2025, workshop paper). A small panel of different models agreed with people better than one large judge, at lower cost (Verga et al., 2024, preprint). But strong models are starting to make the same mistakes even across makers (Kim et al.; Goel et al.). A weak second checker added little, and a fresh session of the same model did about as well as a top cross-family checker in one small study (Song, 2026, preprint). Independent human teams share blind spots too (Knight and Leveson, 1986).
Most of this evidence comes from tests, not real businesses. Read each number as true for its own study only.
What a second checker is, and what it looked like for us
A second checker is anything that reviews the same work on its own: another AI model, another person, or a plain test. Here is one case from our own setup.
We had Claude and ChatGPT check the same part of our setup before adding ChatGPT to our team of AI agents. Merged, the two lists held 21 problems.
| Who found it | Problems |
|---|---|
| Both Claude and ChatGPT | 6 |
| Only Claude | 9 |
| Only ChatGPT | 6 |
| All problems, merged | 21 |
One serious problem only ChatGPT found was a step that could quietly throw away work before it was saved. We fixed the most serious problem first, soon after merging.
Limits: it is one audit, not a study, and the two worked differently (Claude ran the system; ChatGPT went through the code), so method as well as family explains the difference.
A second checker versus asking “are you sure?”
- Asking “are you sure?” The same model re-reads its own answer, with no new information. On the reasoning tests studied this often did not help and sometimes hurt.
- A second checker. A different reader, working alone, from a different family if you can. It does not share the first one’s habits, so it can catch what the first passes.
How to run one on your own work
- Give two checkers from different families the same job, in writing. The same scope, the same questions, the same rule for what counts as a problem.
- Each works alone and saves its report before seeing the other. No peeking. This is a rule from our practice, not a study.
- Every finding comes with a way to check it again. A command, a test, a file.
- Merge the lists. Mark what both found and what only one found.
- Start with what both found. Those are the most likely to be real (a strong signal, not proof). Do not throw away the single finds: hand each to the other checker and point to the spot.
- Sort by how bad each problem would be, and fix the worst first.
- Wherever you can, add a real test. A total that must match the bank, an email address that must exist. Tests as feedback helped code most when tests were available (Chen, Lin, Scharli and Zhou, ICLR 2024). Tests do not think like the AI, but they can be wrong too, so keep them simple.
What you need and what you get back
What you need: one piece of work, the job written down, two checkers, and a place to save each report before the other is read.
What you get back: one merged list, with each problem marked as found by both or by one, a way to check each one again, and an order to fix them in.
The aim is an independent check: a second check that works on its own, so it can catch what the first one missed.
Questions business owners ask
Can an AI check its own answer?
Sometimes, and it helps on some tasks. But studies find it is less reliable at judging its own work than someone else's, and it is softest when the answer is wrong. Treat a self-check as a first pass, not as the final check.
What does "a different family" mean?
A family is a set of AI models from the same maker, built alike. Claude models come from Anthropic. GPT models come from OpenAI. A checker from a different family was built by different people, with different data, so it tends to miss different things.
Is a second AI always better?
No. Different makers can still think alike, and strong models are starting to make the same mistakes across makers. In one small study a fresh session of the same model did about as well as a checker from another family. A second checker helps most when it works alone and every finding comes with a way to check it again.
Do I need two paid AI tools?
The steps work with any two checkers that work alone: two AI models, or one AI and one plain test such as a total that must match the bank. Tests do not think like an AI. They can be wrong too, so keep them simple.
Is your own audit proof?
No. It is one audit, not a study. The two checkers also worked in different ways (Claude ran the system, ChatGPT went through the code), so method as well as family explains the difference.
Does the person still decide?
Yes. The checkers raise problems. A person sorts them by how bad each would be and decides what to fix first.
Sources
- Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685
- Panickssery, Bowman, Feng. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. https://arxiv.org/abs/2404.13076
- Chen, Wei, Zhu, Feng, Meng. Do LLM Evaluators Prefer Themselves for a Reason? 2025, preprint. https://arxiv.org/abs/2504.03846
- Xu et al. Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. ACL 2024. https://aclanthology.org/2024.acl-long.826
- Spiliopoulou et al. Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge. 2025, preprint. https://arxiv.org/abs/2508.06709
- Kim, Garg, Peng, Garg. Correlated Errors in Large Language Models. ICML 2025. https://arxiv.org/abs/2506.07962
- Goel et al. Great Models Think Alike and this Undermines AI Oversight. ICML 2025. https://arxiv.org/abs/2502.04313
- Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024. https://arxiv.org/abs/2310.01798
- Tyen et al. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of ACL 2024. https://arxiv.org/abs/2311.08516
- Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. https://arxiv.org/abs/2303.17651
- Kamoi et al. When Can LLMs Actually Correct Their Own Mistakes? TACL 2024. https://arxiv.org/abs/2406.01297
- Lu, Teehan, Jin, Ren. When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers. 2025, workshop paper. https://arxiv.org/abs/2512.02304
- Verga et al. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. 2024, preprint. https://arxiv.org/abs/2404.18796
- Song. When Does a Second Model Help? Cross-Model Review in LLM Verification. 2026, preprint. https://arxiv.org/abs/2610.01471
- Knight and Leveson. An Experimental Evaluation of the Assumption of Independence in Multiversion Programming. IEEE Transactions on Software Engineering, 1986. https://doi.org/10.1109/TSE.1986.6312924
- Chen, Lin, Scharli, Zhou. Teaching Large Language Models to Self-Debug. ICLR 2024. https://arxiv.org/abs/2304.05128
How we checked the claims
Three checks on this article. Each study number is printed as that study reports it.
Every quote against the paper
Each number and each name above was checked against the paper it comes from, and the papers are listed with links. A preprint is a paper shared before other experts have reviewed it, so it carries less weight. Preprints and a workshop paper are marked as such.
A challenge from the other family
We asked ChatGPT to challenge the first draft of the research. We kept what held up against the papers and dropped what did not.
Our own case, scoped
The counts for our audit come from the merged problem list we keep in our own records, which we cannot publish here. The article calls it one audit, not a study, and notes that method and family are mixed together in it.