How to tell whether AI models really agree
When several AI models answer one question, “they mostly agreed” is not a result. A final vote, a consensus index and the conflicting positions, explained.
“They mostly agreed” is not a result
Ask three models a hard question and you get three long answers that overlap. It is tempting to read the overlap as agreement. Often it is not. They share the framing and differ on the one point that decides the matter.
A summary written by a fourth model makes this worse. A summariser is rewarded for a tidy conclusion, and a tidy conclusion is exactly what a split council does not have.
Make each model commit
The Council ends every session the same way. One motion is put: yours, or one the clerk drafts from the debate. Each member answers in three lines, without seeing the others:
- For, against or abstain.
- A score from 0 to 10 for how far it agrees with the motion as worded.
- One sentence: its final position and, if it does not vote for, what keeps it from it.
The count is arithmetic
No model is asked whether the council agreed. The page counts.
- Agreement follows the rule you chose: every member on the same side, two thirds, or more than half. A council can also agree to reject the motion.
- In favour: how many of the members seated.
- Consensus index: worked out from the scores. It is 100% when every score is the same and 0% when the council is split evenly between 0 and 10. Scores of 8, 2 and 4 give 62%. Scores of 8 and 2 give 49%. (It is Tastle and Wierman’s measure of consensus, for the curious.)
A worked example
Three members vote on the motion “Move the nightly jobs to a queue this quarter.”
- The first votes for, with a score of 8.
- The second votes for, with a score of 7.
- The third votes against, with a score of 3: “Not until the on-call rota is staffed.”
Two of three are in favour. Under the rule “every member on the same side” the chair states no agreement. Under “two thirds on the same side” the same votes are an agreement, because two of three is enough. The consensus index is 67% either way: the scores are what they are, whatever rule you set.
That is why the statement gives both. The count tells you whether the bar you set was met. The index tells you how far apart the members ended. And the third member’s sentence tells you what would have to change.
Which rule to choose
- Every member on the same side for a decision that is hard to undo. One dissent is worth reading before you go on.
- Two thirds when you want a clear lean and can live with an objection on the record.
- More than half for a first sounding, when you mostly want to see where the split is.
Abstentions and answers that cannot be read do not count towards agreement. The bar is counted against every member seated, not only against those who gave a clear vote.
When they do not agree
The final statement says so in two words. It names the limit that was reached: the rounds, the session time, or you closing the debate. Then it sets out the conflicting positions by side, in the members’ own sentences. An answer that cannot be read as a vote is shown as unreadable. It is never counted as agreement.
What the number does not tell you
The index measures how close the models are to each other. It does not measure whether they are right. Three models can agree at 100% and be wrong together. Read a high figure as “no model raised an objection” and a low one as a map of where to look.
Put a question to the council or see how to compare models under the same rules.
More guides
Updated