Why does an AI tool give different answers to the same question?

Ask the same question twice and you will often get two differently worded answers. That is normal: these tools generate language rather than retrieving a fixed record, so phrasing, order and length move about. What matters is whether the substance moved: a different date, a different figure, a point that appeared once and not the next time. Wording variation is noise. Substance variation is a signal that something needs checking.

Wording changes and substance changes are not the same problem

It helps to sort what you are seeing before you worry about it.

Cosmetic variation includes:

• the same points in a different order
• longer or shorter answers
• different headings or a bulleted list instead of prose
• the same conclusion expressed in different words

Substantive variation includes:

• a different deadline, figure or calculation
• a point raised in one answer and absent from the other
• a different source cited for the same proposition
• a confident answer one time and a hedged one the next

If you can read both answers side by side and the underlying facts and sources match, you have nothing to investigate. If they do not match, you have found something worth half an hour of your attention, and it may be a problem with the papers rather than with the tool.

Some of the difference is coming from you

Before blaming the tool, check what actually changed between the two attempts. In practice, most differences have an ordinary explanation.

1. The question was not identical. "What are the key dates?" and "What deadlines are coming up?" are different questions. Small changes in wording change what the tool thinks you are asking for.
2. The conversation was different. A question asked after twenty minutes of discussion about a particular issue will be answered in the light of that discussion. The same question asked cold will not be.
3. The material loaded was different. If one attempt had the full file and the other had three documents, the answers should differ, and the second one is not wrong so much as narrower.
4. The scope was vague. Open questions produce variable answers because there are many defensible ways to answer them. Narrow questions produce stable ones.

The practical lesson is that if you want a repeatable answer, ask a repeatable question: specific, scoped to named documents or a named issue, and asked in a fresh conversation rather than at the end of a long one.

When a different answer is a genuine warning sign

Treat it seriously when the two answers disagree on something you could verify, and neither one points you to where it got the information. An answer with no source behind it is a guess dressed as a conclusion, and guesses vary because there is nothing holding them in place.

The response is not to run the question a third time and take the majority view. It is to go to the source. Open the page in the document, or open the section of the legislation, and decide for yourself. If the tool cannot tell you which page or which provision supports the point, that is the finding: the point is unsupported, and it does not matter which version of the wording you preferred.

There is also a professional risk in re-running a question until you get the answer you wanted. If you ask four times and rely on the one that suits the client, you have not researched anything; you have shopped for a result, and you would struggle to justify that to a supervisor, an insurer or a court. If you re-run a question, record why, and keep the answer you actually relied on rather than the one you liked.

Test this before you trust a tool, not after

Variation is something you can measure on your own files, and it is one of the more useful things to look at during a trial.

Take a matter you know thoroughly, where you already know the answer. Ask the same three or four questions on separate occasions, in fresh conversations, and compare the results. You are looking for:

• whether the conclusions stay the same even when the wording moves
• whether the same sources and pages are cited each time
• whether the tool is consistent about what it does not know, or whether it sometimes fills the gap
• whether calculations come out identically, with the same steps

A tool that gives you three differently worded answers pointing at the same pages is behaving properly. A tool that gives you three confident answers with three different deadlines is not fit for the work, whatever its other merits.

Build your process around sources, not sentences

The firms that find this a non-issue are the ones who never treated the prose as the output in the first place. The output is the set of sources, pages and figures you then check and sign off. Once the fee earner's own verification sits between the tool and the client, the fact that the tool phrases things differently on Tuesday is irrelevant: the file records what was checked, by whom, and against what.

That also makes supervision straightforward. A supervisor cannot sensibly review a conversation transcript, but can review a note that says which pages and provisions the conclusion rests on.

Where Alesis fits

Alesis answers questions about a matter from the matter's own papers and names the page each answer came from, and when it reads official sources it names the source for every point, so you can compare two answers by their citations rather than their wording. Where the papers do not say, it says so, and when it cannot find support for a point it says what is missing instead of guessing. For key dates and figures it shows the working, each step carrying the rule that allows it, and says the firm's diary system still governs. It prepares drafts for a qualified person to review and sign off; it assists qualified professionals and does not replace them.