How to check an AI answer before it goes into a memo
Five steps that take two minutes, from the article number to the date of the last amendment.
[date] · [author] · 6 min read
In June 2023 a federal court in New York fined two lawyers and their firm $5,000 because they filed a brief citing rulings that did not exist, written for them by a general AI program (Mata v. Avianca). The invented rulings were not clumsy; they were convincing, with party names, numbers and page references. That is the whole problem: a fluent answer is not evidence that it is right.
Before any AI answer goes into a memo, take five steps. They take two minutes if the source is in front of you, and much longer if it is not.
1. The number: is it the right article, in the right law?
Open the law and find the article by its number. Arab civil codes are close relatives, and the text that is Article 223 in the Egyptian code is Article 224 in the Syrian one. If the number is right and the law is wrong, the whole citation is wrong.
2. The words: does the article say what the answer says?
Read the whole article, every paragraph. Short answers drop qualifications. "The judge can reduce a penalty clause" is true, but Article 224(2) requires the debtor to prove the amount was grossly excessive or the obligation partly performed.
3. The date: is the text in force today, and on the date of the facts?
Laws change. Labour Law 14 of 2025 replaced Law 12 of 2003 from 1 September 2025, and Law 164 of 2025 changed the old rent rules from August 2025. Find the date of the last amendment and compare it with the date of the facts.
4. The ruling: does it exist, and does it say that?
If the answer cites a Court of Cassation ruling, check the appeal number, the judicial year and the hearing date, then read the principle in the ruling itself. One wrong digit means a different ruling, or none.
5. The leap: does the conclusion follow from the source?
Even if every source is right, ask whether the conclusion actually follows, or whether the answer joined an Egyptian text to a rule from another system.
Why each step exists: the science of the model
Each of the five steps answers one way a general model works:
- Numbers. The model writes the most likely next word. After "Article", every number in the code looks possible, so it picks the one most common in its training text, not the one that governs your question.
- Qualifications. After training, the model is tuned toward answers people prefer, and a short, firm answer is usually preferred to one full of conditions. The qualifications go first.
- Dates. The model's knowledge stops when its training data ends, and years of text about the old law outweigh months of text about the new one.
- Rulings. The format of a cassation citation repeats across thousands of pages, so the model learns it well. The numbers of any one ruling are rare, so it writes a correct format with invented numbers.
- The leap. The model learned from many legal systems at once, and when it does not know the Egyptian answer it fills the gap with the nearest one. OpenAI's researchers showed in 2025 that common ways of scoring models reward a guess more than "I don't know", so the model learns to always answer.
In Bayan
Bayan is built to make these steps fast. Every answer shows the article it rests on, with its number, its text and the law it belongs to, and [Sherif confirms: the date of its last amendment]; a ruling opens with its full citation. You check in seconds instead of starting the search again.
Sources. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) · Kalai et al., "Why Language Models Hallucinate", OpenAI, September 2025.