How we build Bayan

One root, many words

How Bayan searches Arabic legal text when one word has many spellings.

[date] · [author] · 6 min read

Search the Civil Code for «التعويض الاتفاقي», agreed compensation, letter by letter, and you may not find Article 224. The problem is not the law but the spelling: many published texts write the word «الاتفاقى», with a final ى instead of ي, and write «القاضى» rather than «القاضي», the judge. One letter is enough for an exact search to miss the most important article on the subject.

Arabic builds words from roots

An Arabic word is built from a root and a pattern. From the root «ع ق د» come «عقد», «تعاقد», «متعاقد», «معقود», «انعقاد» and «العاقد». Then letters and pronouns attach to the word: «وبعقده», "and by his contract", is written as one word. A search that matches letters as they stand either misses «وبعقده» or gathers everything with the same letters, without telling them apart.

And one word has many spellings

  • A final ي is often written as ى: «القاضى» and «القاضي».
  • The hamza changes its seat: «مسئولية» as in the Civil Code, «مسؤولية» as many people write it today.
  • A final ة can appear as ه in text copied by machine: «المادة» and «الماده».
  • Vowel marks appear sometimes and are mostly absent, and letters can be stretched for layout: «القانـون».

A judge reading sees the same word. A computer sees different strings of characters.

Why a general model gets this wrong

A language model does not read letters the way we do. It first cuts text into small units called tokens, then works with the tokens. In most general models the tokenizer was built mainly on English text, so it cuts Arabic into more pieces. Researchers at the University of Oxford found in 2023 that a tokenizer used by widely used models needs about three times as many tokens to write the same text in Arabic as in English. To such a model, «القاضى» and «القاضي» are two different strings of tokens, and it has seen one far more than the other. When it quotes a text, it leans toward the form it has seen more, so it may "correct" the article as it quotes it. The quotation reads smoothly and no longer matches the published text. That is the root of the problem: the model picks the most likely form, not the published one.

How Bayan searches

[Sherif confirms each step before publication.]

  • It normalises spelling when it indexes and when it searches: final ي and ى, the seats of the hamza, final ة, vowel marks and stretched letters. So it finds «التعويض الاتفاقي» in a text that writes «الاتفاقى».
  • It separates attached letters and takes each word back to its base, so it finds «وبعقده» when you search for «العقد», and it does not mix up near synonyms with different rules, such as rescission and dissolution, which is the subject of its own article.
  • It searches by meaning as well as by words, so it finds the article that governs your question even when its words differ from yours.
  • It always quotes the original text in its published form, never the normalised form it used to search. Normalising is a way to find the text, not to change it.

What this means for you

Write your question the way you normally write, with the hamza and the final letters you are used to. When you see an article in Bayan, it is as it was published, letter for letter.

Sources. Petrov et al., "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023.

Share

The month's key legal updates and what's new in Bayan, once a month.

Ask Bayan your first question.

Or book a team demo and bring a real contract: thirty minutes is enough to know whether Bayan fits your practice.

Book a team demo