Names out of the index
How we anonymise rulings before indexing them.
[date] · [author] · 5 min read
A court ruling is read for its principle, not for the names of its parties. A lawyer looking for a ruling on agreed compensation needs what the court decided, not the claimant's name and address. So we take names out of rulings before they enter Bayan's library.
Why we do it
- Legal research has no need for people's identities.
- Privacy is a commitment before it is a legal requirement. The data protection law's Article 3 leaves some data relating to lawsuits outside its scope; we remove names whatever the reach of that exclusion.
- What goes into a system built on AI can come out of it in ways you do not expect, as described below.
What we remove
[Sherif confirms the list before publication.] The names of parties and witnesses, addresses, national ID numbers, phone numbers, car registration numbers, and any other detail that identifies a particular person. In place of a name we put the person's role in the case, such as claimant, defendant, appellant or respondent, and keep different people distinct within the same ruling.
Why Arabic names are hard
- Many names are ordinary words: «سعيد» (happy), «أمل» (hope), «كريم» (generous), «عادل» (just), «منصور» (victorious). A program cannot tell whether «سعيد» is a man or an adjective.
- Arabic has no capital letters. In English a capital letter signals a name; in Arabic nothing in the writing marks it.
- Compound and attached names: «عبد الرحمن», «أبو بكر», and «وسعيد», "and Said", written as one word.
- Rulings themselves differ: some write names in full, others use roles only.
Why a general model gets this wrong
There are two scientific problems. First, tools that recognise names learn from labelled text, mostly English, and rely on signals such as capital letters that Arabic does not have, so they make more mistakes on Arabic names that look like ordinary words. Second, and more serious: large language models can memorise rare strings they saw in training and repeat them word for word. In 2021, researchers showed they could extract memorised text from a published language model's training data, including the names, phone numbers and email addresses of real people. A name that goes into an AI system can come out in an answer that has nothing to do with its owner. So we take names out before any model sees them, not after.
How we do it
[Sherif confirms before publication.]
- Automatic detection of names and identifying details, tuned on Arabic court texts.
- Rules specific to the layout of Egyptian rulings, such as where parties are usually named.
- Human review of samples from every batch before indexing.
- No ruling is indexed, or seen by a model, until the names are removed.
Sources. Carlini et al., "Extracting Training Data from Large Language Models", USENIX Security 2021.