Two blind agents and a tie-breaker: matching 4,887 Qur'an words to four translations

Context#

Al Quran shows a meaning under each Arabic word, taken from the translations the app already ships: Urdu by Junagarhi, Hindi by Ahsanul Kalam, English by Sahih International, and Roman Urdu by Abu Rayyan. Those translations are idiomatic, so each Arabic word has to be mapped to the translator's own words, and the map has to be exact enough that tapping a word lights up the right span in the verse.

Al-Fatihah was mapped by one agent and reviewed by a second. Doing the remaining 6,229 verses that way was out of the question. The owner's question, verbatim:

How do we automate for entire quran, getting it done manually is almost impossible?

The answer was a pipeline where agents do the matching, agents check the matching, and a person sees only what is left.

Constraints#

• A span must be verbatim text of the verse, start and end at word boundaries, and occur exactly once or carry its occurrence number. No character may belong to two Arabic words. Every word is mapped or carries a reason. A validator enforces all of this, and the app's data emitter refuses to run when any rule fails.
• The pipeline code calls no model. Python prepares task files, agents write answers, Python validates, compares, scores, and merges. Every stage is replayable from files on disk.
• The subject allows no quiet errors. A wrong meaning under a Qur'anic word is not a typo, so "mostly right" is not a result.

Method#

1. Prepare. For each surah and translation, a task file lists the Arabic words and the verse text.
2. Run A and run B. Two agents work the same file blind to each other. Run A sees hints from an existing word-meaning source; run B sees none, so the hints cannot quietly become the answer.
3. Compare and score. Where A and B agree, the match is accepted automatically. Where they differ, the word is flagged with both candidates.
4. Run C, the tie-breaker. A third agent sees only the flagged words, with both answers, and picks one or says neither.
5. Review. A reviewer agent receives every remaining meaning dispute plus a random sample of accepted matches, and writes verdicts and corrections as overrides.
6. Apply. Overrides are applied last and re-validated. Each word's final state is recorded: auto, resolved, reviewed, or corrected, with the previous answer kept beside any correction.

Two ranges were run: Juz 30 (surahs 78 to 114, short verses) and the second half of Al-Baqarah (2:142 to 252, the longest verses in the Qur'an), to see whether verse length changed the numbers.

Results#

How each word match was settled, as a share of all matches in the range.

Across both ranges: 4,887 Arabic words, 19,548 matches, 16,226 accepted without a review (83.0%), 362 words for review (1.85%). Long verses did not hurt agreement. The lower automatic share in Juz 2 came from conflicts between the hint and the translator's wording in Urdu, and the tie-breaker settled nearly all of those.

What the review found#

• Roman Urdu carried the largest review share: 97 of the 221 Juz 2 cards, 25 of them corrected. Its two runs often split a phrase differently from how the Urdu map split the same Urdu words. Following the Urdu decided almost every card.
• Recurring calls became rules. A correlative goes with its relative; a supplied copula stays unlit; a verb the translator repeats for a second clause stays unlit; a cleft goes with the fronted word. These went into a rules addendum that the next run starts from.
• The pipeline caught its own reporting bugs. Two tooling fixes were made on the way, each with a failing test first: a map header that named the wrong coverage, and a summary that called part of a surah by the whole surah's name.

What we would change#

• Match Roman Urdu after Urdu is final, with the reviewed Urdu map in the brief. That should remove most of the Roman Urdu review share.
• Treat hints as a trade-off. They raise agreement and they import the hint source's choices. The blind run B is what keeps that honest, and it stays.
• A machine reviewer is not the last word. The owner's review on the phone is still to come. On the Roman Urdu edition that step found two wrong words in 120 sampled verses that two vendors had passed, and it will be what turns these maps from checked into released.

Tooling#

One Python module runs the deterministic stages: prepare → validate → compare → score → merge → report, with tiebreak-prepare, tiebreak-validate, and resolve for run C, and apply-review for the overrides. The rules for how a word ends up auto, resolved, or review are pinned by tests. The final maps are tracked in git, one file per translation and surah, with a status file beside each that records how every word was settled.

$ python pipeline/word_alignment_auto.py report --resolved pilot-juz30
auto      84.8%
resolved  13.7%   (run C)
review     1.5%   (141 words)