Verifying 6,236 AI-drafted verses with two vendors and a 120-verse sample

Context#

Al Quran ships a Roman Urdu rendering of the Junagarhi Urdu translation, so readers who speak Urdu but do not read its script can follow the meaning. A third-party Roman Urdu text was tested first and lost on every axis that matters: consonants rendered wrongly, nasalisation dropped, footnote markers fused into words in 309 verses. The decision in August 2026 was to ship our own Roman Urdu or none at all.

Our own text was drafted by an agent from the Urdu, verse by verse, under a written orthography. That left the question this case study is about. A careful read-aloud review of 6,236 verses by one person is 35 to 70 hours. The owner's ruling, verbatim:

It's next to impossible to review the entire Quran in Roman Urdu, I would need something automated.

The method that replaced the full human read is below, with its numbers, including the ones that went against it.

Constraints#

• The text must follow the Urdu word for word. A Roman rendering that reads naturally but drops a repeated word or swaps a particle is wrong, and the only way to see it is next to the Urdu.
• The orthography is fixed in writing and enforced by lint, so two reviewers cannot pass two spellings of the same word.
• A verdict is only valid for the exact text it approved. Any later edit must invalidate it.
• The public credit names the human author of the transliteration. The app shows no AI credit, and the method is published here instead.

Method#

1. Lint as the first gate. About a dozen deterministic rules: canonical spelling, parity with the Urdu, gloss parity, length outliers, forbidden forms, honorific typography, and the owner's rulings on specific forms. A verse that fails lint is never sent to a reviewer.
2. Verdict A. One review per verse by the first model, stored beside the text.
3. Verdict B, from a different vendor. A second review per verse with a different prompt and no access to verdict A. Codex (gpt-5.5) covered 92 surahs, 4,292 verses, before the account's usage limit. The owner chose Claude Sonnet for the remaining 22 surahs, 1,944 verses, with the same blind prompt. For those surahs both verdicts come from one vendor's models, so they are less independent, and the record says so.
4. Verified means both verdicts are ok for the exact current text, pinned by hash, and lint is clean. A disagreement from either verdict goes to the owner in chat, as an exception. Any later text change drops the verse back to pending.
5. An owner's random sample. About 150 verified verses, read by the owner. With zero errors the verified set's wrong-word rate would be bounded below about 2% at 95% confidence. If errors are found, the pattern is fixed across the whole text and a fresh sample is drawn.
6. Record. Owner-read verses are approved; machine-verified verses are verified with the reviewer recorded as "AI (double) + owner sample". Every row is hash-pinned.

Results#

The clean sample did not happen. The first draw of 60 verses held one wrong word (6:107, a noun for "guardian" rendered from the wrong root) and about eight spelling inconsistencies. The second draw of 60 held one more wrong word (9:76, "miserliness" rendered as the bare noun where the Urdu has the abstract form) and one spelling fix. Combined: 2 wrong words in 120 verified verses, 1.7% observed, with an exact-binomial 95% upper bound of 5.2%. That is weaker than the bound a clean sample would have given, and it is the number the record carries.

What the review found#

• The sample worked as a detector, not as a certificate. Both wrong words had passed both model verdicts. Each was fixed across the whole text, not only at the sampled verse: the first error also sat at 6:104, the second at 4:37 twice.
• The owner's first notes were themselves suspect. Several notes on sample 1 contradicted the Urdu and missed the sample's one wrong word; they appear to have been AI-assisted. The record keeps that observation, because a sample read with a model's help is not a human sample.
• Targeted sweeps found what random sampling cannot. A word-count comparison against the Urdu found 7 verified verses whose wording did not follow Junagarhi. A spelling-family sweep found 12 more, three of them changing the meaning: "comes out" for "brings out" twice at 10:31, "die" for "strike" at 8:50, and the same pair at 8:60. All 19 had passed both model verdicts. Each is fixed, owner-approved, and pinned by a test so it cannot return.
• The errors share a shape. A dropped nasal, a dropped repeated word, a swapped particle, a transitive verb read as its intransitive twin. Each reads naturally in Roman and is only visible beside the Urdu. Two model reviewers reading the Roman text alone share that blind spot, which is why independence between vendors did not catch them.

What we would change#

• Sweep before sampling. Corpus-wide pattern checks against the source text (word counts, verb pairs, spelling families) found 19 errors; the random sample found 2. The sweeps are cheap and deterministic and should run before any sample is drawn, so the sample measures what is left rather than what a script could have found.
• Give the second reviewer the source, not only the output. Verdict B read the Roman text. A verdict that compares each verse against its Urdu, word by word, targets the error class that actually occurred.
• State the bound you have. "Under 2% with 95% confidence" was the design target. The result was 1.7% observed and 5.2% as the upper bound, with a further 19 errors found outside the sample. Publishing the second set of numbers is what makes the method worth anything to the next team.

Outcome#

The owner dropped the "Experimental" label on 29 September 2026 on his own word, after the sample fixes and the sweeps. The exporter refuses any surah file whose status is not verified or approved, so an unreviewed file cannot ship unlabelled. The edition remains opt-in and a download. The review ledger, the verdict files from both vendors, the sample sheets, and the sweep findings are all in the repository, hash-pinned to the text they approved.