Expertise
Evaluating AI-Generated Content
Verdict ledgers, second vendors, and samples you can defend
Overview
When agents draft content at scale (translations, transliterations, extracted records, documentation), the question is not whether it is good but how you know. Evaluation is the set of methods that answers that: deterministic lint as the first gate, independent model verdicts, hash-pinned ledgers, and human sampling with stated confidence.
Positioning
Evaluation has to survive change. A verdict pinned to a hash expires the moment the text changes, so nothing stays approved by accident. A sample with a stated error bound tells you what you actually know.
What this means in practice
For a complete Roman Urdu Qur'an we ran lint, then one verdict per verse from one vendor and a second blind verdict from another (Codex for 4,292 verses, Claude for 1,944), kept every verdict pinned to the verse hash, and had the owner read 120 random verses in two samples. The sample found two wrong words that both vendors had passed; both were fixed across the text. Observed 1.7%, 95% upper bound 5.2%, and the record says so (ADR 0006).
Why this matters
Without evaluation, AI-drafted content is either shipped on faith or reviewed line by line. With it, the human effort goes to disagreements and a defensible sample, and the record shows exactly what was checked.
What we deliver
A record of every verdict, tied to the exact text it approved
A second opinion from a different vendor
A sampling plan, and the confidence it buys
Knowledge paths
Related insights
Insight / Evaluating AI-Generated Content
Two blind agents and a tie-breaker: matching 4,887 Qur'an words to four translations
A multi-pass agent pipeline made 19,548 word matches, accepted 83% automatically, and left 362 for a reviewer agent and the owner's final read. What the review caught, and what changes next.
Insight / Evaluating AI-Generated Content
Verifying 6,236 AI-drafted verses with two vendors and a 120-verse sample
How a complete Roman Urdu Qur'an went from an agent's draft to a shipped edition: lint, two independent model verdicts pinned to the text, an owner's random sample that found two wrong words, and the sweeps that found nineteen more.