Scaling Ahmad-Eval to 196 Hand-Reviewed Questions

The setup

Chapter 03 ended with a working Ahmad-Eval at 68 questions, six buckets, and an honest set of numbers. Cross-lingual R@5 was 21.2% via English BM25, semantic Arabic sat at 6.2%, Roman-Urdu was broken at 1.1%, and unanswerable detection had been proven impossible to do by score. Those were real findings and they changed the design.

But 68 questions is a diagnostic, not a benchmark. With 4 semantic questions, a single move shifts the bucket by 25 points. With 2 multi-hop questions, no metric means anything. The file was honest, but it was thin.

This is the log of what happened when we scaled it to 196 questions and then had to prove every single one of them. It turned out that finding candidates was easy. Verifying them was the work.

Part 1 — The pivot: generating 130 more questions

The obvious move was to expand every bucket roughly 3-5x: cross-lingual English from 20 to 60, lexical Arabic from 15 to 60, semantic Arabic from 4 to 40, Roman-Urdu from 20 to 40, unanswerable from 7 to 30. That target would put Ahmad-Eval in the same weight class as AyaTEC and the Qur'an QA 2023 shared task.

To generate them, we used a large language model as a drafting assistant. It proposed candidate questions and gold verses, bucket by bucket. In one sitting it produced roughly 130 new candidate questions across all six categories.

Each question was inserted through add_label.py, a validation tool that checks the refs against the canonical Quran DB, checks for duplicates, and assigns a numeric ID. Once the batch was in, validate_qids.py reported the file as structurally valid: no duplicate IDs, no missing refs, no format errors.

The file was now 199 questions. It was also, we realized shortly after, unverified. The validator checks form. It does not check whether the gold verse actually answers the question. Every one of the 130 new entries had been structurally validated and none had been semantically read.

Structural validation and semantic validation are different jobs. A file can pass every schema check and still be full of wrong answers. If your benchmark matters, someone has to read it.

Part 2 — The ID collision bug

Before the gold review could start, a smaller bug had to be fixed. When validate_qids.py was first run against the expanded file, it reported 25 duplicate IDs. Two questions in lexical_ar and 20 in roman_urdu shared IDs with earlier questions.

The cause was a small mistake in add_label.py:

n = sum(1 for q in data if q["bucket"] == bucket)
qid = f"ae_{bucket}_{n:03d}"

The ID was derived from the count of questions in a bucket, not from the highest existing number. If a bucket's numbering had gaps — for example, questions existed at 005-014 and 020-024 — the counter would eventually issue an ID that already existed. The file grew past the first gap and collisions started.

The fix had two parts. First, a one-time repair script renumbered the later duplicate of each collision to the next free number in the bucket. The original questions kept their IDs unchanged, so nothing downstream broke. Second, add_label.py was rewritten to derive the next ID from the maximum existing number in the bucket, with a collision guard as a second line of defense.

Part 3 — Reading every gold verse by hand

With the ID bug fixed, we did the thing that took the actual hours. Every question in the file was re-read, bucket by bucket, with all gold verses shown in the original Arabic, and a decision made per question: keep all, trim, or delete.

The method was deliberately manual. For each bucket we generated a review file:

python tools\show_bucket.py <bucket> --with-verses --out review_<bucket>.txt

The file listed every question with its query, its gold verses, and the Arabic text of each verse. We read them in Notepad, wrote a verdict under each question, and then applied the verdicts in one batch through a fix script with a backup. Each bucket was committed and pushed separately, so every batch is auditable in git history.

Across 199 questions, the review produced 5 corrections and 3 deletions. A 4% error rate. Small in absolute terms, and large in the specific sense that matters: all 8 of the wrong answers would have shipped. Any downstream system evaluated against them would have been scored against a wrong target.

The four patterns the review caught

By the end of the pass we could name what we were catching. These are the categories a structural validator cannot see, now documented in the benchmark repo's REVIEW_NOTES.md:

Part 4 — The benchmark gets its own repository

By the time the review was done, it was clear the benchmark needed to be separable from the retriever. Two different concerns, two different release cadences, two different audiences. And a real risk, not just a stylistic one: if the same working directory contains both the dataset and the code that measures it, a careless commit can entangle them.

The benchmark now lives at its own public repository:

https://github.com/emadaed/quran-retrieval-benchmark

The repository contains the 196 questions, the labeling tools, the review notes, the workflow documentation, per-source license notes, and the provenance file. It does not contain the retriever code. It does not contain the canonical Quran database.

What's in the repo

• data/ahmad_eval_v1.json — 196 questions, 6 buckets
• tools/add_label.py — commit a verified question (with the max+1 ID fix)
• tools/label_helper.py — search candidate verses by keyword or root
• tools/validate_qids.py — the structural validator
• tools/show_bucket.py — browse a bucket for review
• docs/HANDOFF.md, SCHEMA.md, WORKFLOW.md, PROVENANCE.md, REJECTED_SOURCES.md, REVIEW_NOTES.md
• LICENSES/ — per-source license notes (Tanzil, wahi)

The retriever repo points at it from PROJECT.md and consumes it as a read-only dependency. If someone clones only the benchmark, they get the benchmark. If someone clones only the retriever, they get the retriever. The two are not entangled.

Part 5 — The Roman-Urdu dictionary expansion

When we tried to re-run the evaluation against the cleaned benchmark, the eval crashed. The immediate cause was a small bug in the eval script: eight Roman-Urdu questions had produced empty translations, and the script tried to look up ranks for them anyway.

The real cause was deeper. The Roman-Urdu transliteration pipeline maps Roman-Urdu content words to either Arabic or English equivalents via a hand-curated dictionary. When the benchmark grew from 20 Roman-Urdu questions to 39, eight of the new questions used words that were not in the dictionary, and produced empty output. There was no error — the tool simply returned nothing for those queries, and the eval did not know how to handle it.

The fix had two parts. First, the dictionary was expanded with roughly sixty new entries to cover every missing word: shirk, chori, mitti, musa, firaun, kharch, hifazat, ashab, kahf, and others. All 39 Roman-Urdu queries now produce non-empty output in both Arabic and English modes. Second, the eval script was patched to fail gracefully when a query has no translation, instead of crashing after two hours of reranking.

Part 6 — The re-evaluation on cleaned gold

With the ID bug fixed, the gold reviewed, the benchmark split into its own repo, and the Roman-Urdu dictionary expanded, we re-ran the full evaluation on all 196 questions. The table below is the honest output. Sample sizes are included because they matter.

Bucket	N	R@1	R@5	R@10	MRR
Cross-lingual (EN→AR)	40	8.9%	23.4%	29.1%	0.450
Lexical Arabic	60	4.9%	19.1%	30.1%	0.295
Multi-hop Arabic	2	16.7%	33.3%	50.0%	0.571
Roman-Urdu→AR	39	1.3%	6.5%	10.4%	0.116
Semantic Arabic	39	2.2%	11.9%	17.0%	0.168

Compared to the 68-question baseline:

Bucket	R@5 (68q)	R@5 (196q, cleaned)	Delta
Cross-lingual	21.2%	23.4%	+2.2
Lexical Arabic	20.8%	19.1%	−1.7
Multi-hop Arabic	22.2%	33.3%	+11.1
Roman-Urdu	1.1%	6.5%	+5.4
Semantic Arabic	6.2%	11.9%	+5.7

Four things worth saying about this table:

• Cross-lingual held at 23.4% with a doubled sample. The English BM25 path is not an artifact of a small bucket. It wins on 40 questions the same way it won on 20.
• Lexical Arabic moved from 20.8% to 19.1%. The direction is a drop, and it is honest. The original 15 hand-labeled lexical questions were easier than the 45 new ones. 19.1% is the number on 60 questions; the older 20.8% was a small, favorable sample.
• Semantic Arabic nearly doubled without any change to the retriever. The only change was a cleaner gold set and 35 more questions. The original 6.2% was under-measured at n=4.
• Lexical Arabic has a reranking problem, not a retrieval problem. R@10 (30.1%) is substantially higher than R@5 (19.1%). The correct verse is usually found but not ranked high enough. That is a different fix than improving the retriever.

Part 7 — What three external reviews changed

We shared the review ledger with three frontier language models and asked them one question honestly: are there other Quranic retrieval benchmarks, and how does this one compare?

The answers converged. There are existing benchmarks — AyaTEC (207 questions, scholar-annotated, Arabic, includes unanswerable), IslamicFaithQA (EMNLP 2025, bilingual, abstention-focused), QRCD (extractive reading comprehension), QurSim (verse-to-verse similarity), the Qur'an QA 2023 shared task. But none combines verse-level gold with cross-lingual English queries, Roman-Urdu queries, and an explicit diagnostic bucket structure.

The reviews were also direct about what not to claim:

• Do not call it "gold" publicly. With one annotator, what is gold for a thematic question is exactly where scholars legitimately differ. Call it "human-reviewed v1."
• Get a second reviewer on a sample. Even 20–30 questions reviewed by someone else would strengthen the benchmark substantially.
• Report per-bucket, never a single blended number. Two questions cannot support a metric. Sixteen questions means one answer shifts the score by 6 points.
• Trimmed gold means valid alternatives count as misses. Report hit@k alongside recall and state the limitation.
• Freeze and version this file. Compute a hash, tag it v1.0, and preserve it. Future corrections get versioned, not silently applied.

The reframing this produced is the honest framing going forward: Ahmad-Eval v1 is a 196-question, human-reviewed Quranic retrieval benchmark with six diagnostic buckets, including Roman-Urdu and cross-lingual coverage that no existing public benchmark provides. It is designed to complement AyaTEC and IslamicEval, not replace them.

Where it stands now

The benchmark is public, hand-reviewed, split from the retriever, and committed at a frozen state. The retriever has honest numbers on the cleaned gold. The Roman-Urdu dictionary covers all current queries. The bugs found during this session — the ID collision, the empty-translation crash, the counting-not-identity mistake — are all documented in the journal and in the code.

There is real work left. Roman-Urdu retrieval is at 6.5% and the diagnosis (query density, not dictionary coverage) is now understood but not fixed. Semantic Arabic at 11.9% is better than before but still the hardest bucket. Lexical Arabic's reranking gap between R@5 and R@10 needs its own investigation. And the abstention policy for unanswerable questions is designed but not yet implemented — it needs a verifier, not a threshold.

But the benchmark itself is real. It has more questions than AyaTEC's original set. It has more buckets. It has more languages. And every gold verse in it has been read by a human against the actual Quranic text.

The next person who builds a Quranic retrieval system can measure against it, cite it, and argue with it. That is what a benchmark is for.

Next: Roman-Urdu retriever improvements, lexical-Arabic reranking, and verifier-based abstention for the unanswerable bucket. Then a freeze of v1.0 and publication on Hugging Face.

One benchmark at a time.

---

New to the Ahmad-AI build? Start with Chapter 01: Guardrails & Hallucination Prevention →