The Reranker Was Hurting, and the Metric Was Wrong

The setup

Chapter 04 ended with three things in place: a 196-question hand-reviewed benchmark, a corrected pipeline running against a clean index, and a list of open problems. The most visible open problem was Roman-Urdu, still stuck near the floor at 7.1% R@5. The second was lexical Arabic, at 19.1% R@5 with an R@10 of 32.5% — a 13-point gap that suggested the right verses were being retrieved but not ranked high enough. The Session 7e write-up called it a reranker problem.

Both of those labels turned out to be wrong. Roman-Urdu's problem was not the retriever. Lexical Arabic's problem was not the reranker. And the diagnostic that revealed both was one that had been built for a completely different purpose.

Part 1 — The diagnostic that found the wrong thing

The Roman-Urdu pipeline in Chapter 04 was being extended with a semantic layer: Roman-Urdu transliterated to Urdu script, translated to Arabic by NLLB-200-distilled-600M, then fed into the standard Arabic retrieval pipeline. Before spending 100 minutes on the eval, a cheap gate was run: for each of 39 Roman-Urdu queries, does the NLLB Arabic output share at least one root with any of the gold verses? If not, retrieval is structurally impossible, and the eval will not help.

The gate returned 34 of 39. That was close to the threshold. Two of the five misses looked wrong, and one of them stood out:

qid 057  quran ki hifazat ka wada
gold 15:9  إِنَّا نَحْنُ نَزَّلْنَا الذِّكْرَ وَإِنَّا لَهُ لَحَافِظُونَ
  "Indeed, it is We who sent down the Quran, and indeed, We will be its guardian."

overlap check said 15:9 had 3 roots and did not include حفظ.
حفظ is the root of لحافظون ("guardian") — the exact word the
query is about.

The hypothesis was that the query extraction was wrong. Running the same extractor on the same verse in two forms produced the answer:

extract_roots("إِنَّا نَحْنُ نَزَّلْنَا الذِّكْرَ وَإِنَّا لَهُ لَحَافِظُونَ")
  -> ['_نن', 'ذكر', 'ن_ح', 'نزل']              # حفظ missing

extract_roots("إنا نحن نزلنا الذكر وإنا له لحافظون")
  -> ['_نن', 'حفظ', 'ذكر', 'ن_ح', 'نزل']      # حفظ present

Same verse, same analyzer. Different input. The Uthmani form (with full Quranic diacritics: superscript alef, small high seen, shadda combinations) tokenizes differently than the simplified form, and camel-tools returns different roots. The index had been built from Uthmani. The query path runs on simplified. The two sides of the system spoke different dialects.

The measurement was not subtle. 11,156 root occurrences were missing across the corpus. The verse 56:78 — فِي كِتَابٍ مَّكْنُونٍ, "in a well-guarded Book" — returned zero roots from Uthmani extraction, versus four from simplified. Whole verses had been invisible to the retriever for the entire project.

Part 2 — Rebuilding the index

The fix was to rebuild data/quran_roots.json from data/quran_simple_clean.json, so index-time and query-time extraction used the same input form. The old index was preserved as quran_roots.json.uthmani_bak and not deleted. A rebuild script (scripts/rebuild_root_index.py), a per-word trace tool (scripts/review_added_roots.py), and updated audits (measure_uthmani_vs_simple.py, audit_root_index.py) were added.

One complication: simplified extraction is not a pure copy of Uthmani extraction minus noise. It introduces a small set of spurious camel-tools stems from function words and demonstratives. The 11 that were curated and filtered:

كذلك فذلك لذلك بذلك بذلكم    (demonstrative ذلك artifacts, 123 verses)
ذلل بذل                       (demonstrative ذلك variants, 474 + 5 verses)
علنّة                         (shadda rendering artifact)
كنه لكن خصة                   (function-word fragments)

The two-phase curation mattered. The first pass identified the obvious demonstrative artifacts. The second pass — after printing the per-word trace for all 159 added roots — caught ذلل (474 verses, traced to ذلك variants) and بذل (traced to بذلك, and ب-ذ-ل is not a Quranic root). Both were invisible in the first pass because their example word in the compact table was a demonstrative that looked like noise, not a content word. This is exactly what the review tool was built to catch.

Verification after the 11-entry filter: 0 hidden roots, 100% index-versus-extraction match, zero orphans, zero asymmetries.

Part 3 — The corrected baseline, honestly read

The eval was re-run on the corrected index. The result was not what the size of the bug predicted.

Bucket	Before	After	Delta
cross_lingual_en	26.6%	26.6%	0.0
lexical_ar	19.1%	19.1%	0.0
multi_hop_ar	33.3%	33.3%	0.0
roman_urdu	7.1%	7.1%	0.0
semantic_ar	11.9%	16.3%	+4.4

Only semantic_ar moved. +4.4 points, on 39 questions, is six additional gold verses retrieved into the top-5 across the bucket. That is real. But it is not the 68%-of-corpus number suggested.

The explanation is that the missing roots were the rare ones. Queries whose gold verses use high-frequency roots — prayer, charity, patience — were retrievable before the fix, because those roots were already in the index. Queries whose gold verses depend on rarer roots were the ones that gained. That is what the bucket names suggested, but it took the measurement to confirm.

The index asymmetry was a real bug and is now fixed. It was not the bottleneck for the hardest buckets, and it was only the bottleneck for one. Both statements belong in the record.

Part 4 — The reranker, measured instead of assumed

Chapter 04's write-up had blamed the reranker for lexical_ar's R@5 / R@10 gap: 19.1% vs 32.1%, a 13-point spread. The suggestion was to swap the reranker for something better, or widen the rerank pool, or delete it outright based on a Session 3 experiment where the reranker had hurt on 15 questions.

None of those steps were taken, because none of them were justified by a measurement. The last thing before touching the reranker was a diagnostic: score the pre-rerank order with the same metrics as the current eval, and see whether the reranker is helping or hurting on 196 questions.

Two additive report sections were added to the eval: one comparing pre-rerank (ar_ranks) against post-rerank (ar_rerank) on the Arabic path, and one doing the same on the English path. No new logic, no bucket rules, same compute_metrics(). The reranker was untouched.

The measurement when it came:

Bucket (Arabic path)	no-R@5	R@5	Delta
cross_lingual_en (A)	8.2%	15.8%	+7.6
lexical_ar	16.3%	19.1%	+2.8
multi_hop_ar	33.3%	33.3%	0.0
roman_urdu (A)	8.4%	7.1%	-1.3
semantic_ar	10.4%	17.0%	+6.6

The reranker helps on the Arabic path. Not marginally — meaningfully, on the same buckets the write-up had blamed it for failing on. lexical_ar's 13-point R@5 / R@10 gap persists with and without the reranker, which means it is a property of the retrieval pool, not the rerank stage. The reranker was not the lexical_ar bottleneck.

Then the English path:

Bucket (English path)	no-R@5	R@5	Delta
cross_lingual_en (B)	20.9%	26.6%	+5.7
roman_urdu (B)	17.5%	5.8%	-11.7

The reranker helps cross_lingual_en path B by 5.7 points and destroys roman_urdu path B by 11.7 points. And 17.5% — the no-reranker score on roman_urdu path B — is the highest roman_urdu number ever measured, on any path, including the s06d keyword path (7.1%) and the NLLB semantic pipeline (4.5%).

Part 5 — Diagnosing the -11.7 before deleting the reranker

Deleting a component that produced a 12-point regression in one path and a 5.7-point gain in another was not obviously correct. The next step was to look at what the reranker was actually being asked to score.

scripts/diagnose_reranker_inputs.py reads the checkpoint (no compute) and prints, for three Roman-Urdu qids, the query text, the gold verses, the pre-rerank top-5 and post-rerank top-5 with Arabic and English text, and the reranker score for each candidate.

Part 6 — The routing fix

The Session 6/7 auto-selector had been choosing a path per bucket by whichever path produced the higher R@5. That selector was structurally unable to find the better Roman-Urdu answer, because it compared path B's reranked score (5.8%) against path A's (7.1%), and never considered path B's pre-rerank score (17.5%).

The fix was to replace the auto-selector with an explicit per-bucket routing table:

cross_lingual_en   en [rerank]
lexical_ar         ar [rerank]
multi_hop_ar       ar [rerank]
roman_urdu         en [no_rerank]   <-- new
semantic_ar        ar [rerank]

Any bucket not in the table falls back to the auto-selector and prints a warning, so future buckets require no code change. An invariant check was added: after routing, all non-roman_urdu buckets must match the Session 7f baseline within 2 points R@5. A routing bug cannot silently pass.

Result, from --from-checkpoint (no new compute, ~5 seconds):

Bucket	Before	After	Delta
cross_lingual_en	26.6%	26.6%	0.0
lexical_ar	19.1%	19.1%	0.0
multi_hop_ar	33.3%	33.3%	0.0
roman_urdu	7.1%	17.5%	+10.4
semantic_ar	17.0%	17.0%	0.0

Roman-Urdu went from 7.1% to 17.5% by removing a step, not adding one. MRR went from 0.110 to 0.373. R@10 went from 13.0% to 24.7%. The reranker on path B had been leaving 11.7 points of R@5 on the floor since Session 5, invisible because the auto-selector never picked that combination.

Part 7 — The noise floor

The reranker is deterministic on these bucket sizes. Across the last three eval runs, the English path was byte-identical, and the Arabic path was identical in the last two (differing by one hit on lexical_ar R@10 in an earlier run). Noise is ≤1 hit per bucket at n=39 and n=60. Any change of 1 hit or less is not a real finding.

The 11.7-point drop on roman_urdu path B is 12+ hits above noise. It was real. The +4.4 on semantic_ar from the index fix is 6 hits above noise. Also real.

Part 8 — Reading 15 failures by hand

This was the highest-value hour of the session.

Fifteen questions were selected where zero gold verses appeared in the top-5 of their routed path — 10 from lexical_ar (Arabic path) and 5 from roman_urdu (English path, no rerank). For each, the query, the gold verses, and the top-5 returned were printed with Arabic and English text. Each returned verse was marked R (clearly relevant), P (plausibly relevant), or W (wrong) by a human reviewer.

Result: 13 of 15 questions (87%) had at least one R or P in the top-5. By the decision rule (>50% → gold sets too narrow, R@5 under-counts), the finding is clear.

Per-bucket:

• lexical_ar (10 sampled failures): 10/10 had at least one R or P. 32 R, 11 P, 7 W out of 50 returned verses.
• roman_urdu (5 sampled failures): 3/5 had at least one R or P. 4 R, 4 P, 17 W out of 25 returned verses.

The clearest example is lexical_ar_012 (الإحسان). The retriever returned 37:110, 31:3, 39:34, 92:6, 55:60 — every one of them a verse containing محسنين or الإحسان, every one of them a legitimate answer to "tell me about الإحسان." None of them was in the benchmark's specific gold set (2:195, 3:134, 4:125, 7:56, 16:90), also on the topic. R@5 counts the result as a miss. It was not a miss. It was a different set of valid answers.

The lexical_ar R@5 of 19.1% is not a retrieval-quality number. It is a gold-set-specificity number. The bucket is not broken; the metric is measuring something narrower than what we want to know.

The checklist recommended reporting hit@5 alongside R@5. That number is not computed across all 60 lexical_ar questions in this session — the manual read is the relevance evidence, and a systematic hit@5 metric belongs with the next session's agent work. The honest position is: the manual read establishes that relevance is far higher than 19.1% on lexical_ar, and the exact metric is deferred.

Part 9 — Three real failures

Not all of the sample was gold-set narrowness. Three questions identified genuine failures with identifiable root causes:

• lexical_ar_008 (بر الوالدين, kindness to parents): retriever returned four of five verses about fathers in passing (rhetorical questions about ancestral disbelief) or parent-child ties as metaphor. It matched on the root والد / أب / أم without matching on the meaning (the obligation of بر الوالدين as a command). A genuine "matched on root, missed on meaning" failure.
• roman_urdu_024 (ghibat kya hai): English translation is "backbiting", light stemmer strips -ing producing "backbit". The Yusuf Ali English corpus contains neither "backbiting" nor "backbit" (the gold verse 49:12 uses "behind their backs"; 104:1 uses "backbiter," which -er protection keeps intact). BM25 finds zero matches and returns an arbitrary order from an all-zero score array.
• roman_urdu_025 (dua ki qabooliyat): English translation is "supplication supplicate acceptance accept". BM25 matches the literal words "accepted" / "supplication" in verses where they are used in a different sense. The gold verses use استجابة / إجابة, which Yusuf Ali renders as "answer," not "accept."

These are noted as findings. They are not on the backlog and not scheduled. The stop rule applies to them too.

Part 10 — The stop rule

Session 7e wrote the rule: "after these four steps, freeze retrieval until the verifier is wired into the agent end-to-end."

Step 1 found a real bug (reranker on roman_urdu path B) and fixed it. Step 2 measured the noise (≤1 hit). Step 3 found the metric is wrong for lexical_ar and mixed for roman_urdu, with three identifiable exceptions. Step 4 (GPU) is deferred indefinitely — with checkpointing, report-only runs are seconds, and no further eval changes are needed.

Retrieval is frozen. The final numbers:

Bucket	n	R@5	Routing
cross_lingual_en	40	26.6%	en [rerank]
lexical_ar	60	19.1%	ar [rerank] (gold-set narrow)
multi_hop_ar	2	33.3%	ar [rerank] (n=2)
roman_urdu	39	17.5%	en [no_rerank]
semantic_ar	39	17.0%	ar [rerank]

These are the numbers the agent inherits. If the agent performs well despite these numbers, the retriever is useful enough for the agent's purpose. If the agent performs poorly, the cause will be identifiable from the agent-side logs (verifier false negatives, retrieval misses, query routing bugs), not from R@5.

What we did not do

• No GATE-Reranker-V1 swap. Would have been attempted if the reranker had hurt across the board. It helps on the Arabic path and on cross_lingual_en path B. The measurement said no.
• No bge-reranker-v2-gemma download. Same reason.
• No stemmer fix for roman_urdu_024. Identified; not on the backlog.
• No query rewrite for roman_urdu_025. Same.
• No fix for lexical_ar_008. Same.
• No retriever training. The retrieval pool is not the failure mode on lexical_ar; the metric is.
• No GPU migration. Deferred as part of the stop rule. If the agent work reveals a need for one more retrieval experiment, that is the trigger. Until then, CPU plus checkpoint is enough.

What's next

Wire the verifier into the agent. Run the 196 questions end-to-end. Count three things:

• Verified answers: the agent returned an answer with a citation the verifier confirmed supports the claim.
• Abstentions: the agent refused to answer because the verifier could not confirm support.
• Wrong answers: the agent returned an answer with a citation the verifier confirmed does not support the claim (the failure the verifier exists to catch).

That count — three numbers — is the reliability promise. Not R@5, not relevance@5, not hit@5. The number of questions the agent gets right with a verified citation, out of 196, is the number that matters.

The verifier was built and tested in Chapter 03 (13/13 on its own test set). It has not been wired into the agent. That is the next milestone.

Reflections

Three things are true at once.

First: the session found a real bug (the reranker on the Roman-Urdu English path) and fixed it. Roman-Urdu went from 7.1% to 17.5%, the first real improvement to that bucket since the benchmark existed. That is a win, and it came from removing a step, not adding one.

Second: the index asymmetry was a real bug, correctly fixed, and it moved only one bucket. The temptation was to expect the 68%-of-corpus number to produce headline movement. It did not, because the missing roots were the rare ones. The honest reading is narrower than the size of the bug suggested.

Third: the lexical_ar R@5 of 19.1% is not a retriever-quality number. It is a gold-set-specificity number. The bucket is not broken; the metric measures something narrower than what we want to know. That finding changes how the bucket should be reported going forward — and it is the reason the retrieval work stops here rather than chasing a number that was measuring the wrong thing.

The lesson from Chapter 04 holds and gets sharper here: measure the component before blaming it, and measure the metric before trusting it. The reranker was blamed for lexical_ar's gap and turned out to be helping it. The 19.1% number was trusted as a retriever-quality measure and turned out to be a gold-set measure. Both corrections came from looking at the actual failures and the actual pre-rerank scores, not from swapping models or widening pools.

Retrieval has been the focus for seven sessions. The verifier is the piece that turns a retriever into an agent. The next reliable thing this project can produce is not a better R@5. It is an end-to-end agent that verifies every citation and abstains when it cannot. That is the milestone worth chasing next.

Next: the verifier wired into the agent, 196 questions end-to-end, three numbers — verified, abstained, wrong. That is the reliability promise. Not R@5.

One retrieval-metric at a time, and then the agent.

---

New to the Ahmad-AI build? Start with Chapter 01: Guardrails & Hallucination Prevention →