Where Chapter 06 left off
Chapter 06 ended with the gate passing its frozen 30-question internal test at 8/10 out-of-scope catches, 0/4 answerable false positives. The pass rule had been written before the run, frozen in the eval file, and not moved after the numbers were in. That was the internal calibration of the gate itself. It was not the publishable number.
The publishable number was always going to require a fresh set — questions the pipeline had never seen, contributed by someone other than the pipeline's designer, phrased in ways the pipeline was not tuned on. That's what this chapter is. It's also where the series ends: the benchmark was published on Hugging Face with both the passing numbers and the failing numbers, and the project closed with the honest result instead of the flattering one.
Part 1 — The 196 calibration (Session 7j)
Before the fresh set was run, the pipeline was run end-to-end on all 196 benchmark questions. These are the numbers it produced, all on questions whose phrasing the pipeline had been tuned against:
| Outcome | Answerable (n=180) | Out-of-scope (n=16) |
|---|---|---|
| Verified (gold citation confirmed) | 32 | — |
| Abstained (gate fired) | 33 | 16 |
| Wrong (cited off-gold verse) | 115 | 0 |
The pipeline ran deterministically across re-runs. Every draft the gate let through was verified against the canonical text and passed. Every out-of-scope question the gate fired on was abstained. The 196 numbers looked like a working pipeline.
They were calibration. They were produced on questions the pipeline was tuned against. They were not the number the project could publish as a claim about a system's behavior on questions it had never seen.
Calibration numbers are a system's behavior on questions it was tuned against. They measure what the system can do at all. They do not measure whether the system generalizes.
Part 2 — The 36 fresh questions (Session 7k)
Thirty-six fresh questions were contributed by an external model, hand-reviewed, and frozen in the benchmark before any run. They were distributed across the six buckets (6 questions per bucket), phrased in the full-prose register that the pipeline had not been tuned on. They were added to the benchmark repo, and the benchmark grew from 196 to 232 questions.
The pass rule was written before the run:
- (a) Out-of-scope false-answer rate must be ≤ 1/6.
- (b) The 95% Wilson lower bound on the verified rate on answerable questions must be ≥ 10%.
Both conditions were frozen in the scope message and the benchmark file before the run. Both must hold for the pipeline to pass.
The result:
| Condition | Observed | Threshold | Verdict |
|---|---|---|---|
| (a) Out-of-scope false-answer | 2/6 = 33.3% | ≤ 1/6 (16.7%) | FAIL |
| (b) Verified Wilson lower bound | 3.5% (3/30, CI95 3.5–25.6%) | ≥ 10% | FAIL |
Both conditions failed. Not marginally. The verified rate on answerable fresh questions was 10.0% — 3 out of 30 — and the honest reading of a 3/30 count is that the confidence interval's lower bound is 3.5%, well below the pre-registered threshold. The out-of-scope false-answer rate was 33.3%, twice the threshold.
The pipeline does not generalize to fresh questions at the pre-registered thresholds. That is a finding, not a failure of the run.
Part 3 — The failure analysis
The raw numbers on their own are not actionable. The failure-mode analysis (`s07k_analyze_failures.py`, read-only, no LLM re-run) broke the 36 questions into four categories. The dominant failure was not where we expected it.
Both subtype (b) topic-framed. Consistent with the 196's rate.
Two out-of-scope questions were classified as topic by Layer 1, passed the Layer 2 phrase check with no match, and the agent answered them with a canonical verse. The mechanism is the same subtype (b) topic-framed question that failed at 4/16 = 25% on the 196. On the fresh set it failed at 2/6 = 33%. Both rates are within one question. The mechanism is stable across the calibration and fresh sets. This is not a fresh-set-specific failure; it is the failure mode the gate was known to make.
The dominant failure. 57% on fresh vs 18% on calibration.
17 of 30 answerable fresh questions were gate-fired before retrieval. That is 57%, vs 33/180 = 18% on the 196. The fresh over-firing rate is three times the calibration rate. Every one was classified as ruling by Layer 1; 16 of 17 had no Layer 2 phrase match at all. The mechanism is Layer 1's classification prompt, not Layer 2. The prompt was tuned against the 196's more direct phrasing and does not generalize to the fresh set's prose-like phrasing.
The consequences are direct: 16 of the 17 gate-fired answerable questions never reached the retriever. Those three buckets contributed 0 verified, 0 wrong, 16 gate-fired abstentions — the pipeline never ran on them. Fixing Layer 1's over-firing would move 16 questions through the retriever. The number to move is 17, not 3.
Two retrieval failure modes. Secondary to the gate's over-firing.
Mode 1 — prose-query limits on multi-hop and semantic Arabic: the top-1 verse was topically adjacent but not the gold. Contributing cause: the 36 fresh Arabic-path questions had no islamqa query embedding (the frozen eval's query_embeddings.npz covers 196 qids only), so they ran BM25-root only without the RRF branch. This is a limitation of running the frozen retriever on questions not in the frozen eval.
Mode 2 — zero-match arbitrary top-1 on single-word Arabic lexical queries: the BM25 root index has no match for the query and returns an arbitrary order from a zero-score array. This is the same failure mode the Day 28 journal documented for "backbiting" → "backbit." It is a property of the frozen retriever, not a fresh-set-specific failure.
100% precision on drafts the gate let through.
The three verified questions were all cases where the gate was silent, the retriever returned the gold verse at top-1, and safe_answer verified the quote. 3 of 3 attempts that reached the retriever with a gold top-1 verified — that's 100% precision on the drafts the verifier received. The failure is not that the verifier is unreliable. The failure is that the gate didn't let the retriever run on enough questions for the verifier to have more drafts to verify.
Part 4 — What was published
The benchmark was published on Hugging Face at ahmad1979/ahmad-eval-v1 with 232 questions across the six buckets, under CC-BY-4.0. The dataset card documents:
- Composition — by bucket, by source (196 manual calibration, 36 synthetic fresh).
- Intended uses and out-of-scope uses. Out-of-scope includes "answering religious questions authoritatively" and "training a model on this data" — the benchmark is an evaluation set, not a training set, and not a fatwa service.
- Four known limitations: the gold sets are narrow (a valid verse not in
expected_versesis scored as wrong); the out-of-corpus bucket is out-of-corpus, not unanswerable-in-principle; the fresh questions' phrasing distribution differs from the calibration set; and cross-lingual translations are not perfect. - The observed failure mode. The card reports the 196 numbers and the 36-fresh numbers side by side, and it names the pass-rule FAIL explicitly. It says the dominant failure is Layer 1 over-firing on prose phrasings, and it points to
pipeline_results.mdfor the full analysis.
Every number in pipeline_results.md is produced by a script in the code repo. Every question in the benchmark is in ahmad_eval_v1.json. The pipeline that produced the numbers is documented in the four session journals (7h, 7i, 7j, 7k), each of which is in the code repo. Anyone who wants to reproduce the result or refute it has everything they need.
The benchmark is published with both the passing and the failing numbers. A benchmark that only documents the wins is a marketing artifact, not a benchmark.
Part 5 — What the project actually built
Across five chapters of work, the Ahmad-AI project built five layers. Each one does a specific job, and each one has a specific verdict.
Retrieval
Weighted RRF over BM25-root and islamqa embeddings, then BGE-reranker-v2-m3, with a per-bucket routing table. Hit@5 = 50.6% on the 196 answerable questions. The retriever works on questions whose phrasing is like the ones it was tuned on. On the fresh set it degrades — partly because of prose-query limits, partly because the fresh Arabic-path questions had no islamqa embedding. Frozen at the numbers reported in Chapter 05.
Verifier
Canonical-text match against the Uthmani source in canonical_quran.db. 15/15 test coverage including the 2:282 longest-verse case. The verifier confirms every quoted verse against the canonical text, and refuses to accept any quote it cannot confirm. It does not judge relevance — that was never its contract. Every draft the pipeline produced that reached the verifier was correctly verified or correctly refused. The verifier is the part of the system that does exactly what it claims.
Scope-routing gate
Two-layer gate — a 7B intent classifier and a keyword phrase backstop. Fires on ruling/number/biography classifications and on phrase matches. 8/10 out-of-scope catches on the frozen 30; 4/6 on the fresh 6. The dominant failure is Layer 1's classification prompt over-firing on prose phrasings (57% vs 18% on calibration). The fix is prompt-level and testable. It is not a structural limit of the design. The gate does the right thing when it fires. It fires too often on prose phrasings.
End-to-end pipeline
Wires retrieval, a template drafter, the verifier, and the gate together. Deterministic. Runs in seconds from a checkpoint. 32/180 verified on the 196 calibration. 3/30 verified on the fresh 36. Both numbers are true. The pipeline works on questions like the ones it was tuned on, and does not yet generalize to questions phrased differently. This is an honest first-deployment result for a rule-and-classifier pipeline, and it is more useful than a passing number would have been.
Benchmark
232 questions, six buckets, hand-reviewed gold, CC-BY-4.0, live on Hugging Face. The dataset card documents composition, intended uses, out-of-scope uses, four known limitations, and the reference pipeline's failure mode. Anyone using ahmad_eval_v1 as a baseline sees both the benchmark and the pipeline that produced it, with its limitations recorded honestly.
Part 6 — The four-tier promise, checked
Chapter 03 (Log 05) ended with a four-tier model of what a Quran retrieval system could promise. Five chapters later, here is the honest verdict on each tier.
Tier 1 — Citation verification
Near-100% via deterministic string matching against a canonical corpus. The verifier does exactly what it claims. It is proven by the 15/15 test suite, including the 2:282 fix that landed by failing-test-first.
Tier 2 — Free-text retrieval
Hit@5 = 50.6% on the 196. The retriever finds a gold verse in the top 5 for half of the answerable calibration questions. On the fresh set the behavior degrades — the fresh-question retrieval diagnostic shows the same directional pattern as the 196 (English-path buckets equal or better, Arabic-path mixed) but the numbers are within one hit per bucket at n=6. The retriever works on questions it was tuned on. Generalization to fresh phrasings is unproven.
Tier 3 — Interpretive answers
The system routes rulings to qualified scholars. It does not interpret. That was the design from Chapter 03 onward: the Quran alone does not provide many rulings, and the system is built for a Quran-only corpus. The scope-routing gate exists because Tier 3 is out of scope, not because it was attempted and failed.
Scope routing (added during the work)
8/10 out-of-scope catches on the frozen 30, 4/6 on the fresh 6. The dominant failure is Layer 1 over-firing on prose phrasings (57% vs 18% on calibration). The failure is specific, localized, and diagnosable. The fix is prompt-level and testable. The gate does the right thing when it fires. It fires too often on prose phrasings.
Part 7 — What remains, and what does not
The next thing — prompt reformulation, backup per-verse design, or the Hadith extension — is a new chapter, not a continuation. The project as defined in PROJECT.md is closed. All five conditions of the definition of done have been met. The benchmark is published. The reference pipeline's results are published, including the failure. The journals are complete. The series is closed.
That does not mean the system is finished in any absolute sense. It means the specific work the project set out to do — build a hand-reviewed benchmark, build a reference pipeline, measure it honestly, publish the honest result — has been done. What comes next is a different project: improving the pipeline's generalization, or extending the benchmark to hadith, or building a user-facing system on top of the benchmark. Those are new chapters with new pass rules and new fresh-question sets.
The one thing this project could not do honestly is claim more than it measured. It measured what it measured, and it published what it measured. That is the standard the project set for itself in Chapter 03, and it is the standard the project met.
Reflections
The series has one lesson that holds across all five chapters: measure the component before blaming it, measure the metric before trusting it, and measure the system against fresh questions before publishing a claim about what it does.
Chapter 05 discovered that the "working" system was returning the right answer under 20% of the time. Chapter 06 discovered that the reranker was helping most buckets and hurting one, and that the metric was measuring gold-set narrowness, not retriever quality. Chapter 07 discovered that the index was missing 68% of its own content. Chapter 08 discovered that the LLM-selector hypothesis failed at 7B on this hardware, and that the two-layer gate that replaced it worked at its own threshold. Chapter 09 discovered that the gate does not generalize to fresh prose phrasings, and that the honest thing to do is to publish the failure alongside the passing number.
Each chapter's finding is more useful than the chapter before it, because each chapter was built on the honest result of the previous one. The project does not have a single number it is proud of. It has a set of numbers it trusts, because each one was measured against a rule that was written before the run and not moved after it.
The reference pipeline does not generalize to fresh questions at the thresholds the project committed to. The project published the benchmark with that result recorded on its dataset card, and it closed the series with the failure as the finding. That is what an honest engineering log looks like when the honest engineering result is "the system works on the questions it was tuned on and does not yet work on the questions it was not."
The benchmark is public. The pipeline is documented. The failure is recorded. The series is closed. Whatever runs next — a prompt reformulation, a per-verse design, a hadith extension — starts from a fresh question set and a fresh pass rule, and inherits both the benchmark and the honesty of the chapters that built it.
Ahmad-Eval v1 on Hugging Face → · Code on GitHub →
The benchmark is published with both the passing and the failing numbers. A benchmark that only documents the wins is a marketing artifact, not a benchmark.
New to the Ahmad-AI build? Start with Chapter 01: Guardrails & Hallucination Prevention →