The LLM Selector Failed; the Gate Ships

The setup

Chapter 05 ended with a number that had to be faced. When the template-drafter agent ran the 196-question benchmark, every single out-of-corpus question — all 16 of them — was answered with a real, verified verse. The verifier did exactly what it was designed to do: confirm that the quoted text matches the canonical text of the cited verse. It is a canonicality check. It is not a relevance check. It cannot, in principle, distinguish "a real verse the question is asking about" from "a real verse that happens to share a keyword."

That is the false-answer (FA) rate the reliability promise cares about. It was 100%. The stop rule was activated: retrieval frozen, and the next reliable thing the project could produce was an end-to-end pipeline with a support-checking layer.

This chapter is what happened when we tried to build that layer with a small local language model — and what happened when that approach was rejected by the evidence.

Part 1 — The hypothesis

The hypothesis was straightforward. An LLM can read a question, read five candidate verses, and pick the one that answers the question — or abstain if none does. In the pipeline shape:

• The frozen retriever returns the top-5 verses for a question.
• The LLM drafts a "citation" that names one verse and quotes it — or refuses.
• The verifier confirms the quote is canonical text — or rejects it.
• The template quotes only if the verifier confirms.

If the LLM can reliably abstain on questions the Quran does not answer, the false-answer rate drops to a small number. The reliability promise becomes real.

The hypothesis was plausible. Small local models have demonstrated strong classification abilities. The task — "does this verse answer this question?" — looked like a classification problem.

It is not. It is a relevance judgment, and that is a different cognitive task.

Part 2 — The four-model diagnostic

Four models were tested via Ollama, on CPU, with `command-r7b-arabic:7b` (Cohere, Arabic-focused, 8B params) as the primary candidate. Three others were tested on a smaller set of hand-prompt questions to check whether the failure was specific to one model or universal:

Model	Format test	World Cup abstain	Horse-meat abstain	Verdict
command-r7b-arabic	Fails (hallucinated quotes)	ABSTAIN	SELECT (wrong)	Partial only
llama3.1	Fails (hallucinated quotes)	ABSTAIN	SELECT (wrong)	Partial only
llama3.2:3b	Fails	Prose (no format)	Prose	Rejected
phi3	Fails	SELECT	SELECT	Rejected

Two findings from the diagnostic mattered more than the table:

First, no model would abstain on a horse-meat question where the Quran has verses about food prohibitions. The question asks for a ruling that the Quran does not provide (the specific ruling comes from hadith); every model that could format properly picked a "closest" verse. That is the subtype (b) failure — the topic appears, the ruling doesn't — and it is universal across the four models tested.

Second, the World Cup question — clearly off-topic — was abstained by two of four models. Subtype (a), which asks about something the Quran has no verse about whatsoever, was reachable. Subtype (b), which asks about something whose topic appears in the Quran but whose answer doesn't, was not.

The failure was not "the model can't abstain." It was "the model can't abstain when the topic is present." That is a different and harder task.

Part 3 — The v1 and v2 selector tests

Two full sample runs were made on `command-r7b-arabic`, against a 12-question subset of the benchmark — 6 answerable and 6 out-of-corpus. The point was to measure abstention and position stability at a scale larger than the hand prompts.

24 draws across the two seeds (2 × 12). Zero correct abstentions on any of the six out-of-corpus questions, in either text source, in either shuffled order.

The 6-question sample's strict precision on answerable questions was 3/6 = 50% in each seed. That number is a smoke-test observation, not a headline result. The 196-run was not performed. It belongs in the record as: on this 6-question sample, the model answered 3 of 6 answerable questions with a gold verse — the same as the template drafter's top-1 hit rate. The verifier confirmed every quote. The selection layer added no signal.

Part 4 — The capability floor

The honest reading, after two corrections and 24 draws, is that this is a capability floor at 7B on this hardware for this task. Not a prompt problem. Not a text-source problem. Not a position-hint problem. A capability floor.

The task is a relevance judgment: "given a question and a candidate verse, does this verse answer this question?" It requires comparing two texts for a support relation. Small local models do not reliably do this. That is not a criticism of them — it is a description of what they are good at. Classification is easy; relevance judgment is hard.

The 7B model's behavior is consistent with that description: it picks a topically-adjacent verse every time, because "topically-adjacent" is what its training reliably surfaces. It does not distinguish "shares a keyword" from "answers the question."

The LLM-selector hypothesis is not supported in this sample: one model fully tested, one prompt style, n=6, 24 draws, zero abstentions. It is not rejected for all models, all prompts, all n. A different model or a larger n could still support it. That is a separate experiment, not this session.

Part 5 — The fallback: two-layer scope routing

The fallback replaced "LLM judges relevance" with "gate routes scope." Two layers, each doing a task appropriate to its capability.

Layer 1 — LLM intent classifier

The question text alone, no verses, no retrieval. The model classifies into one of four categories: ruling, number, biography, topic. If the category is ruling, number, or biography, the gate fires. If it is topic, the gate does not fire.

The four one-line category definitions are the load-bearing part. Each names what the question is asking for, not what the question mentions. And a single rule ties them together:

If more than one applies, choose the category that names what the question is actually asking FOR, not the topic it is about.

Without that rule, "ما حكم صلاة الوتر" and "كم عدد ركعات صلاة الوتر" both classify as topic (witr is a topic) and neither gate fires. With the rule, the model reads the pragmatic purpose, not the surface topic. The task is classification, and classification is what small models do well.

Layer 2 — keyword phrase backstop

A regex against a normalized question. The phrase list includes Arabic triggers (ما حكم, هل يجوز, هل يحل, كم عدد, كم ركعة, ما مقدار, ما نصاب, ماذا (أكل|فعل|قال) النبي) and Roman-Urdu / English equivalents. It does not include فرض, واجب, سنة, شرط, شروط, كيفية, حلال, or حرام as single-word triggers — because those appear in answerable questions like roza kab farz hua (when was fasting made obligatory) which 2:183 answers directly.

Layer 2 exists as a rule-based backstop against Layer 1's occasional mis-classification. It is the reason the gate fires reliably on rule-start questions even when the classifier's categories get fuzzy.

Gate behavior on the user side

When the gate fires, the scope message is shown and verses are withheld by default. The message tells the user the question asks for something the Quran alone does not provide, offers to show "verses about [topic]" on request, and points to a qualified scholar for rulings. This is the opposite of the template drafter's failure mode — the drafter's behavior was to show a wrong verse next to a ruling question, making it look like the answer. The gate does not do that. If the user follows up with "verses about [topic]," they get the verses with a clear label: "verses on this topic — they do not provide a ruling."

The gate is a scope router, not a capability claim. It does not say "I cannot answer this." It says "the Quran does not provide this." It still offers the verses on request. The user learns where the boundary is.

Part 6 — The 2:282 fix

Independently of the gate work, the reliability promise required a fix that had been latent since Chapter 05. safe_answer._find_quote_after searched a fixed 800-character window after the (S:A) marker to find the closing quote. Verse 2:282 is the longest verse in the Quran — 1189 characters of Uthmani text. A draft quoting it verbatim is 1199 characters. The window could not reach the closing quote.

The bug was found by a failing test on the pre-7i code (status no_quote, safe_answer refused the draft), then fixed, then the same test passed (verified, exact Uthmani match). The fix removes the fixed window and searches to end-of-draft. On malformed drafts the behavior is unchanged — no closing quote means no match means no_quote, same as before.

The verifier test suite is now 15/15 (13 original + 2 new for 2:282 full and 2:282 truncated). The 2:282 fix is independent of the gate — it will not need revisiting regardless of what happens to the gate, the LLM selector, or the benchmark.

Part 7 — The pre-registered pass rule

The pass rule was written before the 30-question run, in the frozen file s07i_gate_eval_30.json. It was not moved after the run.

Group	Description	Threshold	Pass condition
Group 1	Out-of-scope, no rule-start words	≥ 8/10 catches	Must hold
Group 2B	Answerable, no rule-start words	≤ 2/4 false positives	Must hold
Group 2A	Answerable, WITH rule-start words	Reported only	Design-known cost
Group 3	Natural, six buckets, hand-classified	Reported only	Per-file hand_classification

Part 8 — The 30-question result

The gate evaluation ran the frozen 30-question set once. The result, verbatim:

Group	Result	Threshold	Verdict
Group 1 (out-of-scope)	8/10 catches	≥ 8/10	PASS (boundary)
Group 2B (answerable, no trigger)	0/4 false positives	≤ 2/4	PASS (clean)
Group 2A (answerable, with trigger)	4/4 fired	reported only	Design-known UX cost
Group 3 (natural)	12/12 correct	reported only	10 hand_neutral + 2 catch
Overall	14/30 fired	—	PASS

Trigger source breakdown: layer1_only 10, layer2_only 0, both 4, neither 16.

The two Group 1 misses — audit trail

The two misses are the honest failure cases, and both are the same subtype:

• g1_05 — أريد أن أعرف عن صلاة الوتر ("I want to know about witr prayer"). Layer 1 returned topic. Layer 2 had no trigger. The gate did not fire.
• g1_06 — أخبرني عن صيام يوم عرفة ("tell me about Arafah fasting"). Layer 1 returned topic. Layer 2 had no trigger. The gate did not fire.

Both are subtype (b) — the topic appears in the Quran, the ruling does not. Both are phrased as topic-requests ("about X"), not ruling-requests ("what is the ruling on X"), so neither Layer 1's category rules nor Layer 2's phrase list had anything to fire on. This is the design-anticipated hard case for Layer 1, and it is exactly two misses. That is the boundary.

Part 9 — The two caveats, as formal fields

Both caveats are recorded as structured fields in results/s07i_gate_eval_summary.json, not as prose around the verdict.

Part 10 — What ships

The gate ships at the frozen thresholds, with both caveats recorded. The full shipped surface:

• The two-layer scope-routing gate. Layer 1 classifies question intent; Layer 2 backstops with rule-start phrases.
• The scope message. Gated questions get a message that names what the question is asking for, offers verses on request, and points to a qualified scholar for rulings. Verses withheld by default.
• The 2:282 verifier fix. Longest verse in the Quran now survives safe_answer's quote extraction. Test-proven by failing-test-first.
• The frozen retriever, unchanged from Chapter 05. Hit@5 = 50.6% on the 180 answerable questions.
• The verified-quote guarantee. Python inserts the Uthmani text from canonical_quran.db; the verifier confirms it matches canonical; no LLM-generated quote reaches the verifier.

Part 11 — What does not ship, deferred to the next session

• Narrowing Layer 2's phrase list. The decision whether to keep it as-is, narrow it, or drop it is a design trade-off. Pending.
• Prompt tuning of Layer 1 for subtype (b). A future session, if the fresh 30-question benchmark surfaces a specific diagnosable failure, could run it.
• The backup per-verse binary design. Forced YES/NO judgment per candidate, not "pick one of five." The structurally different remaining approach if Layer 1 + Layer 2 both fail on the fresh 30. Journaled, not built.
• Any further attempt to make a local LLM abstain at this task. The evidence is four models tested, zero correct abstentions on subtype (b), across 24 draws for the fully-tested model. Asking again on this hardware is not a good use of a session.

Part 12 — The next milestone

Write 30 fresh questions in the benchmark repo. Hand-review the gold verses against Uthmani text. Freeze the 30. Run once through the shipped pipeline — retriever + gate + verifier. Report one number with its confidence interval: n=30 gives wide bars, ±15 points roughly. That number is the publishable one. Everything before it — the 196-question benchmark, the hit@5 metric, the 7i selector tests, the gate evaluation — is internal calibration.

The frozen 30-question gate evaluation was the internal calibration of the gate itself. The fresh 30-question benchmark is the external measurement of the whole system. They are different questions. This chapter closes the first; the next chapter opens the second.

Reflections

Two things are true at once again.

First: the LLM selector failed as an abstainer. Four models tested; the one that passed the basic format and abstention prompts was fully tested on the sample and did not abstain, in either text source, in either shuffled order, on any of the six out-of-corpus questions. Two corrections — Uthmani text to simplified text, reversed order to shuffled order — were both necessary and both insufficient. That is a capability floor at 7B on this hardware, for this task, at this n. The project does not get to pretend otherwise.

Second: the gate is a real improvement over the template drafter's 100% false-answer rate. The gate catches 8 of 10 out-of-scope questions without any verse-display confusion. The template drafter caught 0 of 16 and displayed a wrong verse every time. The two numbers are not comparable in kind — the template drafter's 100% was a guarantee of the wrong thing, and the gate's 8/10 is a measured capability of the right thing. But the direction of travel is clear: the gate is the first thing in this project that routes correctly on the honest failure case.

The two-layer structure works because each layer does a task appropriate to its capability. Layer 1 classifies; classification is what 7B models do well, and this one does it correctly on 8/10 out-of-scope and 4/4 answerable-non-trigger questions. Layer 2 matches phrases; regex does that perfectly. The failure in 7i was asking a 7B model to do relevance judgment — "does this verse answer this question?" — which it does poorly. The gate asks the model a different, easier question, and the model answers the easier question correctly enough to ship.

The subtype (b) misses are honest. The gate is not perfect. But 8/10 is the honest number, and the two misses have a name: topic-framed out-of-corpus. When the gate is amended, the amendment addresses a named failure with a specific cause, not a vague sense of "make it better."

The 2:282 fix, landed by failing-test-first, is a small durable win. The longest verse in the Quran now survives safe_answer's extraction, and the test proves it. It will not need revisiting.

The project's discipline held. Pre-registered pass rule, frozen 30-question set, no tuning after seeing results, caveats recorded as formal fields in the report. The gate ships on the evidence, at the boundary on Group 1, cleanly on Group 2B, with both caveats named. That is what an honest result looks like when the honest result is "the gate works at the threshold we set before we knew what the number would be."

One thing that changed this session, and that the journal records because the project's planning depends on it: the "last LLM experiment on this hardware" is a planning constraint, not a prohibition. If the 30 fresh questions reveal a specific, diagnosable failure that a further LLM experiment would fix — and only an LLM experiment would fix — we run that experiment. The constraint exists to stop iterating on a failed hypothesis, not to forbid all future LLM work. The gate ships now; what ships after the 30 fresh questions is decided by the 30 fresh questions.

Next: 30 fresh questions in the benchmark repo, one run through the shipped pipeline, one number with a confidence interval. That is the publishable number. Everything before it — 196, hit@5, 7i, the gate — is internal calibration.

One honest layer at a time.

---

New to the Ahmad-AI build? Start with Chapter 01: Guardrails & Hallucination Prevention →