The setup
Chapter 02 ended on a working system: 56,998 Islamic texts indexed, deployed, accessible through a Streamlit UI, sharing correctly between English, Arabic, and Urdu queries. The multilingual embeddings were working. The vector stores were clean. The system was live.
This is the chapter where I found out that "working" and "correct" are different things.
A closer read of the Arabic side — not did it return something, but how often did it return the right verse — produced a number that stopped the project cold: under 20% recall@5 on Arabic queries. And a closer read of why revealed that the Quran vector store, the one I had been describing as "multilingual," contained only English translations. There was no Arabic Quran text indexed at all.
The Arabic queries had been succeeding through cross-lingual luck — the multilingual model mapping Arabic tokens onto English vector space. It worked often enough to look plausible and failed often enough to be useless.
This log is what happened after that discovery. Two sessions of retrieval work that turned into two sessions of benchmark construction. Three public datasets rejected. Sixty-eight questions hand-labeled. And two counter-intuitive findings that changed how I think about cross-lingual retrieval for Arabic.
Part 1 — The forensic investigation
The first question was simple: how bad is it, and is any of it fixable?
I tested five Arabic retrieval approaches against the fifteen Arabic questions I had labeled at the time:
| Approach | Type | R@5 | Verdict |
|---|---|---|---|
| MiniLM on English corpus | Dense (cross-lingual) | 46.7% | Only because corpus is English |
| BGE-M3 on English corpus | Dense (multilingual) | 26.7% | Worse than MiniLM |
| ATM2 on Arabic corpus | Dense (Arabic STS) | 20.0% | Not a retrieval model |
| BM25 v3 (light stemmer) | Lexical | 40.0% | Best of the classic approaches |
| BM25 over camel-tools roots | Lexical (morphological) | 17.0% | Roots alone don't help |
Four of five failed. The one that "worked" (MiniLM at 46.7%) was only succeeding because the corpus was English and the queries happened to land near the right English verses in a cross-lingual vector space. That is not a retrieval strategy. That is luck.
The ceiling discovery
The bigger finding came from asking a different question: not "which retriever is best" but "what is the theoretical maximum any retriever in this family could achieve?"
For each Arabic query, I asked: does any of the verses we expect actually contain the query's root?
The answer was 64.4%. Not 100%. Thirty-eight of fifty-nine expected verses shared a lexical root with the query they were expected to answer. The rest needed semantic understanding — the verses were on-topic but used different words entirely.
Examples that broke every lexical method:
- التهجد (night prayer) → expected verses describe the act of praying at night without using the word for it
- الغيبة (backbiting) → expected verses are about mockery and slander, not the specific word for gossip
- يوم القيامة (day of judgment) → expected verses describe the events of that day without naming it
That 64.4% number killed the original target. The system couldn't reach 70% Arabic recall@5 with any root-based method — because 35% of the benchmark was unreachable by definition.
Always compute the ceiling of your benchmark before optimizing against it. Otherwise you optimize toward a target that is unreachable by construction, and every failure looks like a bug when it is actually a definition.
Part 2 — The pivot to evidence grounding
After the ceiling finding, I stopped trying to force retrieval accuracy and started asking what "reliable" actually meant for this product.
The answer was a three-tier model:
- Tier 1 — Citation verification. Is this quote actually in the Quran as claimed? Achievable at near-100% via deterministic string matching against a canonical corpus.
- Tier 2 — Free-text retrieval. Find the right verse for a user's natural question. Best academic results on Quranic retrieval hover around 0.3 MAP. This is a research problem, not a solved one.
- Tier 3 — Interpretive answers. What is the ruling? Even frontier LLMs reach only 65-68% correctness on fiqh-related questions, because there often isn't a single correct answer.
Reframing the promise as "99% reliable on citation grounding" rather than "99% accurate on Islamic answers" changed what we were building. And it made the release honest.
Part 3 — The dataset audit
To measure retrieval properly, I needed a benchmark. The instinct was to find one. So I searched.
I looked at every "Quran QA" dataset I could find on HuggingFace and in the academic literature. Six candidates. Three were rejected.
It looked like retrieval. It was reading comprehension.
Tarteel's QuranQA is derived from the Qur'an QA 2022/2023 shared task. It contains 238 test rows and 710 train rows of structured {question, passage, answer_span} triples. The format implies retrieval, but the semantics are the opposite: the passage is given, and the model must find the answer span within it.
The tell was that the same question appeared multiple times with different passages and different answers. When I merged each question's passage set into a single gold-verse range, some questions ballooned to 700+ verses. The "gold" was every passage the dataset had ever paired the question with.
Lesson: if a dataset was designed for task X, do not assume it supports task Y just because the field names overlap.
Nineteen million instruction pairs.
The repository Ereeeeef3/Qu-QA-v3 is titled as if it were the QuQA benchmark. It contains 19,210,080 rows across 34 parquet shards, totaling 16.6 GB, with the generic schema {input, output} and no verse references at all.
It is a fine-tuning corpus. For our purposes it was structurally irrelevant.
Lesson: check the schema, not the title. Names are aspirational. Fields are truth.
Scholarly passages, not verse-aligned tafsir.
Kandil7/Athar-Datasets is a large, well-structured Shamela-derived corpus: 15.8M passages across 8,425 books, categorized by aqeedah, fiqh, tafsir, seerah, and so on. It seemed ideal.
But the quran_tafsir split is not verse-by-verse tafsir. It is scholarly prose tagged as being about Quranic commentary, with no verse alignment. Reading it produced long discussions of methodology, author biographies, and general fiqh — anything except "here is tafsir for verse 2:255."
Lesson: investigate the grain of the data. The right category label is not the same as the right unit of alignment.
Part 4 — Building Ahmad-Eval v1
After three rejections I stopped looking and started building. There was no public verse-level Quranic retrieval benchmark with verified gold, and no sign there would be one soon.
Ahmad-Eval v1 is 68 questions across six buckets:
| Bucket | Questions | What it tests |
|---|---|---|
| Lexical Arabic | 15 | Query root appears in expected verses |
| Semantic Arabic | 4 | Query and expected verses share no roots |
| Multi-hop Arabic | 2 | Two themes in one query |
| Cross-lingual (EN) | 20 | English query, Arabic gold |
| Roman-Urdu | 20 | Urdu-in-Latin-script query, Arabic gold |
| Unanswerable | 7 | Questions the Quran does not address |
Every one of the 46 hand-labeled questions was verified verse-by-verse. For each theme — charity, patience, justice, gratitude — I searched the canonical corpus, filtered out false positives from polysemous roots, and picked only verses that genuinely matched the theme.
The unanswerable bucket is deliberately skewed toward hard cases. Four of the seven are modern fiqh questions that a model trained on Islamic text will be confident about but that the Quran does not address: cryptocurrency, smoking, the number of rak'ahs in Witr, rulings on horse meat. These test abstention, not retrieval.
Adversarial cases the labeling surfaced
Two passages in the Quran contain exact words that appear in our unanswerable questions, but not in the meaning the user is asking about:
- الوتر appears in
89:3("and by the even and the odd") — but that verse is an oath about cosmic duality, not about the Witr prayer. - الحصان (horse) shares the root ح-ص-ن with
حصن(fortress) andمحصنة(chaste). Eleven verses contain the root; none is about horse meat. - العملات (currencies) shares the root ع-م-ل with
عمل(work/deeds), which appears in 30+ verses.
These are exactly the cases a surface-level retriever will confuse. They are in the benchmark deliberately.
Part 5 — Two findings that changed the design
The reranker helps on 68 questions and hurts on 15.
In an earlier session I tested BGE-reranker-v2-m3 on 15 Arabic queries. It made things worse: R@1 dropped from 11.9% to 3.4%, R@5 from 25.4% to 16.9%. I concluded that generic cross-encoders don't work for Quranic Arabic.
Then I ran the same experiment on the full 68-question benchmark and the direction reversed: R@5 improved from 15.3% to 20.8%, and MRR improved from 0.200 to 0.286. The earlier conclusion was sample-size noise.
Lesson: fifteen queries is enough to pick a clear winner between two very different retrievers. It is not enough to evaluate a subtle improvement from a reranker.
For cross-lingual retrieval, translate the corpus, not the query.
The obvious cross-lingual approach is: translate the English query into Arabic, then search the Arabic corpus. We built that pipeline with NLLB-600M.
The alternative is: pre-translate the entire Arabic corpus into English once, index it with the English FTS5, and search with the English query directly. Zero translation noise at query time.
The results surprised me:
- Path A (NLLB query → Arabic): improved cross-lingual R@5 from 0.0% to 8.3%.
- Path B (English corpus search): improved cross-lingual R@5 from 0.0% to 21.2%.
Same benchmark. Same 20 questions. The corpus-side translation more than doubles the query-side translation. The reason, on reflection, is obvious: query translation has to be good for one very short string with no context, whereas corpus translation was done offline with full verse context and can be verified and edited. And English BM25 against a clean English translation is a much easier ranking problem than Arabic BM25 against Uthmani script.
Lesson: when two systems are both "translate then retrieve," the one that translates once offline usually wins over the one that translates at query time.
Part 6 — What the numbers actually say
Full results on the 68-question benchmark, best configuration (weighted RRF of BM25-root and islamqa-retriever-quran, reranked with BGE-reranker-v2-m3, English BM25 for the cross-lingual bucket):
| Bucket | R@5 (baseline) | R@5 (best config) |
|---|---|---|
| Lexical Arabic | 15.3% | 20.8% |
| Semantic Arabic | 0.0% | 6.2% |
| Multi-hop Arabic | 22.2% | 22.2% |
| Cross-lingual (EN→AR) | 0.0% | 21.2% |
| Roman-Urdu→AR | 0.0% | 1.1% |
| Unanswerable (abstention accuracy) | score-based abstention impossible | |
Two numbers in that table are worth their own sentences:
- Semantic Arabic is 6.2%. The queries where the answer doesn't share a word with the question — the ones only real semantic understanding can retrieve — are effectively unsolved. This is the open research problem.
- Unanswerable questions score higher than answerable ones on average. The score distributions overlap almost perfectly. This means we cannot use retrieval confidence as an abstention signal. Abstention must be verifier-based: if no retrieved verse can be verified as the answer, refuse, regardless of retriever confidence.
What I'd tell another engineer building an Islamic retrieval system
Measure the ceiling of your benchmark first. If 35% of your queries can only be answered semantically and you are testing a lexical retriever, you will spend weeks optimizing against an unreachable target. The ceiling number changes the entire strategy.
Do not trust public QA datasets to serve as retrieval benchmarks. The two tasks look similar and are structurally different. Check the schema. Check whether the same query appears with different gold answers. Check whether "no answer" means "no answer in this passage" or "no answer in the corpus."
Hand-label your benchmark. It is slower, but it forces you to confront what "correct" actually means for each query. Three of the seven unanswerable questions in Ahmad-Eval only became unambiguous after I checked the specific verse references. A pre-made dataset would have hidden that.
When a reranker improves one small sample and hurts another, run the bigger sample before publishing the conclusion. Small-sample reranker deltas are noise. Two-digit percentage swings on fifteen queries are common.
For cross-lingual retrieval, prefer translating the corpus over translating the query. The corpus is static and editable offline. The query is one-shot and unverifiable.
The Sadaqa plan
Ahmad-Eval v1 will be published on HuggingFace as a public dataset — the questions, the gold verses, the labeling workflow, the rejection notes for the datasets that didn't work, and the evaluation scripts. Under a permissive license, with full provenance per item.
Because when I needed a verse-level Quranic retrieval benchmark, there wasn't one. The next person who tries to build this system should not have to spend two days discovering that every public "Quran QA" dataset is doing something else.
Where it stands now
The Ahmad-AI build has moved past the semantic search chapter into an evaluation chapter. The pipeline is measurable. The numbers are honest. The remaining work — verifier-based abstention, semantic retrieval, hadith extension — has specific answers to specific questions rather than guesses.
Next: verifier-based abstention, a better semantic retriever for the 6.2% bucket, and eventual extension to a hadith retrieval benchmark.
One benchmark at a time.
New to the Ahmad-AI build? Start with Chapter 01: Guardrails & Hallucination Prevention →