Finding the Bugs I Built Ahmad-AI to Prevent

The rule

Ahmad-AI is a local, offline AI agent I built on LangGraph and Ollama, with one non-negotiable design rule: it must never state a specific Quran or Hadith citation unless that citation is independently verified. Not "probably right." Not "sounds authoritative." Verified, or it says nothing.

That rule exists because general-purpose LLMs fabricate religious citations confidently and often, and for a system meant to be trustworthy on exactly this kind of question, "confidently wrong" is worse than "honestly uncertain." I designed the system around that principle from day one.

Then I spent several weeks trying to prove the system would break its own rule anyway.

It did. Repeatedly. Each time in a slightly different way than the last, which turned out to be the actual lesson.

What's actually under the hood

Ahmad-AI is a LangGraph state machine wrapped around a locally-running Llama 3 model (via Ollama), with three tools bound to it:

• get_current_time — trivial, but useful for testing whether the model reports tool output faithfully
• search_knowledge_base — a RAG pipeline backed by ChromaDB, over Ahmad-AI's own internal docs
• query_groweasy_db — read-only, live access to a real Postgres database behind GrowEasy, a supply chain/ERP system I also built

The interesting engineering isn't the tool-calling loop itself — that part is fairly standard. It's everything wrapped around it to stop the model from lying, fabricating, or quietly failing in ways a user wouldn't notice. That's what this log is actually about.

Round 1: the first adversarial pass

I ran a short, deliberately awkward set of test questions against the live system — empty database queries, a request for a Quran citation, a request for a Hadith reference, an ambiguous sort question. I expected a couple of rough edges. I got six real bugs, one of them serious enough to be the whole reason this system exists.

The one that mattered most: I asked which items were classified as "Class A" in a table with no matching rows. The agent correctly said "the query did not return any results." Then, in the same reply, it invented two fake item IDs with fake stock levels anyway — flatly contradicting the sentence it had just written.

The one that was actually dangerous: I asked what Islam says about riba (usury), with a source. The agent cited verse 5:51 — a real Quran verse number, attached to the wrong verse entirely. 5:51 concerns interfaith alliances; it has nothing to do with usury. I'd already built a citation guard specifically to catch fabricated citations. It never fired, because its regex only recognized Surah [name] X:Y formatting, and the model was writing (Quran X:Y). A safety mechanism that silently doesn't match the shape of the thing it's supposed to catch is functionally the same as not having one.

Four smaller bugs came with it: the time tool's return value sometimes never made it into the reply, a supplier-KPI "lowest score" query sorted in the wrong direction (masked by a small dataset), an ABC-classification query hit the wrong table on the first try, and a knowledge-base search fired on general religious questions it had no business touching.

The fix philosophy: code, not more instructions

My first instinct was to add stronger wording to the system prompt. I didn't, because a system prompt is a leash — it can be ignored, misread, or drift under pressure. Code can't. So instead of a stronger rule, I built a deterministic backstop that reads the actual tool output already sitting in conversation state and overrides the model's response when the shape of a known failure appears — regardless of what the model wrote:

def apply_tool_result_guards(state, response):
    """Deterministic backstop: if the last DB query came back empty or hard-
    errored, don't trust the model's prose about what it found. Override
    entirely with a fixed, honest message."""
    ...
    if _is_no_data(last_tool_message):
        return "No matching data was found for that query. The data may not " \
               "exist in GrowEasy yet, or try different filters."

And the citation guard became a whitelist, not a validity check — a distinction that turned out to matter a lot later:

_ALLOWED_CITATIONS = {"20:114"}  # the only pre-verified citation in the system

def enforce_citation_guard(text):
    for match in _CITATION_PATTERN.finditer(text):
        if not any(allowed in match.group(0) for allowed in _ALLOWED_CITATIONS):
            return SAFE_ISLAMIC_FALLBACK  # blocked, even if the citation is real
    return text

That last part is deliberate: even a citation that happens to be correct gets stripped if it isn't on the pre-verified list. No partial credit for guessing right. That decision looked overly cautious in isolation. It turned out to be the only thing standing between "safe" and "safe most of the time."

Round 2: the fix that wasn't running

I retested. The bugs looked gone. Then I looked closer and noticed the model's replies weren't using the exact fixed wording my code was supposed to produce — which is impossible for a hard-coded string unless the code generating it isn't actually running.

It wasn't. Twice. The file I thought I'd deployed had never actually replaced the one on disk — a browser download that silently didn't overwrite the original, and then a second near-miss after that. Running my own unit tests directly against the live file caught it immediately: AttributeError: module has no attribute 'apply_tool_result_guards'. The function I'd been testing against didn't exist in the file that was actually executing.

This turned into the most useful process change of the whole project. I added a build stamp, printed at startup, whose only job is to make "is this fix actually live" a non-question:

AGENT_BUILD = "2026-08-16.1-quran-quote-and-verse-citation-guards"

Every retest since starts by reading that line before trusting anything else in the output. It sounds almost too simple to mention. It's also the single change that stopped an entire category of wasted debugging cycles.

Round 3: a sneakier gap

With the fix confirmed live, adversarial testing kept finding new shapes of the same underlying problem:

• A hard database connection error (not just an empty result) hit a code path my guard didn't cover yet — closed by extending the same override logic to database errors, not just empty result sets.
• The model correctly identified the lowest-scoring supplier's value from a properly sorted query, then attached it to the wrong row's identity — two suppliers happened to share a display name, and the model quietly mixed fields between them. Fixed by having the query tool state the answer explicitly ("row 1, id=4, has the lowest value — don't combine this with fields from a different row") instead of handing over raw JSON and hoping the model read it correctly.
• Most instructively: the model quoted real, verbatim Hadith text — with zero citation number attached at all. My citation-number regex had nothing to match, because there was no number. It slipped through as a false pass in my own test suite. The project has zero verified Hadith text in its knowledge base, so a long quoted string sitting next to a scripture-related word is exactly as unverifiable as a fabricated citation number — the guard just hadn't been taught to look for that shape yet.

Building an evaluation harness that doesn't lie to you either

Manually retyping the same test prompts into a terminal doesn't scale, and it's exactly the kind of process that let the "fix wasn't running" bug hide for two rounds. So I built two layers of automated testing:

A unit-test suite that runs the guard functions directly against synthetic inputs — fast, no live model or database needed, currently 22 tests covering every bug found so far.

An end-to-end harness that drives the real compiled graph against the live Ollama model and live database, split deliberately into two kinds of checks:

• Hard checks — deterministic guard behavior. These must always pass. If one fails, it's a real regression, not noise.
• Review checks — LLM reasoning quality (e.g., "does it correctly identify the lowest-scoring supplier"). These get printed for a human to read, not force-fit into pass/fail. Pretending fuzzy model output is reliably pass/fail testable is how you end up with a flaky suite you learn to ignore.

That split is the part I'd defend hardest to another engineer. Treating everything as binary pass/fail either hides real regressions behind noisy false failures, or — worse — quietly downgrades your guardrail checks to "eh, close enough" the same way the model itself might.

Round 4: engineering for requirements, not just bugs

Once the fabrication bugs were closed, I went looking for gaps against what a production AI agent role actually asks for, rather than what my own testing happened to stumble onto:

• Bounded execution. LangGraph has a default recursion limit, but it fails by throwing an unhandled exception — not the same as an engineered decision. I added an explicit, much lower cap (MAX_TOOL_ROUNDS = 6) that degrades into a clear message instead of a crash.
• Retry logic, scoped honestly. A transient connection failure (container still starting, brief network blip) is worth one retry. A bad column name will fail identically every time — retrying it just wastes time and hides the real problem. The retry logic only fires on a connection-level error, deliberately excluding everything else.
• Observability. Every turn now logs tool-call count and latency, specifically because the deployment-mismatch bug in Round 2 was hard to diagnose without exactly this kind of visibility.

Round 5: the same bug, wearing a different coat

Weeks later, with everything green across two full test suites, one more adversarial pass against the live system produced this, verbatim, from a "what does Islam say about riba, with a source" prompt:

"The Quran explicitly states in Surah Al-Baqarah, verse 275..." — followed by several sentences of verbatim translated verse text.

A specific citation and a verbatim scripture quote, both showing up as a passing test. The content was actually accurate — 2:275 genuinely is the riba passage, and the quote was a real translation. That's exactly why it mattered: correctness on one run guarantees nothing about the next, which is the entire premise this project is built on. Two independent gaps let it through — my citation regex required a colon-separated format (2:275), and "verse 275" has no colon at all; and my quote-detection guard only checked for the word "Hadith" nearby, so a Quran-side verbatim quote was invisible to it. Both closed, both regression-tested. Nothing about this system's guardrails is ever really "done" — it's a set of specific, named gaps closed one at a time, with the humility to assume there's another shape out there I haven't tested for yet.

Where it stands now

22 unit tests. An end-to-end harness with 5 hard checks, all passing against the live model and live database as of the most recent run. Every bug listed above is closed and regression-tested against the real system, not just believed fixed. Known, accepted limitations are documented rather than hidden: the system can't yet answer questions that require joining across two tables in a single query (it fails safely rather than guessing), and a knowledge-base search occasionally fires on questions it doesn't need to — harmless, just an efficiency cost, and a low priority by design.

What I'd tell another engineer building something like this

A system prompt is a leash, not a lobotomy — deterministic code is the only thing that reliably holds under adversarial pressure.

A whitelist beats a correctness checker. Verifying "is this citation accurate" is a much harder problem than checking "is this citation on the pre-approved list." The second one is boring and it actually works.

Don't trust that a deploy happened — verify it. A version stamp printed at startup costs nothing and eliminates an entire category of wasted debugging time.

Test the test. A weak assertion produces a false pass just as dangerously as a missing guard does — my own harness let two real gaps through as green checkmarks before I caught them on close reading.

Distinguish transient failures from permanent ones before you decide to retry anything. Retrying a bad column name is just a slower way to fail.

Don't ask a language model to do arithmetic or entity-matching you can compute deterministically in code. Sorting the rows correctly didn't stop the model from misreading them — stating the answer explicitly did.

What's next

Two directions I'm actively considering: sourcing a properly verified Quran and Hadith corpus so the system can move from "safely decline every time" to "retrieve and correctly cite real, verified sources" — a meaningfully bigger architectural step than anything above, since it means teaching the guard to verify citations against retrieved content rather than a small fixed whitelist. And extending the database tool with real cross-table join support, so compound questions spanning classification and inventory data can be answered instead of safely refused.

Both are for later. For now, this is where the system actually stands — tested, not assumed.

---

Have questions about how any of this works, or want to see the code? Get in touch — I'm happy to walk through it.