The entries
01
The summary passed my checker and still contained an unverified claim
ToolA language model wrote the weekly funnel summary; a rule-based checker verified every number against the database before it was sent.
What it gaveA summary that passed verification (attempt 1: passed). Every number was filled in from the database — the model wrote none. But one sentence said "$968 and $699 being the largest contributors" — a ranking claim that no code had computed.
How I noticedThe checker's checked-word list did not include
largest. The pass was conditional on the checker's blind spots — I read the output against the check and saw the word that slipped through.What I did(1) Added
largest, smallest, biggest, leading, dominant, majority, contributors to the checked words. (2) Gave the model a legitimate route: new code-computed facts rank.top_spend_channel and rank.top_spend. (3) Confirmed the exact sentence from the live run is now rejected.The checker guards only the words it knows. A live run against real output is what catches the blind spots.
02
The forecast model beat a naive baseline on pair count and lost 3× on the metric that matters
ToolETS with additive trend and seasonality, selected per SKU via expanding-window backtesting.
What it gaveA forecast that beat the naive baseline on 58% of pairs — but lost 3× on volume-weighted WAPE (178.7% vs 65.5%). On the top SKU it over-forecast by 155%, because ETS extrapolated a lumpy B2B order spike as trend.
How I noticedA 12-month holdout scored with volume-weighted WAPE, not just pair-count win rate. The flattering number (58%) and the honest number (3× worse on the SKUs that carry the business) diverged, and the honest one was the decision-relevant one.
What I didSurfaced the failure mode before anyone acted on a number. The app's reliability layer flagged 84 of 92 pairs Low; v2 adds a hard-coded trust rule that refuses to auto-dispatch an order on any flagged SKU, and reports
recommended_order_qty_if_validated as the number the analyst takes into the customer call.Honest metrics beat flattering metrics. Report the volume-weighted number, unprompted.
03
The validator scored 14.8% precision — 925 false alarms — and I proved it after the fix
ToolA 12-rule Python validator over synthetic healthcare claims, scored against a ground-truth answer key the generator had logged.
What it gave14.8% precision on the hard test set — 925 false alarms and 21 real errors missed. 314 valid claims flagged "missing service date" because of US date format; 144 good payments flagged "orphan" because of stray spaces or case in the claim ID.
How I noticedScored against the ground-truth manifest with precision and recall, not eyeballed. The easy set had scored 100%/100% — which was circular, because the rules had been written to match the errors I injected. The hard set broke that circle.
What I didAdded a cleaning layer (trim, case, two date formats, currency text) that logs what it corrected and never counts it as a failure, plus a twelfth rule for near-duplicate billing. Result: 182 of 182 found, zero false alarms, 27 tests. Each legitimate look-alike has its own unit test. Parity confirmed across Python, SQLite and Azure SQL.
Score against a ground-truth answer key, not against your own expectation. Easy-set 100% can be circular.
04
The retry setting I trusted did not rescue the lead I lost
Tooln8n webhook workflow calling a strict Flask API. The HTTP node had "Retry On Fail" switched on.
What it gaveWith the API stopped, a test lead was lost. The lead existed only in n8n's execution history. "Retry On Fail" fired no retries on a connection-refused error. I also found two more weaknesses: a 422 (do not retry) and a 500 (retry) took the same path, and the rejected-lead log called the same API — so it failed too.
How I noticedDeliberately tested the failure path — stopped the API, sent a lead, watched the node. The setting's name promised behaviour the test disproved.
What I didWrote the observed result down instead of the expected one, and turned it into an engineering spec: a durable inbox, classified retries, an alert — and duplicate-lead prevention first, because retries are only safe once duplicates can't happen. Added to a 10-item automation backlog.
Test what happens when a dependency is down on day one. A setting's name is not evidence of its behaviour.
05
A prose payload typed as a number — n8n's strict validation caught it before send
What it gaveThe n8n node had its
report field declared as a number, but the workflow was sending the full "Weekly funnel report…" text — a string. n8n rejected the request: "'report' expects a number but we got 'Weekly funnel report…'".How I noticedThe error surfaced on a live run before the request went out — the strict type check did its job.
What I didChanged the field's type to string (the correct fix), not the "Ignore Type Conversion Errors" toggle (a band-aid that hides future mismatches).
Fix the schema, not the toggle. Strict validation is a feature when the payload is prose.
The reusable rule I apply every time
- Check against the source. Every number, date, name, phone number and ranking claim is traced to the real source before it is sent or acted on.
- Check the metric that matters. The flattering number (pair-count win rate, easy-set precision, "retry on fail" setting) is not the decision-relevant one. Use the volume-weighted, hard-set, failure-path number.
- Flag, don't skip. When a step can't be done or a claim can't be verified, say so in writing — never skip it quietly.