--- name: catch-the-ai-checks description: 15 rules from the Noesa course "Catch the AI: verify before you trust". For a professional recently handed AI tools at work — an analyst, coordinator, ops/support/marketing person, or PM — who uses AI daily and doesn't want to be the one who forwarded its confident mistake. --- # Catch the AI: verify before you trust — the rules Use with: Claude Code or Claude (save as a skill), Cursor (save under .cursor/rules as .mdc), ChatGPT or any other assistant (paste the text below into custom instructions or a project's instructions). 15 rules, taken from the course at https://noesa.leafsoft.online/c/catch-the-ai Each heading is one thing the course teaches. Most are checks to run on your own output before presenting it as done; a few are background you are expected to have. "Wrong by default" lists 9 specific ones. Apply these to the thing you are producing — the type, the schema, the query, the copy — not only to how you explain it. Where a rule names a field, a format or an identifier, that name belongs in the output. ## Meet the confident liar Explain why an AI answer can be specific, fluent, and wrong at the same time — and point to the exact moment a check was needed. **Wrong by default:** - Whatever Tiffin's bags actually do is in a spec sheet the model has never seen. Notice this is the most precise sentence in the answer — a temperature, a duration, a rating. That precision is what gets it pasted into a customer reply, and now Tiffin has made a food-safety promise on the strength of a guess. - Heat kills bacteria but does not remove the toxins some foods build up first — which is exactly why the biryani row mattered. It reads like kitchen common sense, and that is the whole problem: it is the kind of claim nobody thinks to look up. ## See why it sounds so sure Explain, in one sentence, why a language model can produce a polished answer without first establishing that its claims are true. ## Name the six ways it goes wrong Classify an AI mistake as fabrication, bad math, stale information, misreading the ask, overreach, or inherited bias and choose the first check each needs. ## Catch invented facts, quotes and sources Trace a factual claim, quotation, or citation to a primary source and reject references that cannot support the exact sentence. ## Catch numbers that don't add up Verify an AI-produced total, percentage, or average by recovering its inputs, recomputing it, and checking whether the comparison makes operational sense. **Wrong by default:** - 2,400 out of 8,000 is 30%. Nothing in Dev's rows counts resolved contacts, so this rate is computed against a population nobody supplied — and once the denominator is unnamed, the number cannot be checked at all. That is the move to watch for: not bad arithmetic, a quietly swapped 'out of what'. - Other issues came to 2,800 against delays' 2,400. Delays are the largest NAMED problem, which is a different claim — and it is the one that reads better on a slide, so it is the one that survives. - 2,400 against 1,200 is exactly twice. 'More than' is doing unearned work: one word inflating a true comparison into a false one, by a margin small enough that the magnitude still feels right. ## Name the claim, then check it Classify a claim as a fact, number, summary, internal claim, or judgment and select the evidence needed before trusting it. ## Verify a fact Verify a factual claim by defining its scope, tracing it to an appropriate primary source, and recording a reproducible verdict. ## Verify a number Produce a number-check record that preserves source inputs, definitions, formula, recomputation, and a sanity verdict. ## Verify a summary Verify an AI summary by mapping each important source point to the summary and marking omissions, additions, distortions, and misplaced certainty. **Wrong by default:** - The notes say the opposite: raised, uncommitted, and waiting on finance sign-off. The topic survived and the certainty flipped — 'unresolved' became 'resolved'. This is the distortion that reads best in a summary, which is exactly why it travels: nobody rereads the notes to check a sentence that sounds like progress. - An omission hiding inside an addition. Nothing said about photography is false — but 'only' quietly deletes Priya's commission point, the item with money and a partner relationship attached. Verify a summary by asking what the source contains that the summary does not, or the consequential thing disappears while every remaining sentence checks out. ## Verify against your world Identify an internal claim, locate the controlled source or accountable owner, and separate company truth from a plausible industry default. **Wrong by default:** - This may well be correct — that is what makes it dangerous. Tiffin's payout calendar is an internal fact the model has never seen, so its accuracy here is an accident until someone who owns the calendar confirms it. Send it and you have told a partner when to expect money, in writing, on the strength of an industry default. - Who absorbs a refund is a commercial term that varies by contract and platform. Stating it commits Tiffin to a position it may not hold, to the party it costs money — and the partner will hold you to the sentence whether or not the policy agrees with it. ## Make the AI check itself Ask an AI to expose sources, assumptions, calculations, uncertainties, and counterarguments while identifying which parts still require independent verification. ## Match the checking to the stakes Set a proportionate verification level using potential harm, audience reach, reversibility, and the authority your output will carry. ## Know when to stop and escalate Recognize when verification exceeds your evidence or authority and escalate with a clear claim, risk, work completed, and decision needed. ## Build your verification checklist Assemble a one-page Verification Checklist that routes claims, scales effort to stakes, records evidence, and triggers escalation. ## Run the full check on real work Run your Verification Checklist on one real AI-assisted artifact and document a ship, revise, stop, or escalate decision.