PREVIOUS MONTH · OCTOBER 2026
Top 10
Ten practical AI testing lessons from the monthly report, collected in one place with links to the original briefings.
-
#1
Lock the Tests Before the AI Fixes the Code: A Green Build Can Mean the Test Was Loosened
The lesson: Separate regression-test authorship from code repair, verify that the test fails on the unpatched code, and lock the test files before the repair agent runs. Require named human approval for test changes.
-
#2
Make the Regression Test Fail on the Old Code, Then Mutate It: Green on Working Code Is Weak Evidence
The lesson: Require bug-specific regression tests to fail on the old code, then use mutation testing to check whether they detect requirement faults. Choose a mutation threshold appropriate to your context and check that tests actually execute the business logic.
-
#3
Assert on the Path, Not Just the Final State: Clean Data at the End Can Hide an Unauthorized Step
The lesson: Test ordered checkpoints in the tool-call log, including authorization before state changes and consistent entity IDs. Allow exploratory steps between checkpoints, and compare critical arguments with code rather than relying only on an LLM judge or the final state.
-
#4
Check the Session, Not Just the Diff: Repository Rules Need Hooks, Not Just Instructions
The lesson: Enforce repository policies with pre-tool hooks, versioned rule manifests, and CI checks of agent session logs. Treat unapproved commands and mid-session policy violations as defects even when unit tests pass and the final diff looks correct.
-
#5
Test the Backend Gate, Not the Polite Refusal: Replay Recorded Attack Calls with the Model Taken Out
The lesson: Replay recorded out-of-scope tool calls directly against backend authorization gates without involving the model. Verify rejection, unchanged state, and an audit record, and ensure delegated agents cannot gain permissions beyond those of their parent.
-
#6
Change the Rules Mid-Run: Agents and Caches Can Keep Acting on the Old Policy
The lesson: Change a business rule in the central policy store during an agent session and verify that it takes effect at the intended time. Check cache invalidation, conflict resolution, and external gateway enforcement against actions based on stale policy.
-
#7
Change a Middle Step, Watch the Answer: Test Whether the Explanation Drives the Decision
The lesson: Change an intermediate premise or number and let the model continue, then classify the result as silent bypass, self-correction, or error propagation. Use the results as evaluation signals rather than proof that the explanation faithfully represents the decision process.
-
#8
Treat Browser-Exposed WebMCP Tools as an Untrusted Surface: Test the Tool Contract, Not Just the Clicks
The lesson: Inspect browser-exposed tool schemas, compare declared behavior with actual storage and network effects, and test cross-origin frames and login/logout lifecycle changes. Confirm that the backend rejects unauthorized calls even after sensitive tools disappear from the page.
-
#9
Check the Numbers, Not the 200: Pin Data Agents to Certified Metrics and Known Answers
The lesson: Restrict data agents to certified metrics and dimensions, inspect compiled SQL for duplicate-producing joins and missing filters, and test against staging data with known answers. Check ambiguous date questions against company definitions or require clarification.
-
#10
Allowed Isn't the Same as Safe: Test Tool Chains and Where Traffic Actually Goes
The lesson: Test system-wide safety rules across multi-turn tool chains because individually harmless tools can combine into data exfiltration. Verify actual destination hosts at the network gateway rather than treating allowed IP addresses or shared CDN infrastructure as sufficient trust boundaries.
