Building Programmable QA with AI Agents

Key Takeaways

  • AI testing is shifting from text validation to outcome validation.
  • Anthropic’s system-level evaluation framework makes it clear that for AI agents, success is not what the model says but what it does. Teams must validate real-world state changes—database updates, API calls, file writes—not chat transcripts.
  • Programmable QA is emerging as the next evolution of test automation.
  • Using Playwright with Model Context Protocol (MCP) allows LLMs to directly control browsers in a deterministic, testable way. This bridges the gap between exploratory AI assistance and production-grade automation.
  • LLM-as-a-Judge is becoming reliable enough for production—if done correctly.
  • ChatBench demonstrates that structured judging prompts, low temperature settings, and Chain-of-Thought evaluation can reach ~85% agreement with human reviewers, making large-scale AI evaluation feasible.
  • Security testing must include every action an AI can perform—not just chat output.
  • Research from Praetorian proves that “write primitives” (forms, fields, actions) can be exploited to extract system prompts even when chat output is locked down. Any writable surface is a potential data exfiltration channel.
  • A clear testing pattern is forming for AI agents in production.
  • Modern AI QA now requires:
  • Functional validation (did the system change correctly?)
  • Quality validation (was the output good enough?)
  • Security validation (can actions be abused?)
  • Practical prompt engineering still matters.
  • For non-reasoning tasks (classification, extraction), repeating the same instruction 2–5 times can significantly boost accuracy—but must be A/B tested, as it degrades reasoning-heavy tasks.

Read Today’s Notes

Today’s briefing highlights a major inflection point in how AI systems—especially autonomous agents—must be tested before entering production.

Anthropic has introduced a system-level evaluation methodology that reframes AI testing entirely. Instead of validating conversation quality, teams must validate outcomes: did the agent reliably change the system state it was supposed to? Traditional unit tests are insufficient for non-deterministic agents, requiring hybrid evaluation strategies combining rules, LLM judges, and human review.

A practical example of this shift appears in thoughtbot’s work on programmable QA using Playwright MCP. Rather than relying on screenshots or brittle selectors, they expose the browser’s accessibility tree to an LLM, enabling structured, repeatable UI interaction. This approach transforms AI from a testing assistant into a controllable automation actor.

On the evaluation side, ChatBench provides a concrete, reproducible LLM-as-a-Judge methodology. Their findings show that prompt structure, temperature control (≤0.3), and Chain-of-Thought reasoning dramatically improve consistency and agreement with human evaluators—making automated quality validation viable at scale.

Security remains a critical concern. Praetorian demonstrates that even when chat responses are locked down, attackers can exploit “write primitives” such as form inputs to extract system prompts. This reinforces the need to test every capability an AI agent has, not just what it says.

Finally, tooling is catching up with these needs. Platforms like LangSmith provide the infrastructure required to trace agent behavior, build evaluation datasets, and operationalize the hybrid testing strategies now required for production AI systems.

Bottom line: AI testing is no longer about prompts and outputs—it is about systems, actions, and real-world impact.

Companion Newsletter

AI Testing Is About What Changed—Not What Was Said

A clear pattern is emerging in AI quality engineering: outputs are not enough.

Anthropic’s agent evaluation framework argues that success must be defined by outcomes—did the database update correctly, did the right API call happen, did the system end up in the expected state? This reframes AI testing as systems testing, not conversation review.

At the same time, tools like Playwright MCP make it possible for LLMs to operate directly inside real automation frameworks. Combined with reliable LLM-as-a-Judge techniques from ChatBench, testers now have the ingredients for scalable, test-driven AI validation.

Security is the forcing function. The “write primitive” attack proves that restricting chat output doesn’t secure an agent. Any action—writing files, submitting forms, calling APIs—must be treated as an attack surface.

Something concrete to try:

  • Validate outcomes, not messages
  • Judge responses with structured prompts
  • Audit every agent action, not just what it says

That’s the shift from AI demos to AI engineering.

Research & References

More posts