From Scripts to Agents: The New Stack for Testing AI Systems

Key Takeaways

The testing industry is undergoing a fundamental shift—from deterministic scripts to agent-driven, observable, and stateful AI systems. As tools like Google Gemini CLI, New Relic, and Promptfoo mature, QA teams must evolve from validating outputs to orchestrating, observing, and evaluating AI behavior across time, state, and cost. AI testing is no longer experimental—it is becoming a core engineering discipline.

Read Today’s Notes

On today’s episode of Testing Tech Horizon, we explore how AI testing is being reshaped by a new generation of tools built specifically for agentic and stateful systems.

We start with Google’s release of Gemini CLI, an open-source AI agent that runs directly in the terminal. Unlike traditional test automation tools, Gemini CLI allows testers and engineers to describe test scenarios in natural language, then have the agent execute shell commands, analyze logs, search documentation, and report findings conversationally. Already used by Google SREs in real incident response, this signals a future where test orchestration becomes conversational, not script-heavy. Crucially, Gemini CLI’s human-in-the-loop safety model and --dry-run mode make it viable for production-grade testing workflows.

Next, we address the long-standing “black box” problem in AI systems with New Relic’s new observability platform for ChatGPT and LLM-powered applications. For the first time, teams can trace an AI interaction end-to-end—from the initial prompt, through model calls, latency, token cost, errors, and safety signals—inside a single observability layer. For QA teams, this transforms AI testing from a pre-release activity into continuous, production-grade evaluation, enabling alerts for hallucinations, cost overruns, or quality regressions in real time.

We then expand beyond software into the physical world with Microsoft Research’s announcement of Rho-alpha, a foundation model designed for robotics and physical AI. Trained using a combination of synthetic data from NVIDIA Isaac Sim and human feedback, Rho-alpha demonstrates how simulation-driven testing can scale validation for complex systems. While robotics-focused, the testing pattern—synthetic scenario generation, simulation, and human-in-the-loop validation—is directly applicable to IoT, embedded systems, and complex enterprise software.

Finally, we examine Promptfoo’s newly released guide for evaluating stateful AI agents built with LangGraph. As AI systems move from single prompts to multi-step workflows with memory and branching logic, testing complexity increases dramatically. Promptfoo provides one of the first production-ready evaluation frameworks for these agents, covering robustness testing, automated scoring, and red-teaming. With a single command, teams can scaffold a full evaluation suite—lowering the barrier to serious agent testing.

The broader signal across all of today’s stories is unmistakable:
AI testing is shifting from output validation to behavior engineering.

Agents must be observed, evaluated, and governed continuously—not just checked before release. The teams that adopt agent-aware testing, observability, and simulation early will move faster with less risk. Those that rely on traditional QA approaches will increasingly struggle to understand why their AI systems fail.

If you are building or testing AI systems in 2026, the message is clear:
Learn to test agents—not just prompts.

Companion Newsletter

Today’s episode focuses on a problem most teams are about to hit before they realize it:

You can build an AI agent that “works” — and still have no reliable way to test whether it behaves correctly over time.

We look at why state, memory, and decision paths fundamentally break traditional test approaches, and what testers can do right now to regain control.

1. The Core Idea (Plainly Stated)

Testing AI Agents Is Not Testing Outputs

It’s testing behavior over time.

Single-prompt evaluations answer:

  • “Was this response good?”

Agent evaluation must answer:

  • “Did this system behave acceptably across a sequence of decisions, with memory, under uncertainty?”

That’s a different testing problem.

2. Where Traditional QA Breaks

Most existing QA approaches assume:

  • Deterministic inputs
  • Isolated executions
  • Clear expected outputs

AI agents violate all three.

Failure modes you won’t catch with classic tests:

  • Memory contamination (earlier steps bias later ones)
  • Compounding small errors
  • Goal drift (“helpful” but wrong actions)
  • Overconfidence in uncertain states

If you only test final answers, you miss how the system got there.

3. The Emerging Testing Pattern (What Actually Works)

A workable pattern is starting to form:

1️⃣ Constrain the agent

  • Limit tools
  • Limit actions
  • Limit scope of autonomy

2️⃣ Observe everything

This is where tools like New Relic matter.
If you can’t trace:

  • prompts
  • intermediate decisions
  • tool calls
  • costs

…you’re not testing. You’re hoping.

3️⃣ Test sequences, not snapshots

Frameworks like LangGraph make state explicit — which makes it testable.

Evaluation tools such as Promptfoo let you:

  • Replay decision paths
  • Inject adversarial steps
  • Compare behavior across runs

The unit under test is now the trajectory, not the response.

4. A Mental Model That Helps

Think of AI agents less like APIs and more like junior engineers:

  • They remember past instructions
  • They make assumptions
  • They take initiative
  • They sometimes sound confident while being wrong

We don’t test junior engineers with unit tests.

We:

  • Review decisions
  • Set guardrails
  • Observe behavior in real scenarios
  • Intervene early

Agent testing should feel similar.

5. Try This Today (Small, Concrete Exercise)

30-minute exercise:

  1. Take an existing AI workflow or agent
  2. Write down:
    • What state it remembers
    • What decisions it makes without you
  3. Identify one step where a wrong assumption would compound
  4. Ask:
    • How would I detect this in production?
    • What signal would warn me early?

If you can’t answer those, you’ve found a real test gap.

That’s success.

6. Where AI Will Likely Fail Here (On Purpose)

Expect problems with:

  • Long-running conversations
  • Ambiguous goals
  • Conflicting instructions
  • Edge cases that require “common sense”

These aren’t bugs to eliminate —
they’re risks to surface and monitor.

Research & References

More posts