Building Practical AI Testing: Evals, Chatbots, and Playwright MCP

#40

Key Takeaways

Modern AI testing shifts from strict pass/fail assertions to probabilistic evaluation using datasets, rubrics, and metrics.
Testers must both evaluate AI outputs and adapt to using AI agents as testing tools.

Read Today’s Notes

Building an LLM Evaluation Pipeline

  • Core problem: LLM outputs are non-deterministic → cannot rely on exact-match assertions
  • Solution components:
    • Golden dataset (15–20 test cases is enough to start)
    • Evaluation rubric (e.g., accuracy, relevance scored 1–5)
    • LLM-as-judge to score outputs
  • Advanced capability:
    • Regression detection using statistical tests (e.g., Wilcoxon test)
    • Alert when quality degrades after prompt/model changes

Automating Chatbot Testing (Multi-turn)

  • Problem:
    • Manual conversation testing is inconsistent and unscalable
    • Scripted expected outputs are brittle
  • Approach:
    • Use Conversation Simulator
    • Define scenario → generate dynamic conversations
  • Metrics:
    • Completeness
    • Turn relevance
    • Knowledge retention

Mindset Shift: From Syntactic → Semantic Testing

  • Traditional:
    • if X → expect Y
  • AI systems:
    • if X → maybe Y or Z depending on context
  • Implications:
    • Replace pass/fail with scoring models
    • Use:
      • Pairwise Testing → input combinations
      • State Transition Testing → agent workflows

Playwright MCP (AI Agents for Testing)

  • Capability:
    • Natural language → generates Playwright tests
    • Self-healing automation
  • Architecture:
    • Planner agent
    • Generator agent
    • Healer agent
  • When to use:
    • ✅ Short, exploratory, sandboxed tasks
    • ❌ Long, complex, maintainable test flows

Core Pattern Emerging

  • “Evals = Unit Tests for AI”
  • Replace:
    • deterministic assertions
  • With:
    • datasets + metrics + scoring

Practical Starting Point

  • Create:
    • 15–20 test cases (golden dataset)
  • Define:
    • 1 scoring rubric (e.g., accuracy 1–5)
  • Run:
    • LLM-as-judge evaluation
  • Outcome:
    • Foundation for regression testing system

Companion Newsletter

AI Testing Is No Longer About “Correct vs Incorrect”

If you’re still writing assertions like:

assert response == expected_output

you’re testing the wrong thing.

AI systems don’t behave like traditional software. They operate in probabilities, not guarantees. That means the job of a tester is no longer to verify exact outputs—but to evaluate quality.


The New Testing Primitive: Evals

Across everything today—LLM pipelines, chatbot testing, even agent workflows—one idea keeps showing up:

Evals are the new unit tests.

Instead of checking correctness, you define:

  • A dataset (what to test)
  • A rubric (how to judge quality)
  • A scoring system (how good is “good enough”)

This is a fundamental shift:

  • From binary → graded
  • From deterministic → statistical

Why This Matters for Testers

Without evals:

  • You can’t detect regressions reliably
  • You rely on manual spot-checking
  • Quality becomes subjective

With evals:

  • You can automate quality checks in CI/CD
  • You can compare model/prompt changes
  • You can scale testing beyond human review

This is the difference between experimenting with AI and engineering it.


Chatbots Expose the Problem Clearly

Try writing a traditional test for a chatbot:

  • Input → expected output

It breaks immediately.

Why?

Because conversations are:

  • Multi-turn
  • Context-dependent
  • Non-deterministic

The better approach is:

  • Simulate conversations
  • Evaluate outcomes using metrics like:
    • Did the task complete?
    • Did the bot stay relevant?
    • Did it remember context?

The Second Shift: Testing AI vs Using AI to Test

There are now two separate problems:

  1. Testing AI systems
    • Using eval pipelines, metrics, datasets
  2. Using AI to create tests
    • Tools like Playwright MCP generating automation

Confusing these leads to poor decisions.

AI-generated tests are useful—but only in:

  • Short-lived
  • Exploratory
  • Low-maintenance scenarios

For stable automation, you still need engineered tests.


What You Can Try Today

Start small. Don’t over-engineer.

  1. Create 15–20 test cases for your AI system
  2. Pick ONE quality metric (e.g., accuracy)
  3. Define a simple 1–5 scoring rubric
  4. Use an LLM to evaluate outputs

That’s it.

This small system becomes:

  • Your first eval pipeline
  • Your regression safety net
  • Your foundation for scaling AI quality

Final Thought

Testing AI isn’t harder—it’s just different.

The sooner you stop asking:

“Is this output correct?”

And start asking:

“How good is this output?”

The faster you’ll adapt.

Research & References