#40
Key Takeaways
Modern AI testing shifts from strict pass/fail assertions to probabilistic evaluation using datasets, rubrics, and metrics.
Testers must both evaluate AI outputs and adapt to using AI agents as testing tools.
Read Today’s Notes
Building an LLM Evaluation Pipeline
- Core problem: LLM outputs are non-deterministic → cannot rely on exact-match assertions
- Solution components:
- Golden dataset (15–20 test cases is enough to start)
- Evaluation rubric (e.g., accuracy, relevance scored 1–5)
- LLM-as-judge to score outputs
- Advanced capability:
- Regression detection using statistical tests (e.g., Wilcoxon test)
- Alert when quality degrades after prompt/model changes
Automating Chatbot Testing (Multi-turn)
- Problem:
- Manual conversation testing is inconsistent and unscalable
- Scripted expected outputs are brittle
- Approach:
- Use Conversation Simulator
- Define scenario → generate dynamic conversations
- Metrics:
- Completeness
- Turn relevance
- Knowledge retention
Mindset Shift: From Syntactic → Semantic Testing
- Traditional:
if X → expect Y
- AI systems:
if X → maybe Y or Z depending on context
- Implications:
- Replace pass/fail with scoring models
- Use:
- Pairwise Testing → input combinations
- State Transition Testing → agent workflows
Playwright MCP (AI Agents for Testing)
- Capability:
- Natural language → generates Playwright tests
- Self-healing automation
- Architecture:
- Planner agent
- Generator agent
- Healer agent
- When to use:
- ✅ Short, exploratory, sandboxed tasks
- ❌ Long, complex, maintainable test flows
Core Pattern Emerging
- “Evals = Unit Tests for AI”
- Replace:
- deterministic assertions
- With:
- datasets + metrics + scoring
Practical Starting Point
- Create:
- 15–20 test cases (golden dataset)
- Define:
- 1 scoring rubric (e.g., accuracy 1–5)
- Run:
- LLM-as-judge evaluation
- Outcome:
- Foundation for regression testing system
Companion Newsletter
AI Testing Is No Longer About “Correct vs Incorrect”
If you’re still writing assertions like:
assert response == expected_output
you’re testing the wrong thing.
AI systems don’t behave like traditional software. They operate in probabilities, not guarantees. That means the job of a tester is no longer to verify exact outputs—but to evaluate quality.
The New Testing Primitive: Evals
Across everything today—LLM pipelines, chatbot testing, even agent workflows—one idea keeps showing up:
Evals are the new unit tests.
Instead of checking correctness, you define:
- A dataset (what to test)
- A rubric (how to judge quality)
- A scoring system (how good is “good enough”)
This is a fundamental shift:
- From binary → graded
- From deterministic → statistical
Why This Matters for Testers
Without evals:
- You can’t detect regressions reliably
- You rely on manual spot-checking
- Quality becomes subjective
With evals:
- You can automate quality checks in CI/CD
- You can compare model/prompt changes
- You can scale testing beyond human review
This is the difference between experimenting with AI and engineering it.
Chatbots Expose the Problem Clearly
Try writing a traditional test for a chatbot:
- Input → expected output
It breaks immediately.
Why?
Because conversations are:
- Multi-turn
- Context-dependent
- Non-deterministic
The better approach is:
- Simulate conversations
- Evaluate outcomes using metrics like:
- Did the task complete?
- Did the bot stay relevant?
- Did it remember context?
The Second Shift: Testing AI vs Using AI to Test
There are now two separate problems:
- Testing AI systems
- Using eval pipelines, metrics, datasets
- Using AI to create tests
- Tools like Playwright MCP generating automation
Confusing these leads to poor decisions.
AI-generated tests are useful—but only in:
- Short-lived
- Exploratory
- Low-maintenance scenarios
For stable automation, you still need engineered tests.
What You Can Try Today
Start small. Don’t over-engineer.
- Create 15–20 test cases for your AI system
- Pick ONE quality metric (e.g., accuracy)
- Define a simple 1–5 scoring rubric
- Use an LLM to evaluate outputs
That’s it.
This small system becomes:
- Your first eval pipeline
- Your regression safety net
- Your foundation for scaling AI quality
Final Thought
Testing AI isn’t harder—it’s just different.
The sooner you stop asking:
“Is this output correct?”
And start asking:
“How good is this output?”
The faster you’ll adapt.
Research & References
- Building an LLM Evaluation Pipeline from Scratch: A Complete Tutorial
https://machinelearningplus.com/gen-ai/llm-evaluation-pipeline/ - DeepEval Conversation Simulator – Testing Multi-Turn AI Conversations
https://deepeval.com/docs/conversation-simulator - Why Traditional QA Fails for AI Agents (And What 10 Years in QA Didn’t Teach Me)
https://dev.to/jkorzeniowski/why-traditional-qa-fails-for-ai-agents-and-what-10-years-in-qa-didnt-teach-me-45if
