From Vibe Testing to Eval-Driven AI Testing

#43

Key Takeaways

AI testing is shifting from subjective “vibe checks” to structured, automated evaluation.
Testers need to think in terms of conversations, goals, and metrics—not just single responses.

Read Today’s Notes

1. Eval-Driven Development (Red Hat)

  • Introduces a structured approach to testing AI agents using evaluation loops
  • Moves testing from:
    • ❌ Manual checking (“looks good”)
    • ✅ Automated metrics + CI/CD integration
  • Key idea:
    • Define what “good” means before scaling AI usage
  • Includes:
    • 8-stage workflow
    • Integration with DeepEval
    • Real code examples

2. Multi-Turn Conversation Testing (DeepEval)

  • Traditional testing = single prompt/response
  • Problem:
    • Misses context drift
    • Misses contradictions across turns
  • New approach:
    • Evaluate entire conversations
  • Key metrics:
    • Coherence (does it stay consistent?)
    • Completeness (does it solve the task?)
  • New testing mindset:
    • From: “Did this answer work?”
    • To: “Did the user achieve their goal?”

3. AI Security Testing (Promptfoo)

  • AI introduces new vulnerabilities:
    • Prompt injection
    • Jailbreaking
    • Data leakage
  • Promptfoo provides:
    • Red teaming templates
    • Automated adversarial tests
  • Practical entry point:
    • Run: npx promptfoo redteam setup
  • Key shift:
    • Security testing becomes part of QA, not separate

4. Testing Pyramid for Agentic AI

  • Classic pyramid adapted for AI agents

Layer 1: Unit Tests

  • Deterministic tools (APIs, functions)

Layer 2: Goal-Based Tests

  • Can the agent complete a task?

Layer 3: Statistical Tests

  • Evaluate performance across many runs
  • Why it matters:
    • AI is non-deterministic → cannot rely on single test runs

5. Practical Exercise

  • Pick 1 AI use case
  • Define:
    • 1 conversation flow
    • 3–5 custom metrics
  • Run evaluation using DeepEval

Companion Newsletter

Stop “Vibe Testing” Your AI

Most testers working with AI systems still rely on a simple question:

“Does this output look right?”

That approach doesn’t scale.

AI systems are non-deterministic. The same input can produce different outputs. A response that looks correct once may fail in subtle ways later—especially across longer interactions.

The Shift: From Outputs to Evaluations

The industry is moving toward eval-driven development.

Instead of manually checking outputs, you define:

  • What “good” means
  • How to measure it
  • When it fails

This changes testing from:

  • Checking examples
    to
  • Measuring behavior

Why This Matters for Testers

Three major changes are happening:

1. The unit of testing is expanding
You’re no longer testing a single response.
You’re testing:

  • Conversations
  • Workflows
  • Goal completion

2. Metrics replace intuition
Instead of saying:

  • “This feels correct”
    You define:
  • Coherence
  • Completeness
  • Task success

3. Security is now your responsibility
AI systems introduce new risks:

  • Prompt injection
  • Jailbreaks
  • Unsafe outputs

These require adversarial testing, not just validation.


What You Can Try Today

Spend 60–90 minutes on one experiment:

  1. Pick a chatbot or AI workflow
  2. Define a single user journey
  3. Create 3–5 evaluation metrics:
    • Did the user goal succeed?
    • Did the system stay consistent?
    • Did it avoid unsafe outputs?
  4. Run one automated evaluation using a tool like DeepEval

Don’t try to cover everything.

The goal is simple:

Learn how to define “quality” for AI in a measurable way.

Research & References