Just-in-Time Testing and the Rise of AI-Generated Validation

Key Takeaways

Testing is shifting from maintaining static test suites to orchestrating dynamic, AI-generated validation. Testers must rethink their role as system designers who guide, evaluate, and control AI-driven testing workflows.

Read Today’s Notes

1. Meta’s Just-in-Time Testing (JiTTests)

Meta introduced a production-scale framework that generates tests automatically at pull request time.

Key characteristics:

  • Tests are generated dynamically using LLMs.
  • Validation happens exactly when new code is submitted.
  • Over 22,000 generated tests were analyzed, catching bugs that would have reached production.

Why this matters:

  • Reduces long-term maintenance of static test suites.
  • Matches the speed of AI-assisted development.
  • Shifts QA effort from writing tests → orchestrating AI validation systems.

System insight:
Testing becomes an adaptive system, not a fixed artifact.

2. Open-Source Coding Models Reach Near-Parity

GLM-5 and Kimi K2.5 now score within ~3% of leading proprietary models on SWE-bench.

Implications for QA teams:

  • High-quality AI test generation is no longer locked behind expensive APIs.
  • Teams can self-host advanced models.
  • Enables experimentation with:
    • AI-generated regression tests
    • Failure analysis
    • Test refactoring

Strategic impact:
AI-powered testing becomes accessible infrastructure, not premium tooling.

3. LLM-as-a-Judge: A New Evaluation Paradigm

The industry is converging on using one LLM to evaluate another.

Why classical assertions fail:

  • AI outputs are probabilistic.
  • Binary pass/fail checks are insufficient.

LLM-as-a-Judge enables:

  • Scoring relevance
  • Measuring correctness
  • Assessing reasoning quality
  • Automating subjective evaluation

Core challenge:
Designing evaluation prompts, mitigating bias, and maintaining stable scoring pipelines.

4. Agenta: Version Control for LLM Test Data

Agenta introduced:

  • Test set versioning
  • Folder organization
  • Workflow tooling

Why this is critical:

  • Prevents silent test drift.
  • Enables reproducible evaluations.
  • Brings Git-style discipline to LLM testing.

Key insight:
LLM test data now requires the same rigor as production code.

5. System-Level Conclusion

AI testing is evolving into systems engineering, combining:

  • On-demand test generation
  • Probabilistic evaluation
  • Version-controlled test data
  • Continuous feedback loops

Companion Newsletter

From Static Test Suites to Just-in-Time AI Validation

Most testing teams still operate under a static mindset: write tests, maintain them, and hope they keep up with development. But AI-assisted development is changing that equation.

Meta’s Just-in-Time Testing framework shows what happens when testing becomes dynamic. Instead of maintaining massive test suites, tests are generated on demand, exactly when a pull request is created. This approach aligns testing speed with modern development velocity and dramatically reduces maintenance overhead.

At the same time, open-source coding models like GLM-5 and Kimi K2.5 are reaching performance levels once reserved for proprietary giants. This means advanced AI-powered test generation is now available to almost any team, even without enterprise budgets.

But generating tests is only half the problem. Evaluating AI output requires a new mindset. Traditional assertions break down when outputs are probabilistic. That’s why the industry is converging on the “LLM-as-a-Judge” pattern — using one model to evaluate another. This unlocks scalable, automated quality checks for AI systems.

Finally, tools like Agenta bring version control and workflow discipline to LLM testing. Test data, prompts, and evaluations now get the same rigor we apply to source code.

What You Can Try This Week

Pick one stable feature in your codebase. At pull request time, ask an LLM to generate regression tests for that feature. Compare those tests to your existing suite.

You’ll quickly see where AI-generated testing shines — and where human-designed tests still outperform. This hands-on experiment is one of the fastest ways to prepare your team for the coming shift toward just-in-time testing.

Research & References

More posts