Archive

This episode examines the industry’s move toward production-realistic testing for AI applications. We cover OpenAI’s deployment simulation patterns, new strategies for testing AI agent security, and frameworks for benchmarking models against specific production toolsets.

This episode explores the new infrastructure landscape for testing agentic AI, featuring Cisco’s agent security harness and Mitiga’s supply-chain scanner. We also discuss how the SkillsBench benchmark and OpenAPI readiness scoring are helping teams improve agent reliability and test generation effectiveness.

Today we examine the move toward execution-based AI validation and how QA teams can shift from deterministic assertions to probabilistic evaluation. We explore new tools for testing LLM apps, the real-world performance gaps revealed by the latest agent benchmarks, and the growing importance of documenting testing intent for AI governance.

The sudden regulatory suspension of Claude Fable 5 highlights immediate operational risks for QA teams relying on single-vendor AI architectures. Concurrently, a surge in AI-driven vulnerability discovery is moving the primary testing bottleneck from bug detection to human verification and triage. This brief explores these shifts alongside new lightweight coding models and the widening skills…

This episode explores new open-source frameworks and CLI tools designed to bring systematic evaluation to AI agent testing. We discuss how to implement autonomous QA loops for AI-generated code and the importance of using multi-phase evaluation methodologies to detect agent hallucinations and tool-use errors.

This episode explores Anthropic’s Claude Fable 5 and how it shifts QA from validating outputs to evaluating multi-day autonomous processes. We also unpack UC Berkeley’s benchmark revealing a massive failure rate for AI agents on real-world tasks. Finally, we discuss DeepEval 4.0’s integration of LLM evaluation into CI/CD pipelines and look at National Instruments’ new…

This episode covers practical techniques for scaling the evaluation of conversational and long-horizon AI agents. We explore the LLM-as-judge pattern using AWS’s new voice test harness and look at Anthropic’s methodology for automated adversarial red-teaming. Finally, we discuss how the Harness-1 architecture isolates search logic from decision-making to provide better diagnostic metrics for AI testers.

This episode covers the shifting technical and economic boundaries of AI driven test automation. We analyze NVIDIA new open weight model optimized for production agents, GitHub open source toolkit for enforcing specification boundaries, and the latest persistent memory features in VS Code. The discussion centers on how these tools provide the guardrails necessary for structured…

This episode analyzes the shift toward policy-driven testing for AI agents, featuring Microsoft’s new ASSERT framework and Google’s Gemma 4 12B. We also review new benchmark data from Testlio and KushoAI that highlights the reliability gaps in autonomous agents and AI-generated testing tools.

AI-accelerated development is forcing a move toward autonomous testing pipelines and trajectory-based evaluation. This episode breaks down how to audit AI model benchmarks against your own production constraints rather than trusting vendor scores. We also cover the impact of new multi-agent workflows and the availability of frontier models on AWS Bedrock.