Catchpoint Selenium Retirement, Cypress Tap Fixes, and Inertia Bias in AI Evaluators

Key Takeaways

Catchpoint has fully deprecated Selenium Transaction Tests, requiring immediate migration and validation of Playwright or Puppeteer scripts. Cypress 15.21.1 improves live agent debugging by reducing terminal noise and surfacing richer runtime context. Research into Inertia Bias shows that LLMs grade their own prior actions too leniently, emphasizing the need to isolate evaluation contexts from generative planning history.

Read Today’s Notes

  • Catchpoint deprecates Selenium Transaction Tests: Support for Selenium Transaction Tests officially ended on August 31, 2026. Teams relying on Catchpoint for synthetic monitoring must migrate existing test suites to modern engines like Playwright or Puppeteer. While Catchpoint provides an automated converter, the translation is an irreversible step. Converted scripts require rigorous validation because differences in how Selenium and Playwright manage waits, element readiness, and asynchronous execution can alter test behavior.
  • Cypress 15.21.1 stabilizes agent-first debugging: The release of Cypress 15.21.1 brings key stability fixes to cypress tap. This patch resolves terminal flooding from repeated session-polling and blocks browser translation prompts from disrupting active runs. These improvements allow local AI coding agents to directly inspect the DOM, accessibility tree, and live test state without relying purely on static log analysis.
  • Inertia Bias in automated evaluators: The IBIS benchmark demonstrates that web research agents become overly lenient when evaluating the outcomes of steps they generated themselves. To prevent this bias, the proposed NIS-Agent architecture isolates the context used to generate actions from the context used to evaluate them. In experimental setups, this context isolation reduced token consumption by approximately 33 percent while maintaining benchmark performance.

Companion Newsletter

Automated testing is evolving rapidly around AI agent interfaces, but with increased autonomy comes the risk of systematic evaluation blind spots.

When an AI system is tasked with both planning actions and evaluating their success, it exhibits Inertia Bias—a tendency to judge its own choices more favorably than an independent observer would. In test automation and synthetic monitoring workflows, allowing an LLM evaluator full access to its own generation history can distort the reliability of quality gates. Isolating the grading context from the planning context ensures objective validation while significantly reducing unnecessary token overhead.

Practitioners should review their autonomous testing setups to verify whether evaluation steps are decoupled from generation history. Additionally, teams utilizing synthetic monitoring tools must proactively audit framework migrations—such as the required move from Selenium to Playwright in Catchpoint—to ensure asynchronous timing and element interactions remain dependable across all environments.

Research and References

More posts