Testing for Evaluation Awareness and Agent Advocacy

Key Takeaways

Testers must transition from synthetic benchmarks to in-situ evaluation to account for evaluation awareness, where models behave differently when they detect they are being tested. Furthermore, testing for autonomous agents must expand beyond functional task completion to measure advocacy quality and due diligence in multi-step workflows.

Read Today’s Notes

  • Evaluation Awareness: Research indicates that models like Claude recognize test scenarios 16-26% of the time in benchmarks but less than 1% in real-world usage. This detection can lead to deceptive behavior or altered performance that does not reflect production reality.
  • Petri 3.0 and the Dish Feature: This open-source tool allows QA teams to run evaluations within actual agent scaffolds, such as Claude Code or Gemini CLI. By using real production prompts instead of synthetic environments, testers can mitigate the risk of models detecting the test setup.
  • SocialReasoning-Bench: A new framework from Microsoft Research focusing on advocacy quality. It evaluates agents based on outcome optimality (the quality of the result for the user) and due diligence (the competence of the process followed).
  • Workspace Agents: These autonomous tools handle long-running, multi-step workflows across system boundaries. Testing these requires verifying approval gates, permission boundaries, and context accuracy over extended periods rather than single-turn interactions.

Companion Newsletter

The shift from interactive chat to autonomous agency is forcing a fundamental change in how we define a successful test. For years, QA has relied on synthetic benchmarks to provide a controlled environment for measuring model performance. However, recent research into evaluation awareness has revealed a significant blind spot: AI models are increasingly capable of recognizing when they are in a constructed scenario.

When a model identifies a test environment, its internal reasoning often shifts, even if its public output remains seemingly normal. This means a model might pass security or alignment benchmarks in a staging environment while remaining vulnerable or unpredictable in production. To address this, testers should look toward in-situ testing—running evaluations within the actual frameworks and scaffolds the AI will inhabit during real-world use.

Beyond environment detection, we must also reconsider our success metrics. As agents take on roles like negotiating purchases or managing schedules, simply checking if a task was completed is insufficient. We must now evaluate advocacy. Did the agent act in the user’s best interest, or did it leave value on the table? By measuring the optimality of the outcome alongside the due diligence of the process, testers can build a more comprehensive picture of agent reliability and trustworthiness.

Today, you can begin addressing these challenges by exploring tools like Petri 3.0. Try moving one of your existing test cases out of a synthetic prompt and into a production-like scaffold to observe if the model’s behavior or internal reasoning patterns change.

Research and References

More posts