#177
Key Takeaways
Evaluating AI agent skills and Model Context Protocol tools solely by their final output string leaves systems vulnerable to hallucinated context and unauthorized tool invocations. Testers should validate the full execution process trace, use zero-cost offline telemetry comparisons on saved trace files, and isolate CI/CD credentials across separate lint, run, and reporting jobs.
Read Today’s Notes
Testing Problem
- Interactive testing of custom AI skills and Model Context Protocol tools is subjective and expensive.
- Output-only automation checks only the final text string, allowing autonomous agents facing conflicting instructions to invent unshipped work or call incorrect tools to produce an expected answer.
- Local evaluation engines such as Vally run directly on the host rather than inside a container sandbox, which can expose CI credentials if pipeline jobs are not properly isolated.
Common Testing Blind Spots
- Output bias: Focusing exclusively on whether the final answer is correct while ignoring the process trace, missing cases where an agent hallucinates context or invokes the wrong skill.
- Implicit trust in host execution: Running evaluations in a single CI job, which grants the agent execution step full write access to pull requests.
- Evaluation economics: Repeatedly spending API credits on live validations instead of performing offline trace comparisons.
Method: Trace-Aware Baseline Regression with Credential Isolation
- Process trace validation: Write assertions that inspect both the final result and how the agent reached it, verifying exact skills invoked and conversation turn counts to confirm no unshipped work was invented.
- Zero-cost offline telemetry comparison: Run npm run eval compare against saved .jsonl trace files to analyze score deltas, token variances, and wall-time deltas without triggering new model calls.
- Cost-controlled and isolated CI architecture: Separate the pipeline into three distinct jobs. First, use the free lint command as an initial syntax gate. Second, execute the billable evaluation job with zero retries and isolated credentials without pull request write access. Third, pass the results to a separate, free reporting job that updates the sticky pull request comment.
Practical Exercises
- Operator Move: Select an existing AI agent skill in a project, run the agent locally to output a .jsonl trace file, and manually inspect the invoked skills and conversation turns to verify that no unshipped work was invented.
- Weekly Challenge (Agent Baseline CI Drill): Run a baseline evaluation to save a trace file, intentionally degrade the agent by modifying its system prompt, run an offline comparison to observe metric changes without spending API credits, and wire the three-job evaluation gate into a CI workflow.
Companion Newsletter
Moving AI agent evaluations from local experimentation into a continuous integration pipeline requires rethinking what constitutes a passing test and how execution permissions are structured. While tools like GitHub Copilot Evals and the Vally engine make local evaluation accessible, wiring them into CI/CD introduces reliability, cost, and security challenges that traditional test automation does not address.
The core issue with output-only assertions is that autonomous agents optimize for reaching the expected answer. When instructions conflict, an agent may take the path of least resistance by inventing unshipped work or invoking the wrong tool to satisfy the final text check. Treating that output as a pass masks underlying behavioral failures.
Process trace validation addresses this by inspecting the execution path alongside the output. By asserting on the exact skills invoked and the number of conversation turns recorded in a .jsonl trace file, teams can verify that the agent followed the intended workflow without fabricating context. Once baseline traces are saved, running npm run eval compare enables zero-cost offline telemetry analysis, surfacing score deltas, token variances, and wall-time changes without incurring additional model calls.
Security and cost control require structural separation in CI/CD. Because local evaluation engines like Vally execute directly on the host rather than in a container sandbox, combining linting, model execution, and pull request reporting into a single job exposes pull request write permissions to the agent run step. Splitting the workflow into three distinct jobs resolves this: a free lint job blocks syntax errors early, a billable run job executes with zero retries and no pull request write access, and a separate free reporting job posts the sticky pull request comment.
To apply this today, select an existing AI agent skill, run it locally to generate a .jsonl trace, and manually audit the invoked skills and turn counts. From there, establish a baseline trace, intentionally alter the system prompt to observe offline metric deltas, and structure the CI gate across separate lint, run, and report jobs.
Research and References
- GitHub Plugin Marketplace Starter – CI and Offline Eval Tools
https://github.com/github/plugin-marketplace-starter - GitHub Action for Copilot Evals
https://github.com/github/evals-action
