Before you begin
Before you start, make sure you have:- Coding agent: A coding agent with write access to your agent’s repository.
- Agent entry point: A function in the repository that takes a user request and returns the answer.
- API key and workspace ID: The same values as in Run your first eval. You add them to CI, not to the repository.
- Side effects known: A list of the tools your agent calls that send email, write to production, or change anything else. The brief tells the agent to stop and ask about these.
1
Connect agent to documentation
The agent follows the current docs more reliably than its training data. Every page is available as Markdown at
<page>.md, and https://platform.atlan.com/llms.txt lists every page.-
Connect docs MCP server: For Claude Code, run the command below. See Give coding agents Atlan docs for other clients.
2
Paste setup brief
The brief lists the files to create and the rules agents most often get wrong. Paste it as written, then answer any question the agent asks about side effects.
3
Review agent's changes
Check the agent’s pull request before you merge it. Each item can be checked by running a command or reading a file.
python -m evals.runexits0, ornpx tsx evals/run.tsdoes andnpx tsc --strict --noEmitpasses. No case inrun.resultshas anerror.run.summary["scores"](run.summary.scores) has one entry per scorer, each with acountequal to the cases it applies to.verify_experiment(client, run.experiment_id).raise_for_status()passes, or(await verifyExperiment(client, run.experimentId)).raiseForStatus()in TypeScript. This confirms every result has a trace and the sampled traces carry their score spans.- The eval name in
Eval(...)is a constant, not derived from the branch or the time. git grep -n "ATLAN_API_KEY="finds only placeholders..github/workflows/atlan-evals.ymltriggers onpull_requestand onpushto the default branch, setsATLAN_API_KEYandATLAN_WORKSPACE_IDin the action step’senv, grants onlycontents: readandpull-requests: write, and skips fork pull requests.
4
Add CI secret and variable
The workflow reads the key and workspace from your repository settings, so the eval can run without credentials in the code.
- Add a repository secret named
ATLAN_API_KEY. - Add a repository variable named
ATLAN_WORKSPACE_ID.
5
Verify pull request gate
The gate compares each pull request with the latest run on
main, so main needs one run first.- Merge the agent’s pull request, or run the workflow once on
main. This creates the baseline. - Open a test pull request.
main. If the check fails with invalid, see Eval errors. See What the PR comment shows.Troubleshooting
Common failures are a scorer name collision, a TypeScript eval without the results file, and a workflow that does not pass the key to the command. See Eval errors for each error and its fix.Next steps
- Gate pull requests with evals: Tune the gate’s thresholds and run expensive suites on demand.
See also
- Write code scorers: Scorer signatures, return shapes, and scorers to copy.
- Score agent behaviour: Score tool calls, trajectories, and multi-turn conversations.
- Eval options and errors: Every
Evaloption, environment variable, and limit.