Skip to main content
Give a coding agent, such as Claude Code, ChatGPT, or Cursor, a brief that sets up evals and a pull request gate for your agent’s repository. Use it when you want the setup done for you instead of following each page yourself. You still review the result: the last steps check the agent’s work and turn on the gate.

Before you begin

Before you start, make sure you have:
  • Coding agent: A coding agent with write access to your agent’s repository.
  • Agent entry point: A function in the repository that takes a user request and returns the answer.
  • API key and workspace ID: The same values as in Run your first eval. You add them to CI, not to the repository.
  • Side effects known: A list of the tools your agent calls that send email, write to production, or change anything else. The brief tells the agent to stop and ask about these.
1

Connect agent to documentation

The agent follows the current docs more reliably than its training data. Every page is available as Markdown at <page>.md, and https://platform.atlan.com/llms.txt lists every page.
  • Connect docs MCP server: For Claude Code, run the command below. See Give coding agents Atlan docs for other clients.
2

Paste setup brief

The brief lists the files to create and the rules agents most often get wrong. Paste it as written, then answer any question the agent asks about side effects.
3

Review agent's changes

Check the agent’s pull request before you merge it. Each item can be checked by running a command or reading a file.
  • python -m evals.run exits 0, or npx tsx evals/run.ts does and npx tsc --strict --noEmit passes. No case in run.results has an error.
  • run.summary["scores"] (run.summary.scores) has one entry per scorer, each with a count equal to the cases it applies to.
  • verify_experiment(client, run.experiment_id).raise_for_status() passes, or (await verifyExperiment(client, run.experimentId)).raiseForStatus() in TypeScript. This confirms every result has a trace and the sampled traces carry their score spans.
  • The eval name in Eval(...) is a constant, not derived from the branch or the time.
  • git grep -n "ATLAN_API_KEY=" finds only placeholders.
  • .github/workflows/atlan-evals.yml triggers on pull_request and on push to the default branch, sets ATLAN_API_KEY and ATLAN_WORKSPACE_ID in the action step’s env, grants only contents: read and pull-requests: write, and skips fork pull requests.
4

Add CI secret and variable

The workflow reads the key and workspace from your repository settings, so the eval can run without credentials in the code.
  1. Add a repository secret named ATLAN_API_KEY.
  2. Add a repository variable named ATLAN_WORKSPACE_ID.
5

Verify pull request gate

The gate compares each pull request with the latest run on main, so main needs one run first.
  1. Merge the agent’s pull request, or run the workflow once on main. This creates the baseline.
  2. Open a test pull request.
The gate works when the pull request shows the eval check and a comment with the score diff against main. If the check fails with invalid, see Eval errors. See What the PR comment shows.

Troubleshooting

Common failures are a scorer name collision, a TypeScript eval without the results file, and a workflow that does not pass the key to the command. See Eval errors for each error and its fix.

Next steps

See also