Gate pull requests with evals

The Atlan Evals GitHub Action runs your eval on every pull request, compares it case by case with the last run on the default branch, comments the score diff on the pull request, and fails the check on a regression. Use it when a change to your agent must not make it worse on the cases you care about. Outside GitHub Actions, the last step shows a script that applies the same rule.

Before you begin

Before you start, make sure you have:

Eval script: A command, such as python -m evals.run, that runs one or more Eval calls. Keep each eval’s name the same on every branch: baselines are matched on the name and the dataset.
Dataset: Recommended. Results pair across runs by dataset_record_id, or by case_id for inline cases with stable IDs. See Create eval datasets.
API key: A workspace-scoped key that can create experiments, stored as the repository secret ATLAN_API_KEY.
Workspace ID: Stored as the repository variable ATLAN_WORKSPACE_ID.

Write eval script

The script builds the suite and runs it. It does not compare anything or choose an exit code: the action does that.

Python
TypeScript

evals/run.py

import os
from atlanai import AtlanClient, Eval, push_dataset

from my_agent import answer               # your agent
from evals.scorers import exact_match, required_facts

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "support-agent-cases", [
    {"name": "refund-status", "input": {"question": "Where is my refund for order 1042?"},
     "expected": {"value": "processed"}},
    # ... the rest of your cases, or load them from a file in the repo
])

Eval(
    "support-agent",                    # the same name on every branch
    dataset=suite.id,
    task=answer,
    scores=[exact_match, required_facts],
    client=client,
)

Run it from the repository root as a module, python -m evals.run, with an empty evals/__init__.py, so evals.scorers and your agent’s package import. python evals/run.py puts evals/ itself on the import path, and from evals.scorers import ... then fails.Inside the action, Python Eval stamps the commit, branch, and pull request number onto the experiment config, and reports each experiment to the action. There is nothing else to wire up.

The action learns which experiments a command produced from a results file, and checks each experiment’s commit and pull request. A TypeScript eval stamps the git identity into config and appends the experiment to the results file:

evals/run.ts

import { appendFileSync } from "node:fs";
import { Eval } from "@atlanai/sdk";

const env = process.env;
const git = Object.fromEntries(
  [["branch", env.ATLAN_EVAL_GIT_BRANCH], ["sha", env.ATLAN_EVAL_GIT_SHA], ["pr", env.ATLAN_EVAL_GIT_PR]]
    .filter(([, v]) => v),
);

const run = await Eval("support-agent", { dataset: "support-agent-cases", task: answer, scores: [exactMatch] }, {
  config: { git, ...(env.ATLAN_EVAL_MODEL ? { model: env.ATLAN_EVAL_MODEL } : {}) },
});

if (env.ATLAN_EVAL_RESULTS_FILE) {
  const e = run.experiment as { id: string; name: string; displayName?: string; datasetId?: string };
  appendFileSync(env.ATLAN_EVAL_RESULTS_FILE, JSON.stringify({
    experiment_id: run.experimentId, name: e.name, display_name: e.displayName ?? "support-agent",
    dataset_id: e.datasetId ?? null, status: "completed",
  }) + "\n");
}

display_name must be the eval name you pass to Eval: the action matches baselines on it. The action sets the ATLAN_EVAL_* variables for any command, so this works unchanged inside it. To check it locally, set ATLAN_EVAL_RESULTS_FILE to a scratch path and ATLAN_EVAL_GIT_BRANCH, ATLAN_EVAL_GIT_SHA, and ATLAN_EVAL_GIT_PR to test values, run the command, and read the line it appends.The project needs tsx, typescript, and @types/node as dev dependencies (npm install -D tsx typescript @types/node), "type": "module" in package.json for top-level await, and a tsconfig.json such as:

tsconfig.json

{
  "compilerOptions": {
    "target": "es2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "strict": true,
    "noEmit": true,
    "skipLibCheck": true
  },
  "include": ["src", "evals"]
}

To run the command locally against a gateway other than https://api.atlan.com, also set ATLAN_BASE_URL. The workflow does not need it, because the action works only with https://api.atlan.com.

Add workflow

Python
TypeScript

.github/workflows/atlan-evals.yml

name: Atlan evals

on:
  pull_request:
  push:
    branches: [main]            # each run on main becomes the next baseline
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write          # to post the comment

concurrency:
  group: atlan-evals-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  eval:
    # Fork PRs never receive secrets; skip them rather than fail confusingly.
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          persist-credentials: false
      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
        with:
          python-version: "3.12"
      - run: pip install -U 'atlanai[tracing]' -r requirements.txt   # your agent's dependencies
      - uses: atlanai/agent-registry-action/eval@83e1371227d6a417c4594e32882a2c3577930381
        with:
          command: python -m evals.run
          api-key: ${{ secrets.ATLAN_API_KEY }}
          workspace-id: ${{ vars.ATLAN_WORKSPACE_ID }}
          thresholds: |
            default=0.05
        env:
          # The command needs its own credentials: the action does not pass api-key to it.
          ATLAN_API_KEY: ${{ secrets.ATLAN_API_KEY }}
          ATLAN_WORKSPACE_ID: ${{ vars.ATLAN_WORKSPACE_ID }}
          # Secrets your agent needs go here too.
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

.github/workflows/atlan-evals.yml

name: Atlan evals

on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: atlan-evals-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  eval:
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          persist-credentials: false
      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
        with:
          node-version: "22"
      - run: npm ci
      - run: npx tsc --noEmit
      - uses: atlanai/agent-registry-action/eval@83e1371227d6a417c4594e32882a2c3577930381
        with:
          command: npx tsx evals/run.ts
          api-key: ${{ secrets.ATLAN_API_KEY }}
          workspace-id: ${{ vars.ATLAN_WORKSPACE_ID }}
          thresholds: |
            default=0.05
        env:
          ATLAN_API_KEY: ${{ secrets.ATLAN_API_KEY }}
          ATLAN_WORKSPACE_ID: ${{ vars.ATLAN_WORKSPACE_ID }}
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

The api-key input is used only by the action, to read results. It is deliberately not placed in the command’s environment, so the step’s env must set ATLAN_API_KEY and ATLAN_WORKSPACE_ID for Eval; without them the verdict is invalid. Your eval command runs in the same job, so treat it as trusted code. The action refuses to run under pull_request_target. The examples pin the action to an immutable commit; review a newer commit before updating that reference.

Do not add a paths: filter to pull_request if branch protection requires the check. A pull request that touches no matching path never runs the workflow, so the required check stays pending and the pull request cannot merge. To skip unrelated pull requests, test the paths inside the job instead.

Establish baseline

Merge the workflow, or run it once on main with Run workflow. That run becomes the baseline. Until one exists, pull requests pass with a no-baseline warning.

push to main ─► eval run ─► becomes the baseline
pull request ─► eval run ─► compared with the baseline ─► PR comment + pass/fail check

The action picks the most recent completed run on the default branch with the same eval name (the experiment’s display_name) and dataset, and prefers one with the same model input. Set baseline to an experiment ID to pin a run, or to none to report scores with no comparison.

Verify gate on pull request

Open a pull request. The action comments, for each eval, a table of each score’s baseline mean, pull request mean, and change, with counts of improved and regressed cases. It also shows cost, tokens, and latency for information, the verdict, and links to both experiments. The comment lists record IDs and scores only, never inputs, outputs, trace text, or error messages, and it is updated in place on each push.

Verdict	When	Check
`invalid`	The command exited non-zero or reported no experiment. Or the run had no cases or no scores, more than `max-error-rate` of its cases errored, it did not complete, it came from another commit or pull request, or its results could not be read in full.	Fails, always
`regression`	A score’s mean dropped by more than its threshold, and at least `min-regressed-cases` cases got worse	Fails, unless `fail-on: invalid`
`no-baseline`	No baseline, or no cases shared with it	Passes, with a warning
`pass`	Otherwise	Passes

The gate works when the check shows pass or no-baseline on a pull request that does not change the agent. Cost, tokens, and latency never affect the verdict. If the verdict is invalid, see Eval errors.

Tune gate (optional)

Input	Default	Use
`thresholds`	`default=0.05`	How far a mean may drop, one line per score: `default=0.05`, `exact_match=0.02`, `tone=off`
`min-regressed-cases`	`2`	Guards against one flaky case failing the build
`max-error-rate`	`0.2`	Share of cases that may error before the run counts as broken
`fail-on`	`regression`	`invalid` reports regressions without failing on them, which is useful while you calibrate
`baseline`	`auto`	`auto`, `none`, or an experiment ID
`baseline-branch`	the default branch	Compare against a release branch instead
`model`	none	Record and prefer baselines for a specific model

The action’s outputs, eval-status, regression-count, experiment-ids, baseline-experiment-ids, and comment-id, let a later step act on the verdict. Scores are assumed to be 0..1, higher is better; invert a score where lower is better, such as a step count, in the scorer.For a suite too slow or costly for every push, trigger only on a label:

on:
  pull_request:
    types: [labeled, synchronize]
jobs:
  eval:
    if: contains(github.event.pull_request.labels.*.name, 'run-evals')

Gate in other CI systems (optional)

Outside GitHub Actions, or against an Atlan deployment other than https://api.atlan.com, run your eval in a plain step, then compare it with a baseline by the same rule the action uses and exit non-zero on a regression.

evals/gate.py

"""Fail the build when a candidate experiment regresses against a baseline.

    python evals/gate.py <baseline_experiment_id> <candidate_experiment_id>

Exit 0 on pass, 1 on a regression, 2 on a broken run.
"""
import os
import sys
from atlanai import AtlanClient

THRESHOLD = 0.05          # a score's mean may drop by at most this much
MIN_REGRESSED_CASES = 2   # ...and at least this many cases must get worse
MAX_ERROR_RATE = 0.2

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)


def results(experiment_id):
    rows, offset = [], 0
    while True:
        page = client.experiments.results.list(experiment_id, limit=500, offset=offset).items
        rows += page
        if len(page) < 500:
            return {r.dataset_record_id or r.case_id: r for r in rows}
        offset += 500


baseline_id, candidate_id = sys.argv[1], sys.argv[2]
if client.experiments.get(candidate_id).experiment_status != "completed":
    print("broken run: the candidate did not complete")
    sys.exit(2)
base, cand = results(baseline_id), results(candidate_id)
errors = sum(1 for r in cand.values() if r.error)
if not cand or errors / len(cand) > MAX_ERROR_RATE:
    print(f"broken run: {errors}/{len(cand)} cases errored")
    sys.exit(2)

shared = base.keys() & cand.keys()
names = {n for r in cand.values() for n in (r.scores or {})}
regressed = False
for name in sorted(names):
    pairs = [((base[k].scores or {}).get(name), (cand[k].scores or {}).get(name)) for k in shared]
    pairs = [(b, c) for b, c in pairs if isinstance(b, (int, float)) and isinstance(c, (int, float))]
    if not pairs:
        continue
    b_mean = sum(b for b, _ in pairs) / len(pairs)
    c_mean = sum(c for _, c in pairs) / len(pairs)
    worse = sum(1 for b, c in pairs if c < b)
    bad = b_mean - c_mean > THRESHOLD and worse >= MIN_REGRESSED_CASES
    regressed |= bad
    print(f"{'REGRESSED' if bad else 'ok':9} {name}: {b_mean:.3f} -> {c_mean:.3f} ({worse} of {len(pairs)} cases worse)")
sys.exit(1 if regressed else 0)

A regression prints a line such as REGRESSED exact_match: 1.000 -> 0.500 (3 of 6 cases worse).To find the baseline, list the latest completed run of the same eval and dataset on your main branch. Outside the action, Eval does not record the branch, so pass it yourself, for example config={"branch": os.environ["CI_BRANCH"]}, and filter on it:

client.experiments.list(config="branch:main", dataset_id=suite.id, experiment_status="completed", sort="-created_at", limit=1)

. config filters take a dotted path; runs made by the action carry the branch at git.branch.

Troubleshooting

Gate failures include an invalid verdict when the command has no credentials, a required check stuck pending because of a paths: filter, and no-baseline on every pull request. See Eval errors.

Next steps

Compare two agent versions: Inspect the cases behind a regression one by one.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Gate pull requests with evals

Before you begin

Troubleshooting

Next steps

See also