For coding agents

​

Load the docs

• Every page is Markdown at <page>.md, for example https://platform.atlan.com/evals/quickstart.md. https://platform.atlan.com/llms.txt lists every page with its description.
• Connect the docs MCP server so the agent can search current docs instead of relying on training data: claude mcp add --transport http atlan-docs https://platform.atlan.com/mcp. See For AI agents for other clients.
• Read in this order: quickstart, concepts, scorers, agents, CI.

​

A brief to paste

Set up Atlan evals for this repository's agent.

1. Find the agent's entry point: the function that takes a user request and returns the
   answer. Write a thin adapter `task(input) -> {"answer": ..., "tool_calls": [{"name", "args"}]}`
   (`toolCalls` in TypeScript). Record tool calls where the tools execute, not from the
   model's own account. Replace tools with side effects by recording stand-ins.
2. Create evals/cases.py with 10–30 cases covering the main behaviours and known failures.
   Each case: a stable name, input {"question": ...}, and expected (the answer facts and
   the tools that should be called). Push them with atlanai.push_dataset.
3. Write deterministic scorers first (exact or required facts, tool choice, tool
   arguments). Each returns 0..1, or None when it does not apply. Add an LLM-judge scorer
   only for what code cannot decide.
4. Create evals/run.py (with an empty evals/__init__.py) that pushes the dataset and calls
   atlanai.Eval with a fixed eval name. Run it locally with `python -m evals.run`; every case
   must finish without an error. For TypeScript, create evals/run.ts with pushDataset and
   Eval from @atlanai/sdk, run it with `npx tsx evals/run.ts`, and make it pass
   `tsc --strict`.
5. Add .github/workflows/atlan-evals.yml using atlanai/agent-registry-action/eval,
   triggered on pull_request and on push to main, plus a requirements.txt with the agent's
   dependencies (or the TypeScript workflow in the CI page). Set ATLAN_API_KEY and
   ATLAN_WORKSPACE_ID in the action step's env: the action does not pass its api-key
   input to the command. A TypeScript eval must also write the results file.
6. Do not hardcode credentials. Read ATLAN_API_KEY and ATLAN_WORKSPACE_ID from the
   environment; the workflow reads them from secrets.ATLAN_API_KEY and
   vars.ATLAN_WORKSPACE_ID.

Docs: https://platform.atlan.com/evals/for-coding-agents.md. Stop and ask if the agent has side
effects (sends email, writes to production) that an eval run would trigger.

​

Invariants

1. Atlan does not run the agent. Eval executes in the local process or CI job. There is no hosted runner to configure.
2. Install the latest SDK. pip install -U 'atlanai[tracing]' for Python, npm install @atlanai/sdk@latest for TypeScript. The tracing extra is required in Python.
3. Eval reads ATLAN_API_KEY, ATLAN_WORKSPACE_ID, and ATLAN_BASE_URL. AtlanClient reads nothing from the environment; pass its arguments explicitly.
4. Keep the eval name fixed. Scorer identity and CI baselines key on it. Put variants in config.
5. Scorers return 0..1, higher is better, or None to abstain. Never return a free 1.0 for a case a rule does not cover.
6. Results are create-only and finished experiments are frozen. To fix a run, start a new one. summary.scores is computed by Atlan, not written by you.
7. The task sees only input, and scorers see only what the task returns. Return the tool calls from the task if scorers need them. Record them at the tool, not from the agent’s self-report.
8. Dataset records need an object input with the question under input_key (default question). {"value": x} as expected reaches the scorer as x.
9. Hosted llm_judge scorers score recorded sessions, not Eval cases. For a judge inside Eval, call a model from a code scorer.
10. Python Eval cannot run inside an event loop. Use await async_eval(...) in notebooks and async code.
11. The CI action reads experiments that Python Eval reports. A TypeScript eval needs the results-file shim. The action’s api-key never reaches the command; set ATLAN_API_KEY and ATLAN_WORKSPACE_ID in the step’s env.
12. An agent version is not pinned automatically. Record the commit and model in config.
13. Upgrade the SDK before debugging a registration error. “reuses a Registry name with a different identity” means the SDK is older than the gateway.
14. Hosted judges need sessions. App traffic is judgeable only after it is recorded as a session whose external_session_id equals the spans’ session ID.
15. Compare structured values key-order-insensitively in TypeScript. Stored expected values do not keep key order, so raw JSON.stringify equality scores 0.
16. Dataset names are workspace-wide. Pushing to an existing name adds to that dataset; check suite.created.

​

Task map

​

Definition of done

• python -m evals.run exits 0, or npx tsx evals/run.ts does and npx tsc --strict --noEmit passes. Every case in run.results has no error.
• run.summary["scores"] (run.summary.scores) has one entry per scorer, each with a count equal to the number of cases the scorer applies to.
• verify_experiment(client, run.experiment_id).raise_for_status() passes, or in TypeScript (await verifyExperiment(client, run.experimentId)).raiseForStatus(): every result has a queryable trace, and the sampled traces carry their score spans.
• The eval name in Eval(...) is a constant, not derived from the branch or the time.
• No credential appears in the repository. git grep -n "ATLAN_API_KEY=" finds only placeholders.
• .github/workflows/atlan-evals.yml triggers on pull_request and on push to the default branch, sets ATLAN_API_KEY and ATLAN_WORKSPACE_ID in the action step’s env, grants only contents: read and pull-requests: write, and skips fork PRs.
• The human has been told to add the ATLAN_API_KEY secret and the ATLAN_WORKSPACE_ID variable, and to run the workflow once on main to create the baseline.