Run your first eval

A first eval gives you a stored experiment you can open, and proof that your credentials, workspace, and atlanai SDK work together. Use this guide before you connect a real agent: its script uses a stub that always answers "4", so one case passes and one fails. Once it works, Score agent behaviour replaces the stub with your agent.

Before you begin

Before you start, make sure you have:

Runtime: Python 3.10 or later, or Node.js 20 or later.
API key: A key that can create datasets, scorers, experiments, and results in your workspace, stored in your secret store. See Send bearer token.
Workspace ID: In the web app or Atlan Trident Desktop, open the workspace and select Copy workspace ID from the More actions (⋯) menu at the top of the page.
Evals access: Evals appears in the app. If it does not, ask your administrator to turn it on for your organization.

Install atlanai SDK

Install the SDK with its tracing packages. Every eval case is traced, so Python needs the tracing extra. The TypeScript package includes tracing.

Python
TypeScript

pip install -U 'atlanai[tracing]'

npm install @atlanai/sdk@latest

Set API key and workspace

Eval reads its credentials and workspace from the environment. Load the key from your secret store; do not paste it into source files or shell history.

export ATLAN_API_KEY="<token-from-your-secret-store>"
export ATLAN_WORKSPACE_ID="workspace_01example"

Set ATLAN_BASE_URL only when you use a gateway other than https://api.atlan.com.

Create two-case eval script

The script defines two cases, a task that stands in for your agent, and one scorer that returns 1 for a match and 0 otherwise. Keep the eval name, answer-regression, the same on every run: scorer identity and CI baselines are keyed on it.

Python
TypeScript

Save as eval_quickstart.py:

from atlanai import Eval


def answer(question: str) -> str:
    return "4"  # Replace with a call to your agent.


def accuracy(*, output: str, expected: str, **_) -> float:
    return float(output == expected)


run = Eval(
    "answer-regression",
    data=lambda: [
        {"id": "arithmetic", "input": "What is 2 + 2?", "expected": "4"},
        {"id": "geography", "input": "What is the capital of France?", "expected": "Paris"},
    ],
    task=answer,
    scores=[accuracy],
    config={"candidate": "example-v1"},
)

print(run.experiment_id, run.summary["scores"])
for case in run.results:
    print(case.case_id, case.scores, case.trace_id)

Save as eval-quickstart.ts:

import { Eval } from "@atlanai/sdk";

const run = await Eval("answer-regression", {
  data: () => [
    { id: "arithmetic", input: "What is 2 + 2?", expected: "4" },
    { id: "geography", input: "What is the capital of France?", expected: "Paris" },
  ],
  task: async (_question: string) => "4", // Replace with a call to your agent.
  scores: [{
    name: "accuracy",
    scorer: ({ output, expected }) => Number(output === expected),
  }],
}, { config: { candidate: "example-v1" } });

console.log(run.experimentId, run.summary.scores);
for (const c of run.results) console.log(c.caseId, c.scores, c.traceId);

Run eval script

The script runs both cases in your process, uploads one result and one trace per case, and completes the experiment.

Python
TypeScript

python eval_quickstart.py

npx tsx eval-quickstart.ts

The output shows the experiment ID, the score summary, and one line per case. Your IDs differ:

experiment_01example {'accuracy': {'mean': 0.5, 'count': 2}}
arithmetic {'accuracy': 1.0} 260d10c2be18fdf5f3ef6322a4d1430e
geography {'accuracy': 0.0} 2dff00b3bfd7cf25a2b0ce8409468d0c

Copy the experiment ID. You use it in the next step.

Verify experiment in Atlan Trident

The printed output comes from your process. Confirm that Atlan Trident stored the same results.

Check in the app: Open Evals, then Experiments, and select the experiment. It shows two cases. Select geography to see its trace: the input the task received and the answer it returned.

Check in code: List the stored results. AtlanClient does not read the environment, so pass each value.

Python
TypeScript

import os
from atlanai import AtlanClient

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)
for r in client.experiments.results.list("<experiment-id>").items:
    print(r.case_id, r.scores, r.trace_id)

import { AtlanClient } from "@atlanai/sdk";

const client = new AtlanClient({
  gatewayOrigin: process.env.ATLAN_BASE_URL ?? "https://api.atlan.com",
  bearerToken: process.env.ATLAN_API_KEY!,
  workspace: process.env.ATLAN_WORKSPACE_ID!,
});
const page = await client.experiments.results.list("<experiment-id>");
for (const r of page.items) console.log(r.caseId, r.scores, r.traceId);

The eval worked when both views show arithmetic with accuracy 1.0 and geography with accuracy 0.0, each with a trace ID. If the script raised Eval() cannot run inside an event loop, see Eval errors.

Troubleshooting

First-run failures include an Eval call inside an event loop, tracing turned off with ATLAN_TRACING, and a scorer name error from an older SDK. See Eval errors for each error and its fix.

Next steps

Score agent behaviour: Replace the stub with your agent and score the tools it calls.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Before you begin

Troubleshooting

Next steps

See also