Score agent behaviour

Agent scoring checks both the final answer and the trajectory: which tools the agent called, with which arguments, in how many steps. Use it when an agent can reach the right answer by the wrong path, or the wrong answer by a reasonable one. For answer-only checks, the scorers in Write code scorers are enough.

Before you begin

Before you start, make sure you have:

Working eval: An Eval call that runs, as in Run your first eval.
Agent entry point: A function that takes a user message and returns the agent’s reply, with its tools callable from your code.
Side effects listed: The tools that send email, issue refunds, write to production, or change anything else.

Wrap agent as task

Scorers see the case and, of the run, only what the task returns. Give the agent a thin adapter that returns the answer plus the tool calls it made.Record tool calls at the tool boundary, where they execute, not from what the agent says it did: a model’s own account can differ from what ran. Here each tool is traced and records its calls into a per-case list. The list is a context variable, so concurrent cases do not mix their calls.

import contextvars
import functools
import inspect
import re

from atlanai.tracing import auto_instrument, init_logger, observe

logger = init_logger(project_name="order-agent")   # the service name on traces; nothing to create first
auto_instrument()  # enable installed model and framework instrumentors

_tool_calls = contextvars.ContextVar("tool_calls", default=None)


def recorded(fn):
    """Record each call the agent actually makes: tool name and bound arguments."""
    signature = inspect.signature(fn)

    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        calls = _tool_calls.get()
        if calls is not None:
            calls.append({"name": fn.__name__, "args": dict(signature.bind(*args, **kwargs).arguments)})
        return fn(*args, **kwargs)
    return wrapper


@observe(as_type="tool")
@recorded
def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "shipped", "eta": "Friday"}


def order_agent(message: str, history: list[dict]) -> str:
    # Stand-in for your agent: it decides which tools to call.
    text = " ".join([m["content"] for m in history if m["role"] == "user"] + [message])
    order_id = re.search(r"\d{4}", text).group()
    order = lookup_order(order_id)
    return f"Order {order_id} arrives {order['eta']}."


def order_task(question: str) -> dict:
    calls = []
    token = _tool_calls.set(calls)
    try:
        answer = order_agent(question, [])
    finally:
        _tool_calls.reset(token)
    return {"answer": answer, "tool_calls": calls}

import { AsyncLocalStorage } from "node:async_hooks";
import { initLogger, wrapTraced } from "@atlanai/sdk/tracing";

const logger = initLogger({ projectName: "order-agent" });   // the service name on traces

type ToolCall = { name: string; args: Record<string, unknown> };
const toolCalls = new AsyncLocalStorage<ToolCall[]>();

// Trace the tool, and record each call the agent actually makes.
function recorded<A extends Record<string, unknown>, R>(name: string, fn: (args: A) => Promise<R>) {
  return wrapTraced(async (args: A) => {
    toolCalls.getStore()?.push({ name, args });
    return fn(args);
  }, { name, type: "tool" });
}

const lookupOrder = recorded("lookup_order", async ({ order_id }: { order_id: string }) =>
  ({ order_id, status: "shipped", eta: "Friday" }));

type Message = { role: "user" | "assistant"; content: string };

async function orderAgent(message: string, history: Message[]): Promise<string> {
  // Stand-in for your agent: it decides which tools to call.
  const text = [...history.filter((m) => m.role === "user").map((m) => m.content), message].join(" ");
  const orderId = text.match(/\d{4}/)?.[0] ?? "";
  const order = await lookupOrder({ order_id: orderId });
  return `Order ${orderId} arrives ${order.eta}.`;
}

async function orderTask(question: string) {
  const calls: ToolCall[] = [];
  const answer = await toolCalls.run(calls, () => orderAgent(question, []));
  return { answer, toolCalls: calls };
}

Pass the logger to Eval as trace_logger in Python or logger in TypeScript, so your agent’s spans nest under each case. Model calls, tool spans, tokens, cost, and latency then land on the case’s trace without the task returning them.Model and framework spans appear when their instrumentation is installed. In Python, auto_instrument() enables installed OpenTelemetry instrumentors for providers such as OpenAI and Anthropic, and frameworks such as LangChain, CrewAI, and the OpenAI Agents SDK. In TypeScript, wrap the Vercel AI SDK with wrapAISDK, or set ATLAN_INSTRUMENT=pi to trace a pi agent (see Use pi with Atlan). See Agent frameworks. Every eval case is traced in full: Eval refuses a logger that samples or drops content.

Stub tools with side effects

An eval runs your agent for real. A tool that sends email, issues a refund, or writes to production does that on every case of every run.

Give the agent a stand-in for each such tool. It records the call, so scorers still see it, and returns a plausible result without acting.

@observe(as_type="tool", name="issue_refund")
@recorded
def issue_refund(order_id: str, amount: float) -> dict:
    return {"refund_id": "refund_test", "status": "queued"}   # nothing is refunded

const issueRefund = recorded("issue_refund", async ({ order_id, amount }: { order_id: string; amount: number }) =>
  ({ refund_id: "refund_test", status: "queued", order_id, amount }));   // nothing is refunded

Pass the stand-ins where your agent looks up its tools, such as its tool registry or constructor. Keep each stand-in’s name and signature identical to the real tool’s, so the tool schema the model sees does not change.

Score tool calls

Put what a good trajectory is in each case’s expected: the expected tools and their arguments. These scorers read output["tool_calls"] (output.toolCalls in TypeScript) and expected["tools"] as lists of {name, args}. Both languages read the same expected keys, so one dataset serves both.

from atlanai import ScoreValue


def tool_choice(*, output, expected, **_):
    """Did the agent call exactly the expected tools, in order?"""
    got = [c["name"] for c in output["tool_calls"]]
    want = [c["name"] for c in expected["tools"]]
    return ScoreValue(score=float(got == want), metadata={"called": got, "expected": want})


def tool_recall(*, output, expected, **_):
    """Share of expected tools that were called, ignoring order and extras."""
    got = {c["name"] for c in output["tool_calls"]}
    want = {c["name"] for c in expected["tools"]}
    return len(got & want) / len(want) if want else None


def tool_arguments(*, output, expected, **_):
    """Share of expected calls made with exactly those arguments. Each actual call matches once."""
    unmatched = list(output["tool_calls"])
    hits = 0
    for want in expected["tools"]:
        if want in unmatched:
            unmatched.remove(want)
            hits += 1
    return hits / len(expected["tools"]) if expected["tools"] else None


def step_budget(*, output, expected, **_):
    """1.0 at or under the expected number of calls, falling off after."""
    budget = max(len(expected["tools"]), 1)
    return min(1.0, budget / max(len(output["tool_calls"]), 1))


def answer_contains(*, output, expected, **_):
    return float(expected["answer_contains"].lower() in output["answer"].lower())

type Output = { answer: string; toolCalls: ToolCall[] };
type Expected = { tools: ToolCall[]; answer_contains: string };
type Args = { output: Output; expected?: Expected };
// Compare as JSON with object keys sorted: stored `expected` values do not keep key order.
const canonical = (v: unknown): unknown =>
  Array.isArray(v) ? v.map(canonical)
  : v !== null && typeof v === "object"
    ? Object.fromEntries(Object.entries(v).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)).map(([k, x]) => [k, canonical(x)]))
    : v;
const same = (a: unknown, b: unknown) => JSON.stringify(canonical(a)) === JSON.stringify(canonical(b));

const toolChoice = {
  name: "tool_choice",
  scorer: ({ output, expected }: Args) => {
    if (!expected) return null;
    const got = output.toolCalls.map((c) => c.name);
    const want = expected.tools.map((c) => c.name);
    return { score: Number(same(got, want)), metadata: { called: got, expected: want } };
  },
};

const toolRecall = {
  name: "tool_recall",
  scorer: ({ output, expected }: Args) => {
    if (!expected?.tools.length) return null;
    const got = new Set(output.toolCalls.map((c) => c.name));
    return expected.tools.filter((c) => got.has(c.name)).length / expected.tools.length;
  },
};

const toolArguments = {
  name: "tool_arguments",
  scorer: ({ output, expected }: Args) => {
    if (!expected?.tools.length) return null;
    const unmatched = [...output.toolCalls];
    let hits = 0;
    for (const want of expected.tools) {
      const i = unmatched.findIndex((c) => same(c, want));
      if (i >= 0) { unmatched.splice(i, 1); hits++; }
    }
    return hits / expected.tools.length;
  },
};

const stepBudget = {
  name: "step_budget",
  scorer: ({ output, expected }: Args) =>
    expected ? Math.min(1, Math.max(expected.tools.length, 1) / Math.max(output.toolCalls.length, 1)) : null,
};

const answerContains = {
  name: "answer_contains",
  scorer: ({ output, expected }: Args) =>
    expected ? Number(output.answer.toLowerCase().includes(expected.answer_contains.toLowerCase())) : null,
};

In TypeScript, declare expected optional and return null when it is missing; a required expected fails tsc --strict. Compare structured values with same, not raw JSON.stringify: expected read from a dataset comes back with its object keys in a different order, and a raw comparison scores 0 on every case. tool_arguments matches each expected call to at most one actual call, so one call cannot satisfy two expected calls to the same tool.Pick the scorers for the questions you need answered:

Question	Scorer	When it matters
Did it pick the right tools?	`tool_choice` (strict order) or `tool_recall` (set)	Always. Use strict order only when order is part of correctness.
Did it call them correctly?	`tool_arguments`	When the wrong ID or filter silently returns a plausible wrong answer.
Did it take a sane path?	`step_budget`	When loops, retries, or redundant calls cost money or latency.
Is the final answer right?	`answer_contains`, exact match or required facts, or an LLM judge in your own code	Always. The trajectory scores explain it.
Did it refuse or leak?	Pattern and refusal checks from the scorer catalog	For user-facing agents

Compare arguments on the fields that matter. Normalise or drop volatile ones, such as timestamps and request IDs, or the score measures noise.

Run eval on agent

Run the agent, the scorers, and Eval from one file. Set subject_kind and subject_id so the run appears on the agent’s page in the app, and so you can list its history.

import os
from atlanai import AtlanClient, Eval

run = Eval(
    "order-agent",
    data=[{
        "id": "eta",
        "input": "When will order 2210 arrive?",
        "expected": {"tools": [{"name": "lookup_order", "args": {"order_id": "2210"}}],
                     "answer_contains": "Friday"},
    }],
    task=order_task,
    scores=[tool_choice, tool_recall, tool_arguments, step_budget, answer_contains],
    trace_logger=logger,
    subject_kind="agent",
    subject_id="agent_01example",
)
print(run.summary["scores"])

client = AtlanClient(os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
                     bearer_token=os.environ["ATLAN_API_KEY"], workspace=os.environ["ATLAN_WORKSPACE_ID"])
history = client.experiments.list(subject_kind="agent", subject_id="agent_01example")

import { Eval } from "@atlanai/sdk";

const run = await Eval("order-agent", {
  data: () => [{
    id: "eta",
    input: "When will order 2210 arrive?",
    expected: { tools: [{ name: "lookup_order", args: { order_id: "2210" } }], answer_contains: "Friday" },
  }],
  task: orderTask,
  scores: [toolChoice, toolRecall, toolArguments, stepBudget, answerContains],
}, { logger, subjectKind: "agent", subjectId: "agent_01example" });
console.log(run.summary.scores);

subject_kind is agent, harness, or skill, and subject_id must be an artifact of that kind you can read.The experiment does not record which version of the agent ran. Put the agent version and prompt revision in config, or pin them with a context manifest. In the CI action, Python Eval records the commit, branch, and pull request in config.git, and the action’s model input in config.model, unless your config already has a git or model key; a TypeScript eval records them only through the results-file shim.If the task raises, that case records an error and its scorers do not run; the run still returns. If no case produces a score, Eval completes the experiment and then raises, with the experiment ID on the exception. Pass require_scores=False (requireScores: false) to get the result back instead.

Verify trajectory and cost

Check scores: No entry in run.results has an error, and run.summary["scores"] has one entry per scorer.

Check trace: Each case’s trace has this shape:

eval.case.1            task
├── eval.task          task       ← your agent runs here
│   ├── chat …         llm        ← from an instrumented model client
│   └── lookup_order   tool
├── scorer.tool_choice function
└── score.tool_choice  score      ← the score, citing the scorer version

Check cost and tokens: They live on the traces, not in the summary. Query them per experiment with trace stats. A scalar query returns one number per measure; a table query grouped by score_name returns the mean of each score across the run’s score spans.

import time

now = int(time.time())
window = {"start_time_unix_seconds": now - 86_400, "end_time_unix_seconds": now}

totals = client.experiments.traces.query_stats(run.experiment_id, {
    **window, "request_type": "scalar",
    "queries": [{"name": "cost", "measure": "cost"},
                {"name": "tokens", "measure": "tokens"},
                {"name": "p90", "measure": "latency_p90"}],
})
for r in totals.data.actual_instance.results:
    print(r.query_name, r.value, r.unit)

Token and cost values come from model spans that report usage; an agent without instrumented model calls shows 0. To enforce a budget, compare these numbers in your CI script. The CI action reports them but never fails a check on them.

Agents vary run to run, and Eval runs each case once. To measure variance, run the same eval several times with a trial field in config and compare the means:

means = []
for trial in range(3):
    r = Eval("order-agent", dataset="order-agent-cases", task=order_task,
             scores=[tool_choice], config={"model": "example-model", "trial": trial})
    means.append(r.summary["scores"]["tool_choice"]["mean"])
print(min(means), max(means))

Set model temperature to 0 where your agent allows it. A gate threshold narrower than the run-to-run spread fails at random: widen it, or gate on more cases.

Score multi-turn conversation (optional)

Put the user turns in input as a list. The task replays them against your agent with the history so far, and returns the final reply as answer plus every tool call across the turns. It keeps the single-turn output shape, so the scorers above work on both.

# Uses _tool_calls, logger, and order_agent from the adapter above.


def conversation_task(turns: list[str]) -> dict:
    history, replies, calls = [], [], []
    token = _tool_calls.set(calls)
    try:
        for user_turn in turns:
            reply = order_agent(user_turn, history)   # your agent, with the history so far
            history += [{"role": "user", "content": user_turn}, {"role": "assistant", "content": reply}]
            replies.append(reply)
    finally:
        _tool_calls.reset(token)
    return {"answer": replies[-1], "replies": replies, "tool_calls": calls}


def remembers_order_id(*, output, **_):
    # The order ID from turn 1 should still be used in the last reply.
    return float("2210" in output["answer"])


run = Eval(
    "order-agent-multi-turn",
    data=[{"id": "follow-up", "input": ["Where is order 2210?", "And when will it arrive?"]}],
    task=conversation_task,
    scores=[remembers_order_id],
    trace_logger=logger,
)
print(run.summary["scores"])

async function conversationTask(turns: string[]) {
  const history: Message[] = [];
  const replies: string[] = [];
  const calls: ToolCall[] = [];
  await toolCalls.run(calls, async () => {
    for (const userTurn of turns) {
      const reply = await orderAgent(userTurn, history);   // your agent, with the history so far
      history.push({ role: "user", content: userTurn }, { role: "assistant", content: reply });
      replies.push(reply);
    }
  });
  return { answer: replies[replies.length - 1], replies, toolCalls: calls };
}

const multi = await Eval("order-agent-multi-turn", {
  data: () => [{ id: "follow-up", input: ["Where is order 2210?", "And when will it arrive?"] }],
  task: conversationTask,
  scores: [{ name: "remembers_order_id", scorer: ({ output }: { output: { answer: string } }) => Number(output.answer.includes("2210")) }],
}, { logger });
console.log(multi.summary.scores);

Scorers can judge the final turn, every turn, or the whole trajectory. In a dataset, store the turns under the input_key, for example {"question": ["...", "..."]}. To serve single-turn and multi-turn records from one dataset, normalise first: turns = question if isinstance(question, list) else [question].

Troubleshooting

Agent eval failures include a task that raises on every case, a TypeScript comparison that scores 0 because of key order, and a logger that samples or drops content. See Eval errors.

Next steps

Gate pull requests with evals: Run this eval on every pull request and fail the check on a regression.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Before you begin

Troubleshooting

Next steps

See also