Evaluate your analytics agent

Preview When an agent gives the wrong number, the hard part is knowing what to fix, and whether the fix helped without breaking another answer. An eval answers both with evidence instead of guesses. You score the agent on questions your business really asks, see which metric it read for each one, fix the metric, and run the same questions again to see whether the answer improved. This guide sets up that loop for an analytics agent that reads Metric Studio metrics. It uses the Atlan Trident Evals SDK. For the general eval concepts, see Evals.

Before you begin

Evals SDK: The atlanai SDK installed, with ATLAN_API_KEY and ATLAN_WORKSPACE_ID set. See Run your first eval.
Released metrics: The metrics the questions need show Live in Studio. See Release metrics.
An agent you can call from code: A function that takes a question and returns the agent’s answer. It can wrap Claude, a Cortex agent, or your own agent.
Questions with known answers: 15 to 30 to start. If your team already has an eval set, such as questions with expected SQL or expected numbers, use it.

Write golden questions

A golden question is one your business asks, with the answer an analyst has confirmed. Good ones:

Come from real use. Take them from executives’ requests, dashboards people check, or questions agents got wrong.
Have one right answer. “What was average order value in March 2026?” has one. “How are sales going?” doesn’t.
Name the metric that should answer them. That lets you tell a wrong number from a wrong metric.
Cover how people phrase things. Include “AOV”, “average basket”, and “average order value” if people say all three.

For each question, record the expected number, how close counts as right (for example 1%), and the metric that should answer it.

Keep a few questions your metrics don’t cover yet. They show whether the agent says it has no metric, or quietly writes its own SQL.

Push questions to dataset

Store the questions as a dataset in your workspace, so every run is scored on exactly the same cases.

import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "analytics-golden-questions", [
    {
        "name": "aov-march-2026",
        "input": {"question": "What was average order value in March 2026?"},
        "expected": {"answer": 84.12, "metric": "average_order_value"},
        "extra": {"tolerance": 0.01},
        "categories": ["sales"],
    },
    {
        "name": "aov-by-region-q1",
        "input": {"question": "What was average basket size in EMEA last quarter?"},
        "expected": {"answer": 91.40, "metric": "average_order_value"},
        "extra": {"tolerance": 0.01},
        "categories": ["sales", "synonym"],
    },
])

If the questions come from an existing eval set or spreadsheet, set source_kind to external and source_ref to where they came from. See Create eval datasets.

Wrap agent as task

Your task calls the agent and returns two things: the number it answered with, and the metrics it read. Record the metrics from the agent’s tool calls, where they ran, not from what the agent says: look for registry__get_artifact calls on a metric, and extension__metric__search_metrics calls that found one.

def analytics_task(question: str) -> dict:
    result = my_analytics_agent(question)  # Replace with a call to your agent.
    return {
        "answer": result.number,            # the number the agent answered with
        "metrics_read": result.metric_names, # names of metrics it read over MCP
    }

For a full pattern that records tool calls at the tool boundary, see Score agent behaviour.

Score answers

Score each case twice: is the number right, and did the agent read the metric it should have?

from atlanai import Eval, ScoreValue


def right_number(*, output, expected, metadata, **_):
    tolerance = (metadata or {}).get("tolerance", 0.01)
    if output["answer"] is None:
        return 0.0
    return float(abs(output["answer"] - expected["answer"]) <= tolerance * abs(expected["answer"]))


def read_expected_metric(*, output, expected, **_):
    read = output["metrics_read"]
    return ScoreValue(score=float(expected["metric"] in read), metadata={"read": read})


run = Eval(
    "analytics-agent",
    dataset=suite.id,
    task=analytics_task,
    scores=[right_number, read_expected_metric],
    config={"candidate": "metrics-baseline"},
)
print(run.experiment_id, run.summary["scores"])

The first run is your baseline.

Read what went wrong

Open the experiment in Evals. For each failed case, the two scores tell you where to look:

Right number	Read expected metric	What it usually means	What to change
No	No	The agent didn’t find the metric, often because the question uses words the metric doesn’t	Add the words people use to the metric’s description, or add a saved query for this kind of question
No	Yes	The agent found the metric but the number is wrong	Check the formula, filters and grain in Studio. See Write or edit a metric
Yes	No	The agent got there another way, such as its own SQL	Check that’s what you want. If not, make the metric easier to find
Yes	Yes	Working	Nothing

Open a case’s trace to see every tool call the agent made: what it searched for, which metric version it read, and the SQL it ran.

Check fix before everyone gets it

In Studio, edit the metric. Saving creates a new revision, and agents keep reading the released one.
Release the revision on a channel other than latest, or pin your test agent to the new version. See Connect agents to your metrics.
Run the eval again with a new candidate, for example config={"candidate": "aov-fix"}.
Compare the two experiments. See Compare two agent versions.
If the score went up and nothing else went down, release the revision on latest.

Each change you test this way adds to the record: which definition change moved which answers.

Next steps

Turn production failures into test cases: Add the questions agents get wrong in real use.
Run evals in CI: Score every change before it ships.
Metric Studio errors: When an agent can’t find a metric.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Evaluate your analytics agent

Before you begin

Write golden questions

Push questions to dataset

Wrap agent as task

Score answers

Read what went wrong

Check fix before everyone gets it

Next steps