> ## Documentation Index
> Fetch the complete documentation index at: https://platform.atlan.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To act on Atlan objects, use the Atlan MCP server at https://api.atlan.com/mcp or the SDK. To install or publish skills from a terminal, use the `aireg` CLI; `aireg --map json` prints its command map. The `atlanai` CLI is deprecated. Run a read-only identity check before any write.
> The docs MCP server at /mcp searches these docs only. It cannot read or change Atlan objects.
> SDK packages: Python `atlanai` (PyPI) and TypeScript `@atlanai/sdk` (npm). Show Python first, then TypeScript.
> These docs cover Atlan Trident, Atlan's hosted product. AI Registry and Agent Registry are its earlier names. Registry OSS is Atlan's open-source agent registry: the registry specification and its aireg CLI, with their own public repository, docs and release notes; /open-source on this site links them. Registry OSS is open source; Atlan Trident is not. Answer generic aireg or specification questions from the project's docs, and use these docs for Atlan-specific setup such as the Atlan adapter, sign-in and workspaces.
> Atlan offers a second product, the Agentic Data Catalog: Atlan's enterprise context layer, covering the data catalog, connectors, lineage, data governance and glossary. It is documented at https://docs.atlan.com; answer questions about it from those docs, not these.

# Evaluate your analytics agent

> Score an analytics agent on real business questions with known answers, see which metric it read for each one, and check that a metric change made answers better before everyone gets it.

Preview

When an agent gives the wrong number, the hard part is knowing what to fix, and whether the fix helped without breaking another answer. An eval answers both with evidence instead of guesses. You score the agent on questions your business really asks, see which metric it read for each one, fix the metric, and run the same questions again to see whether the answer improved.

This guide sets up that loop for an analytics agent that reads [Metric Studio](https://platform.atlan.com/studios/metrics-studio) metrics. It uses the Atlan Trident Evals SDK. For the general eval concepts, see [Evals](https://platform.atlan.com/evals/index.md).

## Before you begin

* **Evals SDK**: The `atlanai` SDK installed, with `ATLAN_API_KEY` and `ATLAN_WORKSPACE_ID` set. See [Run your first eval](https://platform.atlan.com/evals/quickstart.md).
* **Released metrics**: The metrics the questions need show **Live** in Studio. See [Release metrics](https://platform.atlan.com/studios/metrics/how-tos/release-metrics.md).
* **An agent you can call from code**: A function that takes a question and returns the agent's answer. It can wrap Claude, a Cortex agent, or your own agent.
* **Questions with known answers**: 15 to 30 to start. If your team already has an eval set, such as questions with expected SQL or expected numbers, use it.

## Write golden questions

A golden question is one your business asks, with the answer an analyst has confirmed. Good ones:

* **Come from real use.** Take them from executives' requests, dashboards people check, or questions agents got wrong.
* **Have one right answer.** "What was average order value in March 2026?" has one. "How are sales going?" doesn't.
* **Name the metric that should answer them.** That lets you tell a wrong number from a wrong metric.
* **Cover how people phrase things.** Include "AOV", "average basket", and "average order value" if people say all three.

For each question, record the expected number, how close counts as right (for example 1%), and the metric that should answer it.

> [!TIP]
>
> Keep a few questions your metrics don't cover yet. They show whether the agent says it has no metric, or quietly writes its own SQL.

## Push questions to dataset

Store the questions as a dataset in your workspace, so every run is scored on exactly the same cases.

```python theme={null}
import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "analytics-golden-questions", [
    {
        "name": "aov-march-2026",
        "input": {"question": "What was average order value in March 2026?"},
        "expected": {"answer": 84.12, "metric": "average_order_value"},
        "extra": {"tolerance": 0.01},
        "categories": ["sales"],
    },
    {
        "name": "aov-by-region-q1",
        "input": {"question": "What was average basket size in EMEA last quarter?"},
        "expected": {"answer": 91.40, "metric": "average_order_value"},
        "extra": {"tolerance": 0.01},
        "categories": ["sales", "synonym"],
    },
])
```

If the questions come from an existing eval set or spreadsheet, set `source_kind` to `external` and `source_ref` to where they came from. See [Create eval datasets](https://platform.atlan.com/evals/datasets.md).

## Wrap agent as task

Your task calls the agent and returns two things: the number it answered with, and the metrics it read. Record the metrics from the agent's tool calls, where they ran, not from what the agent says: look for `registry__get_artifact` calls on a metric, and `extension__metric__search_metrics` calls that found one.

```python theme={null}
def analytics_task(question: str) -> dict:
    result = my_analytics_agent(question)  # Replace with a call to your agent.
    return {
        "answer": result.number,            # the number the agent answered with
        "metrics_read": result.metric_names, # names of metrics it read over MCP
    }
```

For a full pattern that records tool calls at the tool boundary, see [Score agent behaviour](https://platform.atlan.com/evals/agents.md).

## Score answers

Score each case twice: is the number right, and did the agent read the metric it should have?

```python theme={null}
from atlanai import Eval, ScoreValue


def right_number(*, output, expected, metadata, **_):
    tolerance = (metadata or {}).get("tolerance", 0.01)
    if output["answer"] is None:
        return 0.0
    return float(abs(output["answer"] - expected["answer"]) <= tolerance * abs(expected["answer"]))


def read_expected_metric(*, output, expected, **_):
    read = output["metrics_read"]
    return ScoreValue(score=float(expected["metric"] in read), metadata={"read": read})


run = Eval(
    "analytics-agent",
    dataset=suite.id,
    task=analytics_task,
    scores=[right_number, read_expected_metric],
    config={"candidate": "metrics-baseline"},
)
print(run.experiment_id, run.summary["scores"])
```

The first run is your baseline.

## Read what went wrong

Open the experiment in **Evals**. For each failed case, the two scores tell you where to look:

| Right number | Read expected metric | What it usually means | What to change |
| - | - | - | - |
| No | No | The agent didn't find the metric, often because the question uses words the metric doesn't | Add the words people use to the metric's description, or add a saved query for this kind of question |
| No | Yes | The agent found the metric but the number is wrong | Check the formula, filters and grain in Studio. See [Write or edit a metric](https://platform.atlan.com/studios/metrics/how-tos/write-metrics.md) |
| Yes | No | The agent got there another way, such as its own SQL | Check that's what you want. If not, make the metric easier to find |
| Yes | Yes | Working | Nothing |

Open a case's trace to see every tool call the agent made: what it searched for, which metric version it read, and the SQL it ran.

## Check fix before everyone gets it

1. In Studio, edit the metric. Saving creates a new revision, and agents keep reading the released one.
2. Release the revision on a channel other than `latest`, or pin your test agent to the new version. See [Connect agents to your metrics](https://platform.atlan.com/studios/metrics/how-tos/use-with-agents.md#pin-an-agent-to-one-release-optional).
3. Run the eval again with a new `candidate`, for example `config={"candidate": "aov-fix"}`.
4. Compare the two experiments. See [Compare two agent versions](https://platform.atlan.com/evals/cookbooks/compare-versions.md).
5. If the score went up and nothing else went down, release the revision on `latest`.

Each change you test this way adds to the record: which definition change moved which answers.

## Next steps

* [Turn production failures into test cases](https://platform.atlan.com/evals/cookbooks/production-to-dataset.md): Add the questions agents get wrong in real use.
* [Run evals in CI](https://platform.atlan.com/evals/ci.md): Score every change before it ships.
* [Metric Studio errors](https://platform.atlan.com/studios/metrics/troubleshooting.md): When an agent can't find a metric.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.