> ## Documentation Index
> Fetch the complete documentation index at: https://platform.atlan.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To act on Atlan objects, use the Atlan MCP server at https://api.atlan.com/mcp or the atlanai CLI; `atlanai --map json` prints its command map. Run a read-only identity check before any write.
> The docs MCP server at /mcp searches these docs only. It cannot read or change Atlan objects.
> SDK packages: Python `atlanai` (PyPI) and TypeScript `@atlanai/sdk` (npm). Show Python first, then TypeScript.

# Turn a production failure into a test

> Curate a reviewed session failure into a repeatable offline eval case.

A low online score is a lead for review, not an automatic ground truth label. Read the session and its trace, check whether the scorer was right, then write a case that expresses the expected behavior. Do not copy a full transcript into a dataset record when a short, redacted input reproduces the failure.

## Prepare a reviewed case

For this synthetic example, a session showed an answer that omitted a required safety step. A reviewer distilled it to a question and an expected answer. The `source_ref` links the case to the session from which it was curated.

```python theme={null}
import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

reviewed_session_id = os.environ["ATLAN_REVIEWED_SESSION_ID"]
suite = push_dataset(client, "support-regressions", [
    {
        "name": "reset-link-safety",
        "input": {"question": "How do I reset access to my example account?"},
        "expected": {"value": "Use the verified reset link and check the sender before opening it."},
        "categories": ["account-access", "safety"],
        "source_kind": "session",
        "source_ref": reviewed_session_id,
    },
])
print(suite.dataset.id)
```

`source_kind: "session"` requires a readable session in the same authorized scope. If you cannot retain that link, omit both `source_kind` and `source_ref` and keep the review record in your approved system.

## Run it on the next candidate

Pass `dataset=suite.dataset.id` to `Eval` with the candidate task and your score functions. The experiment pins the dataset snapshot. A later correction to this case creates a new record version and will affect only new experiments.

Review the new case's trace before considering the failure fixed. A correct final sentence can hide a wrong tool call or an unsafe intermediate step; add a scorer for that behavior if it matters to the release decision.