Turn production failures into test cases

A reviewed production failure becomes a dataset case that every later run and pull request is checked against, linked back to the session it came from. Use it when a hosted judge or a user report surfaces a failure that must not happen again. A low online score is a lead for review, not ground truth: confirm the failure before you turn it into a case.

Before you begin

Before you start, make sure you have:

Failed session: A session that a hosted judge or a person flagged. To find candidates, score recent traffic and read the lowest answers under Evals → Scorers in the app.
Dataset: A dataset for regression cases, or a name for a new one. See Create eval datasets.

Review session and trace

Read the session and its trace, and check whether the scorer was right. Then decide what the expected behaviour is. Do not copy a full transcript into a record when a short, redacted input reproduces the failure.

Push reviewed case

In this synthetic example, a session showed an answer that omitted a required safety step. A reviewer distilled it to a question and an expected answer. source_ref links the case to the session it came from.

import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

reviewed_session_id = os.environ["ATLAN_REVIEWED_SESSION_ID"]
suite = push_dataset(client, "support-regressions", [
    {
        "name": "reset-link-safety",
        "input": {"question": "How do I reset access to my example account?"},
        "expected": {"value": "Use the verified reset link and check the sender before opening it."},
        "categories": ["account-access", "safety"],
        "source_kind": "session",
        "source_ref": reviewed_session_id,
    },
])
print(suite.id)

source_kind: "session" requires a session in the same workspace. If you cannot keep that link, omit both source_kind and source_ref, and keep the review record in your approved system. Set provenance on the first push: a later push does not change source_kind or source_ref on an existing record.

Run case on next candidate

Pass dataset=suite.id to Eval with the candidate task and your scorers, or let your CI workflow pick it up on the next pull request. The experiment pins the dataset snapshot, so a later correction to this case creates a new record version and affects only new experiments.

Verify failure is fixed

Open the new case’s trace before you consider the failure fixed. A correct final sentence can hide a wrong tool call or an unsafe intermediate step. The failure is fixed when the case scores as expected and its trace shows the right steps; if a step matters to the release decision, add a scorer for it.

Troubleshooting

If push_dataset refuses the record, check that the session is in the same workspace. For other dataset errors, see Eval errors.

Next steps

Gate pull requests with evals: Gate every pull request on the regression dataset.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Turn production failures into test cases

Before you begin

Troubleshooting

Next steps

See also