Scores from your application

Some signals only exist in production: a user’s thumbs-up, a support agent’s correction, a guardrail that fired. Record them as scores on the request’s trace. They sit next to that request’s latency, cost, and tool calls, and trace statistics aggregate them by score name. This is separate from Eval, which writes scores for you, and from hosted judges, which store their answers as insights.

1. Register the scorer once

Every score cites a scorer ID and version. Create a code scorer for an automated check, or a human scorer for feedback. Do this once, for example at deploy time, and keep the ID and version_ordinal. Scorer names are unique in the workspace, so the helper below reuses any scorer that already has the name; pick names specific to your application.

import os
from atlanai import AtlanClient

WORKSPACE = os.environ["ATLAN_WORKSPACE_ID"]
client = AtlanClient(os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
                     bearer_token=os.environ["ATLAN_API_KEY"], workspace=WORKSPACE)


def scorer(name: str, kind: str):
    for s in client.scorers.list(limit=500).items:
        if s.name == name:
            return s
    return client.scorers.create({"workspace_id": WORKSPACE, "name": name, "scorer_kind": kind})


format_check = scorer("answer-format-check", "code")
feedback = scorer("user-feedback", "human")

import { AtlanClient, rawEval } from "@atlanai/sdk";

const workspace = process.env.ATLAN_WORKSPACE_ID!;
const client = new AtlanClient({
  gatewayOrigin: process.env.ATLAN_BASE_URL ?? "https://api.atlan.com",
  bearerToken: process.env.ATLAN_API_KEY!,
  workspace,
});

async function scorer(name: string, kind: rawEval.EvalScorerKind) {
  const existing = (await client.scorers.list({ limit: 500 })).items.find((s) => s.name === name);
  return existing ?? client.scorers.create({ workspaceId: workspace, name, scorerKind: kind });
}

const formatCheck = await scorer("answer-format-check", rawEval.EvalScorerKind.Code);
const feedback = await scorer("user-feedback", rawEval.EvalScorerKind.Human);

A score without a valid scorer ID and version is dropped with a warning, not raised, so scoring never breaks a request. Set ATLAN_DEBUG=true to see the warnings while you wire this up.

2. Score while the request runs

score attaches a score to the span you call it on. Call it on the root span to score the whole request, or on a child span to score one step, such as a retrieval’s precision. In Python, score_trace on any span scores the root. init_logger’s project_name is the service name on your traces; there is nothing to create first.

from atlanai.tracing import init_logger

tracer = init_logger(project_name="support-agent").client

with tracer.start_as_current_span("handle-request", as_type="task") as root:
    answer = your_agent("Where is my refund?")
    root.update(input="Where is my refund?", output=answer)
    root.score("cites_a_date", value=any(c.isdigit() for c in answer), data_type="BOOLEAN",
               scorer_id=format_check.id, scorer_version=format_check.version_ordinal)
    trace_id, root_span_id = root.trace_id, root.span_id   # keep these to attach feedback later

import { initLogger } from "@atlanai/sdk/tracing";

const logger = initLogger({ projectName: "support-agent" });
const tracer = logger.client;
let traceId = "", rootSpanId = "";   // keep these to attach feedback later

await tracer.startAsCurrentSpan("handle-request", { asType: "task" }, async (root) => {
  const answer = await yourAgent("Where is my refund?");
  root.update({ input: "Where is my refund?", output: answer });
  root.score("cites_a_date", /\d/.test(answer), {
    dataType: "BOOLEAN", scorerId: formatCheck.id, scorerVersion: formatCheck.versionOrdinal,
  });
  traceId = root.traceId;
  rootSpanId = root.spanId;
});

`data_type`	Pass	Stored as
`NUMERIC` (default)	A number	The number
`BOOLEAN`	`True`/`False`	`1.0`/`0.0`
`CATEGORICAL`	`string_value="approve"`	The label, with value `0.0` unless you pass one. Labels are kept on the span, but only numeric values are aggregated.

Pass comment="..." with any score to record why.

3. Attach feedback after the request finished

Feedback often arrives minutes later, from another process. Keep the request’s trace_id and root span_id, for example on your message record. Then open a span in that trace and score it:

with tracer.start_as_current_span("user-feedback", as_type="function",
        trace_context={"trace_id": trace_id, "parent_span_id": root_span_id}) as span:
    span.score("thumbs_up", value=1.0, comment="Clear and correct",
               scorer_id=feedback.id, scorer_version=feedback.version_ordinal)
tracer.flush()

await tracer.startAsCurrentSpan("user-feedback",
  { asType: "function", traceContext: { traceId, parentSpanId: rootSpanId } },
  async (span) => {
    span.score("thumbs_up", 1.0, {
      comment: "Clear and correct", scorerId: feedback.id, scorerVersion: feedback.versionOrdinal,
    });
  });
await logger.flush();

Use the IDs you kept from the original request. A trace_context with an invented trace ID creates a new, orphaned trace instead. The feedback span becomes a child of the original root, so the request’s trace shows the answer and the verdict together.

Read scores back

Each score is a score.<name> span in the request’s trace, a child of the span it was attached to, with score_value, scorer_id, and scorer_version set. Scores show on the trace in the Atlan app, and trace statistics aggregate score_value by score_name, alongside cost and latency. When the request belongs to a session, read the spans back with client.sessions.traces.list_spans(session_id, trace_id, fields="core,attributes"), or in TypeScript client.sessions.traces.listSpans({ sessionId, traceId, fields: "core,attributes" }). Spans become readable a few seconds after flush(), so poll briefly; see make app traffic scoreable. When a production score finds a failure worth keeping, turn it into a test case.

Keep scores consistent

Stable, lowercase names with underscores, such as thumbs_up and cites_a_date. Aggregation is by name.
One meaning per scorer. When a check’s logic changes, register a new scorer name, or edit it through PATCH /eval/v1/scorers/{id} to record a new version and cite the new version_ordinal.
No user content in comments beyond what your data policy allows. Comments are stored on the trace.

Build an eval

Run and gate

Score production

Reference

1. Register the scorer once

2. Score while the request runs

3. Attach feedback after the request finished

Read scores back

Keep scores consistent

​1. Register the scorer once

​2. Score while the request runs

​3. Attach feedback after the request finished

​Read scores back

​Keep scores consistent

1. Register the scorer once

2. Score while the request runs

3. Attach feedback after the request finished

Read scores back

Keep scores consistent