Add scores from your application

Scores from your application record signals that only exist in production, such as a user’s thumbs-up or a guardrail that fired, on the trace of the request they describe. Use them when you score live traffic from your own code. Eval writes scores for you during an offline run, and hosted judges store their answers as session insights instead.

Before you begin

Before you start, make sure you have:

atlanai SDK with tracing: Installed as in Install atlanai SDK, with init_logger or initLogger set up as in Trace first agent run.
API key and workspace ID: In ATLAN_API_KEY and ATLAN_WORKSPACE_ID. The key must be able to create scorers in the workspace.

Every score cites a scorer ID and version. Create a code scorer for an automated check, or a human scorer for feedback. Do this once, for example at deploy time, and keep each scorer’s id and version_ordinal.Scorer names are unique in the workspace, so the helper reuses a scorer that already has the name. Pick names specific to your application, and give each scorer one meaning: when a check’s logic changes, register a new name or update the scorer to record a new version.

Python
TypeScript

import os
from atlanai import AtlanClient

WORKSPACE = os.environ["ATLAN_WORKSPACE_ID"]
client = AtlanClient(os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
                     bearer_token=os.environ["ATLAN_API_KEY"], workspace=WORKSPACE)


def scorer(name: str, kind: str):
    for s in client.scorers.list(limit=500).items:
        if s.name == name:
            return s
    return client.scorers.create({"workspace_id": WORKSPACE, "name": name, "scorer_kind": kind})


format_check = scorer("answer-format-check", "code")
feedback = scorer("user-feedback", "human")

import { AtlanClient, rawEval } from "@atlanai/sdk";

const workspace = process.env.ATLAN_WORKSPACE_ID!;
const client = new AtlanClient({
  gatewayOrigin: process.env.ATLAN_BASE_URL ?? "https://api.atlan.com",
  bearerToken: process.env.ATLAN_API_KEY!,
  workspace,
});

async function scorer(name: string, kind: rawEval.EvalScorerKind) {
  const existing = (await client.scorers.list({ limit: 500 })).items.find((s) => s.name === name);
  return existing ?? client.scorers.create({ workspaceId: workspace, name, scorerKind: kind });
}

const formatCheck = await scorer("answer-format-check", rawEval.EvalScorerKind.Code);
const feedback = await scorer("user-feedback", rawEval.EvalScorerKind.Human);

Score request while it runs

score attaches a score to the span you call it on. Call it on the root span to score the whole request, or on a child span to score one step, such as a retrieval. Keep the trace ID and root span ID if feedback may arrive later.

Python
TypeScript

from atlanai.tracing import init_logger

tracer = init_logger(project_name="support-agent").client

with tracer.start_as_current_span("handle-request", as_type="task") as root:
    answer = your_agent("Where is my refund?")
    root.update(input="Where is my refund?", output=answer)
    root.score("cites_a_date", value=any(c.isdigit() for c in answer), data_type="BOOLEAN",
               scorer_id=format_check.id, scorer_version=format_check.version_ordinal)
    trace_id, root_span_id = root.trace_id, root.span_id

import { initLogger } from "@atlanai/sdk/tracing";

const logger = initLogger({ projectName: "support-agent" });
const tracer = logger.client;
let traceId = "", rootSpanId = "";

await tracer.startAsCurrentSpan("handle-request", { asType: "task" }, async (root) => {
  const answer = await yourAgent("Where is my refund?");
  root.update({ input: "Where is my refund?", output: answer });
  root.score("cites_a_date", /\d/.test(answer), {
    dataType: "BOOLEAN", scorerId: formatCheck.id, scorerVersion: formatCheck.versionOrdinal,
  });
  traceId = root.traceId;
  rootSpanId = root.spanId;
});

To score the root from a child span in the same process, call score_trace in Python or scoreTrace in TypeScript with the same arguments. If the root span is not in this process, the score attaches to the span you called it on.Match data_type (dataType in TypeScript) to the value:

Data type	Pass	Stored as
`NUMERIC` (default)	A number	The number
`BOOLEAN`	`True` or `False`	`1.0` or `0.0`
`CATEGORICAL`	`string_value="approve"` (`stringValue`)	The label, with value `0.0` unless you pass one. Only numeric values are aggregated.

Use stable, lowercase score names with underscores, such as thumbs_up: statistics aggregate by name. Add comment to record why, and keep user content out of it unless your data policy allows it, because comments are stored on the trace.

Attach feedback after request

Feedback often arrives later, from another process. Open a span in the original trace with the IDs you kept, and score it. The feedback span becomes a child of the original root, so the trace shows the answer and the verdict together.

Python
TypeScript

with tracer.start_as_current_span("user-feedback", as_type="function",
        trace_context={"trace_id": trace_id, "parent_span_id": root_span_id}) as span:
    span.score("thumbs_up", value=1.0, comment="Clear and correct",
               scorer_id=feedback.id, scorer_version=feedback.version_ordinal)
tracer.flush()

await tracer.startAsCurrentSpan("user-feedback",
  { asType: "function", traceContext: { traceId, parentSpanId: rootSpanId } },
  async (span) => {
    span.score("thumbs_up", 1.0, {
      comment: "Clear and correct", scorerId: feedback.id, scorerVersion: feedback.versionOrdinal,
    });
  });
await logger.flush();

Use the trace ID you kept from the original request. A trace_context with an invented trace ID creates a new, orphaned trace.

Verify scores on trace

A score without a valid scorer ID and version is dropped with a warning, not raised, so check that each one arrived.

Check in the app: Open the request’s trace. Each score appears as a child span named score.<name> with score_value, scorer_id, and scorer_version set.
Check in code: When the request belongs to a session, list its spans with client.sessions.traces.list_spans(session_id, trace_id, fields="core,attributes"), or client.sessions.traces.listSpans({ sessionId, traceId, fields: "core,attributes" }) in TypeScript. Spans become readable a few seconds after flush(), so poll briefly.

If a score span is missing, set ATLAN_DEBUG=true and run again: the warning names the missing scorer ID or version. Other causes are in Eval errors.

Troubleshooting

Scores that never appear usually have a missing scorer ID or version, or a value that does not match its data type. See Eval errors. If no traces appear at all, see Install and setup errors.

Next steps

Turn production failures into test cases: Keep a failure your scores found as a repeatable offline case.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Add scores from your application

Before you begin

Troubleshooting

Next steps

See also