Evals

SDK connection

Send evidence from your application's actual executions to use in evaluations and prompt improvement. Use the TypeScript or Python SDK source provided in this repository; these examples do not install a package or send data from this page.

Configure your application

Set OPEN_EVALS_URL to your configured service root without a trailing slash and keep OPEN_EVALS_API_KEY on your server. Hosted trace writes require editor access and either the traces:write or admin scope. The server resolves workspace and project access from authentication; payload fields do not grant access.

TypeScript

Call this function with the trace payload captured by your application.

import { OpenEvalsClient } from "@open-evals/sdk";

type TracePayload = Parameters<OpenEvalsClient["submitTrace"]>[0];

export async function submitExecutionEvidence(payload: TracePayload) {
  const endpoint = process.env.OPEN_EVALS_URL;
  const apiKey = process.env.OPEN_EVALS_API_KEY;
  if (!endpoint || !apiKey) {
    throw new Error("Configure your service URL and project API key.");
  }
  const client = new OpenEvalsClient({ endpoint, apiKey });
  try {
    const response = await client.submitTrace(payload);
    if (!response.ok) {
      throw new Error(`Evidence delivery failed (HTTP ${response.status}).`);
    }
    return response;
  } finally {
    client.close();
  }
}

Python

Use open_evals_sdk.py from python/open_evals_optimizers/src on your Python import path. Pass the actual execution payload.

import os
from typing import Any
from open_evals_sdk import OpenEvalsPythonClient

def submit_execution_evidence(payload: dict[str, Any]) -> int:
    client = OpenEvalsPythonClient(
        endpoint=os.environ["OPEN_EVALS_URL"],
        api_key=os.environ["OPEN_EVALS_API_KEY"],
    )
    return client.submit_trace(payload)

Choose the evidence you send

Use traceId or id for your execution identity, with startedAt, optional endedAt and status. Supply timestamps from the execution.
Include the input and output needed for the evaluation, plus relevant metadata and tags.
Bind prompt evidence with promptDefinitionId, promptVersionId and promptHash when available. Target identity and sessionId can preserve execution context.
Supply spans only when needed. Their id or spanId, name, parentSpanId, startTime, endTime, input, output and attributes describe the captured steps.

Review and minimize evidence before sending it. The SDK sends the supplied payload; server-side redaction does not protect its journey to the service. Do not send credentials or hidden model chain-of-thought. Historical evaluation still requires its resolved data policy.

TypeScript submitTrace returns a Response: check response.ok before recording success. Python submit_trace returns the HTTP status and propagates HTTP or connection errors. For explicit TypeScript buffering, call flush and await it before close; close only stops the timer. Automatic timers do not establish durable delivery.