Evaluation quickstart

Evaluations let you systematically measure AI quality. Compare approaches, catch regressions before deployment, and validate improvements with data instead of intuition. Each evaluation consists of three components:

Data - A dataset of test cases with inputs and expected outputs
Task - An AI function you want to test
Scores - Scoring functions that measure output quality

Set up your environment and run evals with the Braintrust SDK.If you don’t have a Braintrust account, sign up for free at braintrust.dev.

2. Get API keys

Create API keys for:

Braintrust
Your AI provider or framework (OpenAI, Anthropic, Gemini, etc.)

Set them as environment variables:

export BRAINTRUST_API_KEY="<your-braintrust-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>" # or ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.

3. Install SDKs

Install the Braintrust SDK and required libraries:

# pnpm
pnpm add braintrust openai autoevals ts-node
# npm
npm install braintrust openai autoevals ts-node

# Install the bt CLI (macOS and Linux)
curl -fsSL https://bt.dev/cli/install.sh | sh

pip install braintrust openai autoevals

# Install the bt CLI (macOS and Linux)
curl -fsSL https://bt.dev/cli/install.sh | sh

go get github.com/braintrustdata/braintrust-sdk-go
go get github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai
go get github.com/openai/openai-go

# Add to your Gemfile
gem "braintrust"
gem "openai"

# Install:
bundle install

// Add to build.gradle dependencies{} block
// (in production, pin specific versions)
implementation 'dev.braintrust:braintrust-sdk-java:+'
implementation 'com.openai:openai-java:+'

# Add to .csproj file
dotnet add package Braintrust.Sdk
dotnet add package Braintrust.Sdk.OpenAI
dotnet add package OpenAI

4. Run an eval

Build an evaluation that identifies movies from plot descriptions. You’ll define a set of test cases with movie plot descriptions as inputs and expected titles as outputs, write a task function with a prompt to identify movies, and use a scorer to measure accuracy.

Write your evaluation

Create an evaluation that defines your dataset, task, and scorer (built-in ExactMatch scorer for Python and TypeScript, equivalent code-based scorer for other languages).

movie-matcher.eval.ts

import { Eval } from "braintrust";
import { ExactMatch } from "autoevals";
import OpenAI from "openai";

const client = new OpenAI();

Eval("Evaluation quickstart", {
  experimentName: "Movie matcher (TypeScript)",
  // Data: Test cases with inputs and expected outputs
  data: [
    {
      input: "A detective investigates a series of murders based on the seven deadly sins.",
      expected: "Se7en",
    },
    {
      input: "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.",
      expected: "Inception",
    },
    {
      input: "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.",
      expected: "The Matrix",
    },
    {
      input: "A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.",
      expected: "Toy Story",
    },
    {
      input: "An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.",
      expected: "Harry Potter and the Sorcerer's Stone",
    },
  ],

  // Task: The function being evaluated
  task: async (input) => {
    const response = await client.responses.create({
      model: "gpt-5-mini",
      input: [
        {
          role: "system",
          content: "Based on the following description, identify the movie."
        },
        { role: "user", content: input }
      ],
    });
    return response.output_text;
  },

  // Scores: Metrics to measure quality
  scores: [ExactMatch],
});

import braintrust
from braintrust import Eval
from autoevals import ExactMatch
from openai import OpenAI

braintrust.auto_instrument()

client = OpenAI()

def task(input):
    response = client.responses.create(
        model="gpt-5-mini",
        input=[
            {
                "role": "system",
                "content": "Based on the following description, identify the movie."
            },
            {"role": "user", "content": input}
        ],
    )
    return response.output_text

Eval(
    "Evaluation quickstart",
    experiment_name="Movie matcher (Python)",
    # Data: Test cases with inputs and expected outputs
    data=[
        {
            "input": "A detective investigates a series of murders based on the seven deadly sins.",
            "expected": "Se7en",
        },
        {
            "input": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.",
            "expected": "Inception",
        },
        {
            "input": "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.",
            "expected": "The Matrix",
        },
        {
            "input": "A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.",
            "expected": "Toy Story",
        },
        {
            "input": "An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.",
            "expected": "Harry Potter and the Sorcerer's Stone",
        },
    ],
    # Task: The function being evaluated
    task=task,
    # Scores: Metrics to measure quality
    scores=[ExactMatch],
)

package main

import (
	"context"
	"log"
	"os"

	"github.com/braintrustdata/braintrust-sdk-go"
	"github.com/braintrustdata/braintrust-sdk-go/eval"
	traceopenai "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai"
	"github.com/openai/openai-go"
	"github.com/openai/openai-go/option"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	// Setup OpenTelemetry
	tp := trace.NewTracerProvider()
	defer tp.Shutdown(ctx)
	otel.SetTracerProvider(tp)

	// Initialize Braintrust
	bt, err := braintrust.New(tp,
		braintrust.WithAPIKey(os.Getenv("BRAINTRUST_API_KEY")),
		braintrust.WithProject("Evaluation quickstart"),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Create OpenAI client with tracing
	openaiClient := openai.NewClient(
		option.WithMiddleware(traceopenai.NewMiddleware()),
	)

	// Create an evaluator
	evaluator := braintrust.NewEvaluator[string, string](bt)

	// Run the evaluation
	_, err = evaluator.Run(ctx, eval.Opts[string, string]{
		Experiment: "Movie matcher (Go)",
		// Data: Test cases with inputs and expected outputs
		Dataset: eval.NewDataset([]eval.Case[string, string]{
			{Input: "A detective investigates a series of murders based on the seven deadly sins.", Expected: "Se7en"},
			{Input: "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", Expected: "Inception"},
			{Input: "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.", Expected: "The Matrix"},
			{Input: "A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.", Expected: "Toy Story"},
			{Input: "An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.", Expected: "Harry Potter and the Sorcerer's Stone"},
		}),
		// Task: The function being evaluated
		Task: eval.T(func(ctx context.Context, input string) (string, error) {
			response, err := openaiClient.Chat.Completions.New(ctx, openai.ChatCompletionNewParams{
				Messages: []openai.ChatCompletionMessageParamUnion{
					openai.SystemMessage("Based on the following description, identify the movie."),
					openai.UserMessage(input),
				},
				Model: openai.ChatModel("gpt-5-mini"),
			})
			if err != nil {
				return "", err
			}
			return response.Choices[0].Message.Content, nil
		}),
		// Scores: Metrics to measure quality
		Scorers: []eval.Scorer[string, string]{
			eval.NewScorer("exact_match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
				score := 0.0
				if r.Expected == r.Output {
					score = 1.0
				}
				return eval.S(score), nil
			}),
		},
	})
	if err != nil {
		log.Fatal(err)
	}
}

require 'braintrust/setup'
require 'openai'

Braintrust.init

client = OpenAI::Client.new(
    api_key: ENV.fetch('OPENAI_API_KEY', nil)
)

Braintrust::Eval.run(
  project: 'Evaluation quickstart',
  experiment: 'Movie matcher (Ruby)',
  # Data: Test cases with inputs and expected outputs
  cases: [
    {input: "A detective investigates a series of murders based on the seven deadly sins.", expected: "Se7en"},
    {input: "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", expected: "Inception"},
    {input: "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.", expected: "The Matrix"},
    {input: "A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.", expected: "Toy Story"},
    {input: "An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.", expected: "Harry Potter and the Sorcerer's Stone"},
  ],
  # Task: The function being evaluated
  task: lambda do |input:|
    response = client.chat.completions.create(
      model: 'gpt-5-mini',
      messages: [
        {role: "system", content: "Based on the following description, identify the movie."},
        {role: "user", content: input}
      ]
    )
    response.choices.first.message.content
  end,
  # Scores: Metrics to measure quality
  scorers: [
    Braintrust::Scorer.new("exact_match") { |expected:, output:| output == expected ? 1.0 : 0.0 }
  ]
)

OpenTelemetry.tracer_provider.force_flush

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.ChatModel;
import com.openai.models.chat.completions.ChatCompletionCreateParams;
import dev.braintrust.Braintrust;
import dev.braintrust.config.BraintrustConfig;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;
import dev.braintrust.instrumentation.openai.BraintrustOpenAI;
import java.util.function.Function;

class MovieMatcher {
  public static void main(String[] args) {
    var config = BraintrustConfig.builder()
        .defaultProjectName("Evaluation quickstart")
        .build();
    var braintrust = Braintrust.get(config);
    var openTelemetry = braintrust.openTelemetryCreate();

    // Wrap the OpenAI client with Braintrust instrumentation
    OpenAIClient client = BraintrustOpenAI.wrapOpenAI(openTelemetry, OpenAIOkHttpClient.fromEnv());

    // Task: The function being evaluated
    Function<String, String> taskFunction = (String input) -> {
      var response = client.chat().completions().create(
        ChatCompletionCreateParams.builder()
          .model(ChatModel.of("gpt-5-mini"))
          .addSystemMessage("Based on the following description, identify the movie.")
          .addUserMessage(input)
          .build()
      );
      return response.choices().get(0).message().content().orElse("");
    };

    // Build and run the evaluation
    var eval = braintrust.<String, String>evalBuilder()
        .name("Movie matcher (Java)") // experiment name within the "Evaluation quickstart" project
        // Data: Test cases with inputs and expected outputs
        .cases(
          DatasetCase.of("A detective investigates a series of murders based on the seven deadly sins.", "Se7en"),
          DatasetCase.of("A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "Inception"),
          DatasetCase.of("A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.", "The Matrix"),
          DatasetCase.of("A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.", "Toy Story"),
          DatasetCase.of("An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.", "Harry Potter and the Sorcerer's Stone")
        )
        .taskFunction(taskFunction)
        // Scores: Metrics to measure quality
        .scorers(
          Scorer.of("exact_match", (expected, actual) ->
            expected.equals(actual) ? 1.0 : 0.0
          )
        )
        .build();

    var result = eval.run();
    System.out.println(result.createReportString());
  }
}

using System;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Config;
using Braintrust.Sdk.Eval;
using Braintrust.Sdk.OpenAI;
using OpenAI;
using OpenAI.Chat;

class MovieMatcher
{
    static async Task Main(string[] args)
    {
        var config = BraintrustConfig.Of(
            ("BRAINTRUST_DEFAULT_PROJECT_NAME", "Evaluation quickstart")
        );
        var braintrust = Braintrust.Sdk.Braintrust.Get(config);
        var activitySource = braintrust.GetActivitySource();

        var apiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY");
        if (string.IsNullOrEmpty(apiKey))
        {
            Console.WriteLine("Error: OPENAI_API_KEY environment variable is not set.");
            return;
        }

        // Wrap the OpenAI client with Braintrust instrumentation
        var client = BraintrustOpenAI.WrapOpenAI(activitySource, apiKey);

        // Build and run the evaluation
        var eval = await braintrust
            .EvalBuilder<string, string>()
            .Name("Movie matcher (C#)") // experiment name within the "Evaluation quickstart" project
            // Data: Test cases with inputs and expected outputs
            .Cases(

                new DatasetCase<string, string>("A detective investigates a series of murders based on the seven deadly sins.", "Se7en"),
                new DatasetCase<string, string>("A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "Inception"),
                new DatasetCase<string, string>("A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.", "The Matrix"),
                new DatasetCase<string, string>("A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room.", "Toy Story"),
                new DatasetCase<string, string>("An orphaned boy discovers he's a wizard on his 11th birthday when Hagrid escorts him to magic-teaching Hogwarts School.", "Harry Potter and the Sorcerer's Stone")
            )
            // Task: The function being evaluated
            .TaskFunction((string input) =>
            {
                var chatClient = client.GetChatClient("gpt-5-mini");
                var messages = new ChatMessage[]
                {
                    new SystemChatMessage("Based on the following description, identify the movie."),
                    new UserChatMessage(input)
                };
                var result = chatClient.CompleteChat(messages);
                return result.Value.Content[0].Text;
            })
            // Scores: Metrics to measure quality
            .Scorers(
                new FunctionScorer<string, string>("exact_match", (expected, actual) =>
                    actual == expected ? 1.0 : 0.0)
            )
            .BuildAsync();

        var result = await eval.RunAsync();
        Console.WriteLine(result.CreateReportString());
    }
}

Run the evaluation

Run your evaluation:

bt eval movie-matcher.eval.ts

bt eval movie_matcher.py

go run movie_matcher.go

ruby movie_matcher.rb

gradle run

dotnet run

This creates an experiment, a permanent record of how your task performed on the dataset. Each experiment captures inputs, outputs, scores, and metadata, making it easy to compare different versions of your prompts or models.

View results

You’ll see a link to your experiment in the terminal output.Click the link to view your evaluation results, or go to Experiments in the “Evaluation quickstart” project in the Braintrust UI.

5. Iterate

You might notice that some scores are 0%. This is because the scorer requires outputs to exactly match the expected value. For example, if the AI returns “The movie is Se7en” instead of “Se7en”, or uses the UK title “Harry Potter and the Philosopher’s Stone” instead of the expected US title “Harry Potter and the Sorcerer’s Stone”, the score will be 0% for that case.Let’s improve the prompt to return only US-based movie titles and create a second experiment.

Update your evaluation

In your eval code, update the experiment name and change the prompt:Update the experiment name (for example, "Movie matcher v2 (TypeScript)") so the new run creates a separate experiment you can compare against the first.Then change the prompt to:

Identify the movie from the description. Return only the movie title, with no additional text or explanation. Always use the US-based title.

Run the evaluation

Run the improved evaluation:

bt eval movie-matcher.eval.ts

bt eval movie_matcher.py

go run movie_matcher.go

ruby movie_matcher.rb

gradle run

dotnet run

View results

Click the link to your new experiment in the terminal output.The improved prompt should have higher scores because it returns just the movie title. In the Braintrust UI, you can compare this experiment with your first one to see the improvement.

Troubleshoot

Import errors or missing modules?

Install all required packages:

pnpm add braintrust openai autoevals

pip install braintrust openai autoevals

API key errors?

Check your environment variables:

echo $BRAINTRUST_API_KEY
echo $OPENAI_API_KEY

Both should return values. If empty, set them:

export BRAINTRUST_API_KEY="your-braintrust-key"
export OPENAI_API_KEY="your-openai-key"

Get your Braintrust API key from Settings > API keys.

Not seeing experiments in UI?

Check your terminal output for the experiment link after running your evaluation. Click it to navigate directly to the experiment.If you don’t see a link:

Check for error messages in terminal output
Verify network connectivity
Ensure you’re viewing the correct project (“Evaluation quickstart”)

Need help?

Join our Discord
Email us at support@braintrust.dev
Use the Loop agent in the Braintrust UI

Build and iterate on evaluations visually in playgrounds without writing code.If you’re new to Braintrust, sign up free at braintrust.dev.

2. Add an AI provider

Braintrust lets you call AI providers directly from the UI. For this quickstart, you’ll use OpenAI:

Get an OpenAI API key from platform.openai.com/api-keys.
Go to Settings > AI providers.
Select OpenAI from the provider list.
Enter your OpenAI API key.
Click Save.

3. Run an eval

Build an evaluation that identifies movies from plot descriptions. You’ll upload test cases, create prompts, add scoring, and compare results interactively - all in a playground, a workspace for rapid iteration without writing code.

Create a playground

Go to Playgrounds.
Select Create empty playground.
Enter Movie matcher as the name and select Create.
Select the playground in the list.

Create a prompt

A prompt is the instruction you give to an AI model to complete a task.

In the playground, choose GPT-5 mini as your model.

Enter the following system prompt:

Based on the following description, identify the movie.

Select + Message and enter the following user message:
{{input}}
Prompts can use templating syntax to refer to variables. In this case, the input corresponds to the movie description given by the user.

Add a dataset

To evaluate how well your prompt works, you need test data. A dataset contains inputs and the outputs you expect from your AI.

Download this sample dataset.
In the playground, select Select a dataset > Upload new dataset.
Upload your CSV file. Columns automatically map to the input and expected fields. Drag and drop them into different categories as needed:
Click Import.

Add a scorer

Scorers measure the quality of your AI’s outputs using autoevals, custom code, or LLM-as-a-judge. For this task, you’ll use ExactMatch from autoevals because movie titles have clear right and wrong answers.

In the playground, select + Scorer.
Select AutoEvals > ExactMatch.

Run an evaluation

Select Run at the top of the playground to see how well your prompt performs.The playground will execute your prompt against all rows in your dataset and score the results. You’ll see:

The AI’s response for each movie description
The ExactMatch score (100% for correct, 0% for incorrect)

4. Iterate

You might notice that some ExactMatch scores are 0%. This is because ExactMatch requires outputs to exactly match the expected value. For example, if the AI returns “The movie is Se7en” instead of “Se7en”, or uses the UK title “Harry Potter and the Philosopher’s Stone” instead of the expected US title “Harry Potter and the Sorcerer’s Stone”, the score will be 0% for that case.Let’s create an improved prompt that returns only the US-based movie title:

In the playground, select + Task.
Select Prompt > + Blank prompt.

Add a more specific system message:

Identify the movie from the description. Return only the movie title, with no additional text or explanation. Always use the US-based title.

Select + Message and enter the following user message:
{{input}}
Select Save prompt.
Select Run to compare both prompts side-by-side.

You’ll now see results for both prompts in the playground. The improved prompt should have higher ExactMatch scores because it returns just the movie title.This comparison view helps you quickly see which prompt performs better. You can add multiple prompt variations to test different approaches.

Troubleshoot

Playground not running?

Check your AI provider: Verify you’ve added OpenAI in Settings > AI providers with a valid API key.Check dataset uploaded: Go to Datasets and confirm your CSV imported successfully. You should see all 21 movie examples.Browser issues: Try refreshing the page or using a different browser. Clear your browser cache if the playground seems stuck.

Not seeing results?

Refresh the page: The UI updates in real-time, but try refreshing if results don’t appear.Check for errors: Look for error messages in the playground. Common issues:

Invalid API key
Model not selected
Empty dataset

Need help?

Join our Discord
Email us at support@braintrust.dev
Use the Loop agent in the Braintrust UI

Next steps

Explore the full Braintrust workflow
Go deeper with evaluation:
- Write scorers - Measure what matters for your use case
- Compare experiments - Systematically test different approaches
- Build datasets - Create representative test cases from production data
- Run evaluations in CI/CD - Catch regressions automatically

​1. Sign up

​2. Get API keys

​3. Install SDKs

​4. Run an eval

​5. Iterate

​Troubleshoot

​1. Sign up

​2. Add an AI provider

​3. Run an eval

​4. Iterate

​Troubleshoot

​Next steps

1. Sign up

2. Get API keys

3. Install SDKs

4. Run an eval

5. Iterate

Troubleshoot

1. Sign up

2. Add an AI provider

3. Run an eval

4. Iterate

Troubleshoot

Next steps