LLM experiments

An LLM experiment is a structured process for comparing different models, prompts and parameters (such as temperature or top-p) on the same dataset to understand which configuration performs best for a specific task.

Toor provides the possibility to run an LLM experiment over the following parameters:

• models,
• model parameters,
• prompts,
• datasets (a dataset is a collection of entries with variables to replace in the prompts).

Each combination of model, model parameters, prompt, and dataset is evaluated.

NOTE

The total number of evaluations is: models × model parameters × prompts × dataset entries.

Adding one item in a single dimension increases the total by the product of the other dimensions.
For example, adding one model increases evaluations by model parameters × prompts × dataset entries.

This in turn increases the time required to run the experiment and the cost of the evaluation.

Example

const evalPrompt = `
You are a strict evaluator.

Your task is to assess how well the RESPONSE satisfies the PROMPT using the provided SCORING_SCALE.

PROMPT:
<<prompt>>

RESPONSE:
<<response>>

SCORING_SCALE:
<<scoring_scale>>

Evaluation Guidelines:

Assess the response in 2 strict steps.

# Step 1 — Response Format Validation

The RESPONSE must be exactly one of the following values:

- positive
- neutral
- negative

If the RESPONSE is anything else (including different capitalization, additional text, punctuation, whitespace, explanations, or multiple values), immediately return score 1 and stop the evaluation. Do not proceed to Step 2.

# Step 2 — Prompt Satisfaction

Only if the RESPONSE passed Step 1, evaluate how well the RESPONSE satisfies the PROMPT according to the SCORING_SCALE.

Return only the final score.
`;

const results = await runExperiment({
  settings: {
    evaluator: scalarEvaluator({
      scoringScale: SCALAR_SCORING_1_5,
      modelName: 'openai:gpt-4o',
      evalPrompt,
    }),
  },

  // models to evaluate
  models: [
    {
      name: 'openai:gpt-4o-mini'
    },
    {
      name: 'gemini:gemini-2.5-flash'
    },
  ],
  modelProvider: new DefaultModelProvider(),

  // model parameters to evaluate
  modelParameters: [
    {
      name: 'default',
      temperature: 0.2,
    },
  ],

  // prompts to evaluate
  prompts: [
    {
      name: 'sentiment',
      prompt: `
Classify the sentiment of the review.

Review:
<<review>>

Return one of:
- positive
- neutral
- negative
`,
    },
    {
      name: 'review',
      prompt: `
Read the review and tell me what you think.

Review:
<<review>>
`,
    }
  ],

  // dataset with variables to replace in the prompts
  dataset: [
    {
      name: 'positive-review',
      vars: { review: 'The product exceeded my expectations.' }
    },
    {
      name: 'neutral-review',
      vars: { review: `It's okay, nothing special.` }
    },
    {
      name: 'negative-review',
      vars: { review: `Completely unusable after one day.` }
    },
  ],
});

Evaluators

Each combination of model, model parameters, prompt and dataset is evaluated using an evaluator. It takes a prompt and a response generated by the model and returns a score with reasoning.

Toor provides the following predefined evaluators based on LLM-as-a-judge:

• binaryEvaluator
• scalarEvaluator

Both evaluators take a model and model parameters used to evaluate a prompt-response pair and assign a score. They also take an optional evalPrompt, which is a custom evaluation prompt.

modelProvider is an optional parameter that allows using a custom model provider. If not provided, the default model provider is used. See Model provider.

import { runExperiment, binaryEvaluator } from '@gettoor/core';

const results = await runExperiment({
  settings: {
    evaluator: binaryEvaluator({
      modelName: 'openai:gpt-4o',
      modelParameters: {
        temperature: 0.2,
      },
      modelProvider: new DefaultModelProvider(),
      evalPrompt: '...',
    }),
  },
  // other fields of the experiment...
});

The binary evaluator returns a numeric score:

• 0 (failed),
• 1 (passed).

The binary evaluator uses the binary LLM-as-a-judge to evaluate the response. This means that evalPrompt must contain the placeholders:

• <<prompt>> replaced with the prompt to evaluate,
• <<response>> replaced with the response to the prompt.

See Binary evalPrompt for more details.

import {
  SCALAR_SCORING_1_5,
  scalarEvaluator,
  runExperiment,
} from '@gettoor/core';

const results = await runExperiment({
  settings: {
    evaluator: scalarEvaluator({
      scoringScale: SCALAR_SCORING_1_5,
      modelName: 'openai:gpt-4o',
      modelParameters: {
        temperature: 0.2,
      },
      modelProvider: new DefaultModelProvider(),
      evalPrompt: '...',
    }),
  },
  // other fields of the experiment...
});

The scalar evaluator returns a numeric score according to the provided scoring scale (scoringScale). See Scales for predefined scoring scales.

The scalar evaluator uses the scalar LLM-as-a-judge to evaluate the response. This means that evalPrompt must contain the placeholders:

• <<prompt>> replaced with the prompt to evaluate,
• <<response>> replaced with the response to the prompt,
• <<scoring_scale>> replaced with the scoring scale to use for the evaluation.

See Scalar evalPrompt for more details.

Custom evaluator

A custom evaluator can be created by implementing the ExperimentEvaluator type. It takes an ExperimentEvaluatorInput and returns an ExperimentEvaluatorOutput.

type ExperimentEvaluator = (
  input: ExperimentEvaluatorInput
) => Promise<ExperimentEvaluatorOutput>;

ExperimentEvaluatorInput contains the fields:

• promptName - the name of the prompt to evaluate coming from the experiment prompts array,
• prompt - the prompt to evaluate coming from the experiment prompts array,
• response - the response generated by the model based on the prompt and model parameters.

An evaluator returns a ExperimentEvaluatorOutput with the following fields:

• score - the score assigned to the response (see ExperimentScore and toExperimentScore),
• usage (optional)- the LLM usage for the evaluation (see LLMUsage).

Example

The example below shows a custom evaluator which tests if the response is a valid priority for a support ticket. It tests if the response from a model is one of the valid priorities: high, medium, low.

import { toExperimentScore, ExperimentEvaluator } from '@gettoor/core';

const evaluator: ExperimentEvaluator = async (input) => {
  const VALID_RESPONSES = ['high', 'medium', 'low'];
  const isValid = VALID_RESPONSES.includes(input.response);
  return {
    score: toExperimentScore({
      score: isValid ? 1 : 0,
      minScore: 0,
      maxScore: 1,
      reasoning: isValid ? 'Valid response' : 'Invalid response',
    }),
  };
};

Models

The models to evaluate are passed under the models field in the experiment object. The only supported field is name (keep it unique). The name is used to resolve the model using the model provider from modelProvider. If not set, the default model provider is used. See Model provider.

import { runExperiment } from '@gettoor/core';

const results = await runExperiment({
  models: [
    {
      name: 'openai:gpt-4o-mini'
    },
    {
      name: 'gemini:gemini-2.5-flash'
    },
  ],
  modelProvider: new DefaultModelProvider(),
  // other fields of the experiment...
});

Parameters

The model parameters to evaluate are passed under the modelParameters field in the experiment object. Each parameter object must have a unique name. The supported model parameters are:

• maxOutputTokens
• temperature
• topP
• topK
• presencePenalty
• frequencyPenalty.

import { runExperiment } from '@gettoor/core';

const results = await runExperiment({
  modelParameters: [
    {
      name: 'low-temperature',
      temperature: 0.2,
    },
    {
      name: 'low-temperature-with-max-token',
      temperature: 0.2,
      maxOutputTokens: 1000,
    },
    {
      name: 'top-p',
      topP: 0.8,
    },
  ],
  // other fields of the experiment...
});

Prompts

The prompts to evaluate are passed under the field prompts. Each prompt object must have:

• name - unique name,
• prompt - prompt to evaluate.

Each prompt can contain placeholders of the form <<placeholder>>. The placeholders are replaced with the variables from the dataset entries.

For N prompts and M dataset entries, a total of N × M prompts are generated and evaluated. For example, if there are 2 prompts and 3 dataset entries, then 6 prompts to evaluate are generated.

import { runExperiment } from '@gettoor/core';

const sentimentPrompt = `
Classify the sentiment of the review.

Review:
<<review>>

Return one of:
- positive
- neutral
- negative
`;

const reviewPrompt = `
Read the review and tell me what you think.

Review:
<<review>>
`;

const results = await runExperiment({
  prompts: [
    {
      name: 'sentiment',
      prompt: sentimentPrompt,
    },
    {
      name: 'review',
      prompt: reviewPrompt,
    }
  ],
  // other fields of the experiment...
});

Structured output

The output generated by the model can be a structured output. It's passed in the field structuredOutput in the experiment object. It is an object with the following fields:

• schema - JSON schema of the structured output,
• format - format in which the structured output is passed to the evaluation. One of: json or yaml.

import { runExperiment } from '@gettoor/core';

const results = await runExperiment({
  structuredOutput: {
    format: 'yaml',
    schema: {
      type: 'object',
      properties: {
        priority: {
          type: 'string',
          enum: ['low', 'medium', 'high'],
        },
        category: {
          type: 'string',
          enum: ['bug', 'billing', 'feature', 'question'],
        },
      },
      required: ['priority', 'category'],
    },
  },
  // other fields of the experiment...
});

Dataset

A dataset is a set of entries with variables that are substituted in the prompts. A dataset is passed under the dataset field. It is an array of objects, where each object has the fields:

• name - unique name,
• vars - record of variables where the key is a string and the value can be anything.

The values in vars can be of any type (primitives, objects, and arrays). Placeholders let you specify the format in which values are inserted into the prompt. A placeholder with a format has the form <<placeholder:format>>. The supported formats are:

• string (default) - the value is converted to a string using toString(). An error is thrown if the value is not a primitive.
• json - the value is converted to a single-line JSON string.
• json-pretty - the value is converted to a multi-line JSON string.
• yaml, yml - the value is converted to a YAML string.

Consider the example below, where the placeholder <<input:yaml>> is replaced with the input variable converted to a YAML string for each dataset entry.

import { binaryEvaluator, runExperiment } from '@gettoor/core';

const results = await runExperiment({
  settings: {
    evaluator: binaryEvaluator({
      modelName: 'openai:gpt-4o',
    }),
  },
  models: [
    // models to evaluate
    { name: 'openai:gpt-4o-mini' },
  ],
  modelParameters: [
    // model parameters to evaluate
    { name: 'default', temperature: 0.2 },
  ],
  prompts: [
    {
      // prompts to evaluate
      name: 'support',
      prompt: `
Determine whether the candidate is a good match for the job.

<<input:yaml>>

Respond with exactly one of:
match
no_match
`
    },
  ],

  // dataset with variables to replace in the prompts
  dataset: [
    { 
      name: 'match',
      vars: {
        input: {
          job_description: 'Senior TypeScript engineer with NestJS experience.',
          candidate_profile: '10 years of TypeScript experience, 5 years with NestJS.',
        },
      },
    },
    { 
      name: 'no-match',
      vars: {
        input: {
          job_description: 'Senior TypeScript engineer with NestJS experience.',
          candidate_profile: 'Graphic designer with 8 years of Adobe Photoshop experience.',
        },
      },
    },
  ],
});

Callbacks

An LLM experiment can be a long-running process if the number of combinations is high. Listener functions can be used to track the progress of the experiment. They are set in the listeners field in the experiment object and are of type ExperimentListeners.

The following listeners are available:

• datasetEntryEvaluationStarted
• datasetEntryEvaluationCompleted
• generatingResponse
• responseGenerated
• runningEvaluation
• evaluationCompleted