Getting started

Toor is a set of packages of utilities for LLM evaluations and experimentation. It's divided into the following packages:

• @gettoor/core - The core package with all types and functions for evaluations and experimentation.
• @gettoor/cli - Command-line interface to run evaluations and experiments based on a configuration file.

Toor supports the following evaluations and experiments:

• LLM-as-a-judge - An evaluation in which an output from a model is evaluated against predefined criteria by assigning scores (e.g. overall score, correctness, relevance or completeness).
• LLM experiments - A process to measure performance of different models, model parameters, prompts on a dataset, typically to find the best combination.

Installation

Install the core package if you want to run evaluations and experiments in your code:

npm install @gettoor/core

Install the command line interface if you want to run evaluations and experiments from the command line:

npm install @gettoor/cli

Core

Use @gettoor/core when you want full control in code.

Run your first LLM-as-a-judge evaluation

import { binary } from '@gettoor/core';

const result = await binary({
  modelName: 'openai:gpt-4o',
  prompt: 'What is the capital of France?',
  response: 'Paris',
});

console.log(result.result, result.reasoning);

Run an experiment programmatically

import { 
  runExperiment,
  SCALAR_SCORING_1_5,
  scalarEvaluator,
} from '@gettoor/core';

const results = await runExperiment({
  settings: {
    modelName: 'openai:gpt-4o',
    evaluator: scalarEvaluator({
      scoringScale: SCALAR_SCORING_1_5,
    }),
  }
  models: [
    // models to evaluate
    { name: 'openai:gpt-4o-mini' },
  ],
  modelParameters: [
    // model parameters to evaluate
    { name: 'default', temperature: 0.2 },
  ],
  prompts: [
    {
      // prompts to evaluate
      name: 'support',
      prompt: 'Answer clearly: <<question>>' },
    ],

    // dataset with variables to replace in the prompts
    dataset: [
      { 
        name: 'reset-password',
        vars: {
          question: 'How can I reset my password?',
        },
      },
    ],
});

// score of the first combination of model, parameters, prompt and dataset
console.log('Score: ' + results[0].score.scoreAsString);

For full API details, see Core, LLM-as-a-judge, and LLM experiments.

CLI

Use @gettoor/cli when you want to run experiments from configuration files.

Create an experiment config file toor.experiment.yaml

evaluation:
  # binary, 1-3, 1-5, 1-10
  type: 1-5

  # model used to evaluate the prompts and responses
  model: openai:gpt-4o

# models to evaluate
models:
  - model: openai:gpt-4o-mini

# model parameters to evaluate
model-parameters:
  - name: default
    temperature: 0.2

# prompts to evaluate
prompts:
  - name: support
    prompt: "Answer clearly: <<question>>"

# dataset with variables to replace in the prompts
dataset:
  - name: reset-password
    vars:
      question: How can I reset my password?

Run the CLI

npx toor ./toor.experiment.yaml

For more details, see CLI.