@gettoor/core v0.1.0
@gettoor/core v0.1.0
Errors
| Class | Description |
|---|---|
| BadRequestToorError | Thrown when a request is invalid. |
| MissingApiKeyError | Thrown when a required API key is missing. |
| NotFoundToorError | Thrown when a resource is not found. |
| ToorError | A base class for errors. |
Experimentation
| Name | Description |
|---|---|
| DatasetEntryEvaluationMetrics | Dataset entry evaluation metrics. |
| EvaluatorBinaryInput | Input for the binary evaluator. |
| EvaluatorScalarInput | Input for the scalar evaluator. |
| Experiment | Experiment settings and configuration. |
| ExperimentDatasetEntry | Experiment dataset entry. |
| ExperimentDatasetEntryEvaluationCompletedInput | Output for the datasetEntryEvaluationCompleted listener. |
| ExperimentEvaluationCompletedInput | Input for the evaluationCompleted listener. |
| ExperimentEvaluatorInput | Input for an experiment evaluator. |
| ExperimentEvaluatorOutput | Output for an experiment evaluator. |
| ExperimentListenerCommonInput | Input for an experiment listener. |
| ExperimentListeners | Listeners for an experiment. A dataset entry evaluation is a single evaluation of a prompt and response. It comprises of: - Generating the response based on the prompt and dataset entry - Running the evaluation on the response |
| ExperimentModel | Experiment model. |
| ExperimentModelParameters | Experiment model parameters. |
| ExperimentPrompt | Experiment prompt. |
| ExperimentResponseGeneratedInput | Input for the responseGenerated listener. |
| ExperimentResult | Experiment result of a single evaluation. |
| ExperimentScore | Score for a single evaluation. |
| ExperimentSettings | Experiment settings. |
| ExperimentStructuredOutput | Experiment structured output. |
| ToExperimentScoreInput | Input for toExperimentScore. |
| ExperimentDatasetEntryEvaluationStartedInput | Input for the datasetEntryEvaluationStarted listener. |
| ExperimentDatasetVarValue | Experiment dataset variable value. |
| ExperimentEvaluator | Experiment evaluator. |
| ExperimentGeneratingResponseInput | Input for the generatingResponse listener. |
| ExperimentRunningEvaluationInput | Input for the runningEvaluation listener. |
| ExperimentStructuredOutputFormat | Experiment structured output format. |
| binaryEvaluator | Creates a binary experiment evaluator. |
| runExperiment | Runs an experiment. |
| scalarEvaluator | Creates a scalar experiment evaluator. |
| toExperimentScore | Converts an input to an experiment score. |
| toModelParameters | Converts experiment model parameters to model parameters. |
LLM
| Interface | Description |
|---|---|
| LLMUsage | Usage information for a language model. |
| MetricResult | Result for a single metric. |
| ModelParameters | Model parameters for the LLM-as-a-judge. |
LLM-as-a-judge
| Name | Description |
|---|---|
| BinaryInput | Input for the binary evaluation. |
| BinaryOutput | Output for the binary evaluation. |
| Output | Output for the LLM-as-a-judge. |
| ScalarInput | Input for the scalar evaluation. |
| ScalarMetric | A metric for the scalar evaluation. |
| ScalarOutput | Output for the scalar evaluation. |
| ScalarResult | Result for the scalar evaluation. |
| ScalarScoringScale | A scoring scale for the scalar evaluation. |
| BINARY_PROMPT | The default prompt to use for the binary evaluation. It takes the placeholders: - <<prompt>>: The prompt to evaluate. - <<response>>: The response to evaluate. |
| SCALAR_METRIC_ACTIONABILITY | Can the user act on the answer? |
| SCALAR_METRIC_CLARITY | Is it easy to understand? |
| SCALAR_METRIC_COHERENCE | Does it flow logically and remain internally consistent? |
| SCALAR_METRIC_COMPLETENESS | Does it cover everything important? |
| SCALAR_METRIC_CONCISENESS | Is it appropriately brief without unnecessary content? |
| SCALAR_METRIC_CORRECTNESS | Is the answer factually accurate? |
| SCALAR_METRIC_GRAMMAR | Is it free of spelling, grammar, and punctuation errors? |
| SCALAR_METRIC_HELPFULNESS | Does it actually help the user accomplish their goal? |
| SCALAR_METRIC_INSTRUCTION_FOLLOWING | Did it obey all instructions and constraints? |
| SCALAR_METRIC_REASONING_QUALITY | Is the reasoning clear and logical? |
| SCALAR_METRIC_RELEVANCE | Does it address the user's request? |
| SCALAR_METRIC_SAFETY | Does it avoid harmful or policy-violating content? |
| SCALAR_PROMPT | The default prompt to use for the scalar evaluation. It takes the parameters (each in double curly braces): - <<prompt>>: The prompt to evaluate. - <<response>>: The response to evaluate. - <<scoring_scale>>: The scoring scale to use. |
| SCALAR_SCORING_1_10 | Fine-grained scale (1-10) for the scalar evaluation. |
| SCALAR_SCORING_1_3 | Coarse-grained scale (1-3) for the scalar evaluation. |
| SCALAR_SCORING_1_5 | Likert scale (1-5) for the scalar evaluation. |
| SCALAR_SCORING_DEFAULT | The default scoring scale for the scalar evaluation. |
| binary | Evaluates a response against a prompt using a language model with score either 0 (failed) or 1 (passed). |
| scalar | Scores a response against a prompt using a language model. |
Math
| Function | Description |
|---|---|
| average | Calculates the average of an array of numbers. |
| median | Calculates the median of an array of numbers. |
Model Provider
| Name | Description |
|---|---|
| DefaultModelProvider | A default model provider implementation. Supports Gemini, OpenAI and Anthropic. |
| ModelNotFoundError | Error thrown when a model is not found. |
| ModelProvider | A model provider. The returned LanguageModel comes from AI SDK. |
Other
| Function | Description |
|---|---|
| buildModelCallSettings | - |
Utils
| Name | Description |
|---|---|
| InvalidFormatForObjectError | Error thrown when an invalid format is provided for an object placeholder. |
| InvalidPlaceholderFormatError | Error thrown when an invalid format is provided for a placeholder. |
| MissingPlaceholderError | Error thrown when a placeholder is missing. |
| NoValueForPlaceholderError | Error thrown when no value is provided for a placeholder. |
| UnknownPlaceholdersError | Error thrown when unknown placeholders are present in a string. |
| ReplacePlaceholderOutput | Output of the replacePlaceholders function. |
| findPlaceholders | Finds placeholders in a string. |
| rejectUnknownPlaceholders | Rejects unknown placeholders in a string. |
| replacePlaceholders | Replaces placeholders in a string with values. The placeholders are of the form <key> in the string. |
| requirePlaceholders | Validates that the required placeholders are present in the text. |