@gettoor/core / ExperimentListeners
Interface: ExperimentListeners
Listeners for an experiment. A dataset entry evaluation is a single evaluation of a prompt and response. It comprises of:
- Generating the response based on the prompt and dataset entry
- Running the evaluation on the response
The listeners are called in the following order for each dataset entry:
datasetEntryEvaluationStartedgeneratingResponseresponseGeneratedrunningEvaluationevaluationCompleteddatasetEntryEvaluationCompleted
Properties
datasetEntryEvaluationCompleted?
optionaldatasetEntryEvaluationCompleted?: (input:ExperimentDatasetEntryEvaluationCompletedInput) =>Promise<void>
Called when a dataset entry evaluation is completed.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentDatasetEntryEvaluationCompletedInput | Input for the listener |
Returns
Promise<void>
datasetEntryEvaluationStarted?
optionaldatasetEntryEvaluationStarted?: (input:ExperimentListenerCommonInput) =>Promise<void>
Called when a dataset entry evaluation is started.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentListenerCommonInput | Input for the listener |
Returns
Promise<void>
evaluationCompleted?
optionalevaluationCompleted?: (input:ExperimentEvaluationCompletedInput) =>Promise<void>
Called when an evaluation is completed.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentEvaluationCompletedInput | Input for the listener |
Returns
Promise<void>
generatingResponse?
optionalgeneratingResponse?: (input:ExperimentListenerCommonInput) =>Promise<void>
Called when a response is being generated.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentListenerCommonInput | Input for the listener |
Returns
Promise<void>
responseGenerated?
optionalresponseGenerated?: (input:ExperimentResponseGeneratedInput) =>Promise<void>
Called when a response has been generated.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentResponseGeneratedInput | Input for the listener |
Returns
Promise<void>
runningEvaluation?
optionalrunningEvaluation?: (input:ExperimentListenerCommonInput) =>Promise<void>
Called when an evaluation is started.
Parameters
| Parameter | Type | Description |
|---|---|---|
input | ExperimentListenerCommonInput | Input for the listener |
Returns
Promise<void>