Skip to contents

Use a vitals scorer as a dsprrr metric

Source: R/vitals.R
as_dsprrr_metric.Rd

as_dsprrr_metric() turns a vitals scorer into a per-example metric that evaluate(), optimize_grid() and the optimizers can call. Each call builds a one-row vitals sample from the prediction and the data row, runs the scorer, and converts its grade to a number.

Usage

as_dsprrr_metric(
  vitals_scorer,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

Arguments

vitals_scorer

A scorer function that takes a samples tibble and returns a list (or data frame) with a score element, such as vitals::detect_includes().

input_column

Name of the input column, in both the data and the sample (default "input").

target_column

Name of the expected-answer column, in both the data and the sample (default "target").

result_column

Name of the sample column that receives the prediction (default "result").

Value

A metric function function(prediction, expected_row) that returns a number, usually in [0, 1], or NA.

Details

The sample has three columns. The input_column and target_column columns are copied from the data column of the same name (NA when the row has no such column), and result_column holds the prediction. vitals' built-in scorers read the columns input, target and result, so keep the defaults with them and give your data a target column (and an input column for model-graded scorers). The other names are for scorers of your own that read different columns.

Grades are converted as follows: numbers are kept, TRUE/FALSE become 1/0, and "C"/"correct"/"pass", "I"/"incorrect"/"fail" and "P"/"partial" become 1, 0 and 0.5. Anything else gives NA with a warning.

See also

Other integrations: as_dsprrr_traces(), as_ellmer_tool(), as_vitals_cost(), as_vitals_samples(), as_vitals_solver(), as_vitals_task(), create_search_tool(), llm_predict(), ragnar_tool(), reasoning_effort(), register_dsprrr_engine(), summarize_traces_df(), temperature(), top_p(), use_dsprrr_template(), validate_workflow(), vitals_metrics

Other metrics: evaluate(), metric_contains(), metric_custom(), metric_exact_match(), metric_f1(), metric_field_match(), metric_threshold(), metric_with_feedback(), metric_with_trace(), vitals_metrics

Examples

# A scorer written in the vitals style
exact_match <- function(samples) {
  list(score = as.numeric(samples$result[[1]] == samples$target[[1]]))
}
metric <- as_dsprrr_metric(exact_match)
metric("yes", data.frame(input = "Continue?", target = "yes"))
#> [1] 1

# A vitals scorer; the data needs a `target` column
includes <- as_dsprrr_metric(vitals::detect_includes())
includes("The capital is Paris.", data.frame(target = "Paris"))
#> [1] 1