Metrics built on vitals scorers

These functions wrap vitals scorers with as_dsprrr_metric() so they can be used directly as dsprrr metrics:

• metric_model_graded_qa() and metric_model_graded_fact() ask a model to grade the answer against the target.

• metric_detect_match(), metric_detect_includes() and metric_detect_pattern() compare strings without a model.

Usage

metric_model_graded_qa(
  template = NULL,
  instructions = NULL,
  grade_pattern = "(?i)GRADE\\s*:\\s*([CPI])(.*)$",
  partial_credit = FALSE,
  scorer_chat = NULL,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

metric_model_graded_fact(
  template = NULL,
  instructions = NULL,
  grade_pattern = "(?i)GRADE\\s*:\\s*([CPI])(.*)$",
  partial_credit = FALSE,
  scorer_chat = NULL,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

metric_detect_match(
  location = c("end", "begin", "any", "exact"),
  case_sensitive = FALSE,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

metric_detect_includes(
  case_sensitive = FALSE,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

metric_detect_pattern(
  pattern,
  case_sensitive = FALSE,
  all = FALSE,
  input_column = "input",
  target_column = "target",
  result_column = "result"
)

Arguments

template

Grading prompt template, a glue string with input, answer, criterion and instructions fields. NULL uses the vitals default.

instructions

Grading instructions. NULL uses the vitals default.

grade_pattern

Regular expression that extracts the grade from the grader's reply.

partial_credit

Whether the grader may award partial credit (0.5).

scorer_chat

The ellmer Chat that grades, such as ellmer::chat_openai(model = "gpt-6-luna"). Required; see Details.

input_column, target_column, result_column

Column names passed to as_dsprrr_metric(). Keep the defaults for vitals scorers.

location

Where to look for the target in the result: "end" (the default), "begin", "any" or "exact".

case_sensitive

Whether matching is case-sensitive (default FALSE).

pattern

Regular expression with capture groups, such as "([0-9]+)". The captured text is compared with the target.

all

Whether every captured group must match the target (TRUE) or at least one (FALSE, the default).

Value

A metric function function(prediction, expected_row).

Details

The underlying vitals scorers read the expected answer from a target column, and the graded metrics also show the grader the question from an input column. Add those columns to your data, for example transform(data, input = question, target = answer), and keep the default column arguments; see as_dsprrr_metric().

The graded metrics need scorer_chat: outside a vitals Task, there is no solver chat for vitals to fall back on.

See also

Other metrics: as_dsprrr_metric(), evaluate(), metric_contains(), metric_custom(), metric_exact_match(), metric_f1(), metric_field_match(), metric_threshold(), metric_with_feedback(), metric_with_trace()

Other integrations: as_dsprrr_metric(), as_dsprrr_traces(), as_ellmer_tool(), as_vitals_cost(), as_vitals_samples(), as_vitals_solver(), as_vitals_task(), create_search_tool(), llm_predict(), ragnar_tool(), reasoning_effort(), register_dsprrr_engine(), summarize_traces_df(), temperature(), top_p(), use_dsprrr_template(), validate_workflow()

Examples

# String comparisons need no model
ends_with_answer <- metric_detect_match(location = "end")
ends_with_answer("The answer is Paris", data.frame(target = "Paris"))
#> [1] 1

mentions <- metric_detect_includes()
mentions("Paris is the capital", data.frame(target = "Paris"))
#> [1] 1

number <- metric_detect_pattern("([0-9]+)")
number("The total is 42", data.frame(target = "42"))
#> [1] 1

if (FALSE) { # \dontrun{
# Model-graded metrics call scorer_chat once per example
graded <- metric_model_graded_qa(
  scorer_chat = ellmer::chat_openai(model = "gpt-6-luna")
)
graded(
  "Paris",
  data.frame(input = "What is the capital of France?", target = "Paris")
)

fact <- metric_model_graded_fact(
  scorer_chat = ellmer::chat_anthropic(model = "claude-sonnet-4-5"),
  partial_credit = TRUE
)
} # }