as_vitals_task() creates a vitals Task that uses the module as its
solver (see as_vitals_solver()), so you can evaluate the module with
vitals' scoring, logging and viewer. Call the task's $eval() method to
run it and $view() to browse the results.
Usage
as_vitals_task(
module,
dataset,
scorer = NULL,
.llm = NULL,
name = NULL,
epochs = 1L,
metrics = NULL,
dir = NULL,
.concurrency = NULL,
...
)Arguments
- module
A module, such as one created with
module().- dataset
A data frame with the signature's input columns and a
targetcolumn.- scorer
A vitals scorer, such as
vitals::detect_includes(). The default,vitals::model_graded_qa(), asks the solver's chat to grade.- .llm
An ellmer Chat for the solver.
NULL(the default) uses the module's own chat or the default chat, resolved when the task is created.- name
Task name. Defaults to the expression passed as
dataset.- epochs
Integer number of times each sample is run (default
1L).- metrics
Optional named list of functions that summarize a vector of scores into one number.
- dir
Directory for the evaluation logs. Defaults to
vitals::vitals_log_dir().- .concurrency
Optional policy from
concurrency_control().- ...
Further arguments passed to
as_vitals_solver().
Details
dataset needs one column per signature input and a target column. The
input columns are nested into the input list-column that vitals expects;
other columns are kept.
See also
Other integrations:
as_dsprrr_metric(),
as_dsprrr_traces(),
as_ellmer_tool(),
as_vitals_cost(),
as_vitals_samples(),
as_vitals_solver(),
create_search_tool(),
llm_predict(),
ragnar_tool(),
reasoning_effort(),
register_dsprrr_engine(),
summarize_traces_df(),
temperature(),
top_p(),
use_dsprrr_template(),
validate_workflow(),
vitals_metrics
Examples
qa <- module(signature("question -> answer"))
test_data <- data.frame(
question = c("What is 2+2?", "Capital of France?"),
target = c("4", "Paris")
)
tsk <- as_vitals_task(
qa,
test_data,
scorer = vitals::detect_includes(),
.llm = ellmer::chat_openai(model = "gpt-6-luna"),
dir = tempdir()
)
tsk
#> An evaluation task test-data.
if (FALSE) { # \dontrun{
tsk$eval()
tsk$get_cost()
} # }