Skip to contents

Labeled few-shot: add training rows as demonstrations

Source: R/teleprompter.R
LabeledFewShot.Rd

LabeledFewShot() compiles a Predict module by attaching k rows of the training set as few-shot demonstrations. It makes no model calls and scores nothing, which makes it a fast, offline baseline for the other optimizers.

Usage

LabeledFewShot(
  metric = NULL,
  metric_threshold = NULL,
  max_errors = 5L,
  k = 4L,
  sample = TRUE,
  seed = 123L
)

Arguments

metric

Optional. It is not used for scoring, but when it has a field (as in metric_exact_match(field = "sentiment")), that column supplies the demonstrations' outputs.

metric_threshold, max_errors

Accepted for consistency with the other optimizers (see Teleprompter()); LabeledFewShot() does not use them.

k

Integer number of demonstrations (default 4L). When the training set has fewer rows, all of them are used.

sample

If TRUE (the default), draw k rows at random. If FALSE, take the first k rows.

seed

Integer seed for the random draw (default 123L). It is passed to set.seed(), which also resets the session's random number stream.

Value

A LabeledFewShot object to pass to compile().

Details

Rows are drawn at random (or the first k rows are taken when sample = FALSE); nothing checks whether they are good examples. A demonstration's inputs come from the columns named after the signature inputs. Its output comes from the metric's field when the metric has one, otherwise from the first column named output, label, answer, response, result or y, otherwise from the first non-input column.

The program must be a Predict module, such as one from module() or chain_of_thought(). Other programs, including pipelines and RLM modules, are rejected because training rows do not match the signatures of their inner predictors. BootstrapFewShot() compiles pipelines.

See also

Other teleprompters: AutoResearch(), BetterTogether(), BootstrapFewShot(), BootstrapFewShotWithRandomSearch(), COPRO(), GEPA(), GridSearchTeleprompter(), KNNFewShot(), MIPROv2(), MetaHarness(), Omni(), ReAnchor(), SIMBA(), Teleprompter(), compile()

Examples

classifier <- module(signature("text -> sentiment"))
trainset <- data.frame(
  text = c("I love it!", "Terrible experience", "It's okay", "Works well"),
  sentiment = c("positive", "negative", "neutral", "positive")
)

tp <- LabeledFewShot(k = 2L)
tp
#> <dsprrr::LabeledFewShot>
#>  @ metric          : NULL
#>  @ metric_threshold: NULL
#>  @ max_errors      : int 5
#>  @ k               : int 2
#>  @ sample          : logi TRUE
#>  @ seed            : int 123

# Compiling makes no model calls and leaves `classifier` unchanged
compiled <- compile(classifier, tp, trainset)
best_demos(compiled, as_tibble = TRUE)
#> # A tibble: 2 × 2
#>   text       output  
#>   <chr>      <chr>   
#> 1 It's okay  neutral 
#> 2 Works well positive