Grid search over instructions and demos

GridSearchTeleprompter() tries each row of a variants table (different instructions, prompt templates or settings) on one module, scores every variant with metric, and returns a copy of the module with the best variant applied. All variants share one set of k demonstrations drawn from the training set.

Usage

GridSearchTeleprompter(
  metric = NULL,
  metric_threshold = NULL,
  max_errors = 5L,
  variants = tibble::tibble(id = 1L, instructions = NA_character_,
    template = NA_character_),
  k = 2L,
  eval_sample_size = 50L,
  verbose = TRUE
)

Arguments

metric

A metric function such as metric_exact_match(field = "sentiment"). Required when compiling.

metric_threshold, max_errors

Accepted for consistency with the other optimizers (see Teleprompter()); grid search does not use them.

variants

A data frame with one row per variant and an id column; see Details. The default is a single variant that keeps the module's instructions and template.

k

Integer number of demonstrations attached to every variant (default 2L). Use 0L for none.

eval_sample_size

Integer cap on the number of held-out scoring rows when no valset is given (default 50L).

verbose

Whether to show a progress bar (default TRUE).

Value

A GridSearchTeleprompter object to pass to compile().

Details

Each row of variants is one candidate. These columns have an effect:

• id (required) labels the variant.

• instructions replaces the signature's instructions (NA keeps them).

• instructions_suffix is appended to the original instructions.

• template replaces the prompt template.

• Runtime settings such as temperature or top_p are sent to the chat.

Other columns are stored in the module's config but do not change the prompt.

When you pass valset to compile(), variants are scored on it and the demonstrations are drawn from all of trainset. Without valset, min(eval_sample_size, ceiling(0.2 * nrow(trainset))) random rows of trainset are held out for scoring and the demonstrations come from the remaining rows. The drawn demonstrations replace any the module already had. The split and the draw use R's random number generator without a fixed seed, so call set.seed() first for reproducible results.

The search runs optimize_grid() on a copy of the module, so the scores of all variants are available from optimization_result() and top_trials().

See also

Other teleprompters: AutoResearch(), BetterTogether(), BootstrapFewShot(), BootstrapFewShotWithRandomSearch(), COPRO(), GEPA(), KNNFewShot(), LabeledFewShot(), MIPROv2(), MetaHarness(), Omni(), ReAnchor(), SIMBA(), Teleprompter(), compile()

Other grid search: module_metrics(), module_parameters(), module_trials(), optimize_grid()

Examples

variants <- data.frame(
  id = c("terse", "explicit"),
  instructions = c(
    "Answer with one word.",
    "Label the sentiment of the text as positive, negative or neutral."
  )
)
tp <- GridSearchTeleprompter(
  metric = metric_exact_match(field = "sentiment"),
  variants = variants,
  k = 1L
)
tp@variants
#>         id                                                      instructions
#> 1    terse                                             Answer with one word.
#> 2 explicit Label the sentiment of the text as positive, negative or neutral.

if (FALSE) { # \dontrun{
classifier <- module(signature("text -> sentiment"))
trainset <- data.frame(
  text = c("I love it!", "Terrible experience", "It's okay", "Works well"),
  sentiment = c("positive", "negative", "neutral", "positive")
)
set.seed(1)
optimized <- compile(
  classifier,
  tp,
  trainset,
  .llm = ellmer::chat_openai(model = "gpt-6-luna")
)
top_trials(optimized)
} # }