Flex: Optimize the Whole Program • dsprrr

Most dsprrr optimizers improve the instructions or examples inside a program whose shape you chose. flex() makes the implementation itself optimizable. GEPA can change how many predictors run, add a deterministic branch, or call a tool you supplied.

That extra freedom is useful when you can score the outcome but do not yet know the best decomposition. It is unnecessary when the workflow is already clear.

What should change?	Use
Instructions or examples inside a known module	A regular module and teleprompter
A workflow whose steps your team should own	An explicit pipeline
The number or order of Predict and Chain-of-Thought steps	JSON Flex
The choice among R logic, predictors, and selected tools	Executable Flex

A router that learns when not to call a model

Suppose a support team routes tickets to four queues. Some tickets contain an incident code with a known owner. Others need language-model judgment.

A conventional predictor sends every ticket to the model. The Flex objective is more specific: keep routing accuracy, but avoid a predictor call when the incident catalog already contains the answer.

Define the task and its evidence

Use separate training and validation rows during search, then reserve a holdout set for the final comparison:

library(dsprrr)

route_signature <- signature(
  "ticket -> queue: enum('database', 'security', 'payments', 'identity')",
  instructions = "Route each support ticket to its owning queue."
)

train <- data.frame(
  ticket = c(
    "[INC-DB-17] checkout failures in us-east",
    "Password reset links expire immediately",
    "Duplicate charge after a plan upgrade",
    "Queries time out after a schema migration"
  ),
  queue = c(
    "database", "identity", "payments", "database"
  )
)

validation <- data.frame(
  ticket = c(
    "[INC-SEC-9] unusual-login burst",
    "Card charged twice for one invoice"
  ),
  queue = c("security", "payments")
)

holdout <- data.frame(
  ticket = c(
    "[INC-PAY-4] capture queue is growing",
    "[INC-DB-17] replicas unavailable",
    "[INC-SEC-9] login spray detected",
    "Reset email arrives after its token expires",
    "Renewal created two invoices",
    "Read traffic fails after failover"
  ),
  queue = c(
    "payments", "database", "security", "identity", "payments", "database"
  )
)

The catalog is an ordinary R closure wrapped in an ellmer tool definition. Its description tells GEPA what the tool returns; supplying it to Flex gives generated code one named capability rather than the rest of the host session.

lookup_incident_fn <- local({
  catalog <- c(
    "INC-DB-17" = "database",
    "INC-SEC-9" = "security",
    "INC-PAY-4" = "payments"
  )

  function(ticket) {
    hit <- names(catalog)[vapply(
      names(catalog),
      grepl,
      logical(1),
      x = ticket,
      fixed = TRUE
    )]
    if (!length(hit)) {
      return(list(found = FALSE))
    }
    list(found = TRUE, queue = unname(catalog[[hit[[1L]]]]))
  }
})

lookup_incident <- ellmer::tool(
  lookup_incident_fn,
  name = "lookup_incident",
  description = paste(
    "Find the owner of an incident code in a support ticket.",
    "Returns list(found = FALSE) when no known code appears; otherwise",
    "returns list(found = TRUE, queue = <owning queue>)."
  ),
  arguments = list(
    ticket = ellmer::type_string(
      description = "Complete support-ticket text."
    )
  )
)

Start with one predictor

Executable Flex source defines a top-level forward() function. This baseline always calls one predictor, even for a ticket whose incident code is already in the catalog.

baseline_source <- paste(
  "router <- Predict(\"$outer\", instructions = \"Choose the owning queue.\")",
  "forward <- function(ticket) router(ticket = ticket)",
  sep = "\n"
)

sandbox_factory <- function() mcp_repl_runner(timeout = 45)

baseline <- flex(
  route_signature,
  module_src = baseline_source,
  tools = list(lookup_incident = lookup_incident),
  interpreter_factory = sandbox_factory,
  source_format = "r",
  max_predictor_calls = 1L,
  max_tool_calls = 1L
)

Install mcptools and Posit mcp-repl before running this example:

install.packages("mcptools")
# In a shell:
# uv tool install posit-mcp-repl

mcp-repl’s OS sandbox is the execution boundary for optimizer-authored R source.

Score the answer and the work

An exact-match metric alone cannot distinguish two correct programs. A trace-aware metric can. Here, a correct route scores 1; a correct route that needed a predictor scores 0.9; a wrong route scores 0.

route_metric <- metric_with_trace(
  function(prediction, expected, program_trace) {
    observed <- as.character(prediction$queue)
    wanted <- as.character(expected$queue)
    correct <- identical(observed, wanted)

    calls <- program_trace$metadata$predictor_calls
    if (is.null(calls)) {
      calls <- 0L
    }

    list(
      score = if (!correct) 0 else if (calls == 0L) 1 else 0.9,
      feedback = if (!correct) {
        sprintf("Expected '%s'; got '%s'.", wanted, observed)
      } else if (calls > 0L) {
        paste(
          "Correct, but a predictor ran.",
          "Prefer the incident catalog when it covers the ticket."
        )
      } else {
        "Correct and deterministic."
      }
    )
  },
  field = "queue"
)

Optional: run a live search

The deterministic package test below is the reproducible proof. A live search also lets GEPA propose complete module_src candidates, but remote model output and runtime vary. Training rows generate feedback; validation rows decide which candidate survives. Keep the first run deliberately small and bounded:

llm <- ellmer::chat_openai(model = "gpt-5-mini")

optimized <- compile(
  GEPA(
    metric = route_metric,
    metric_threshold = 0.95,
    population_size = 2L,
    generations = 1L,
    selection = "current_best",
    seed = 20260810L,
    track_best_outputs = TRUE,
    verbose = FALSE
  ),
  baseline,
  trainset = train,
  valset = validation,
  .llm = llm,
  control = optimizer_control(
    max_metric_calls = 30L,
    max_provider_calls = 30L,
    max_elapsed_seconds = 300,
    num_threads = 1L,
    progress = FALSE
  )
)

Remote model output is stochastic, so a seed does not guarantee identical source. Judge the result on held-out behavior, not whether it reproduces one particular program string. The limits stop the search and return its best partial result after 30 metric calls, 30 provider calls, or five active minutes.

Compare quality and calls

Report ordinary accuracy separately from the efficiency objective:

accuracy <- metric_exact_match(field = "queue")

before <- evaluate(
  baseline,
  holdout,
  metric = accuracy,
  .llm = llm,
  .parallel = FALSE,
  .progress = FALSE
)
after <- evaluate(
  optimized,
  holdout,
  metric = accuracy,
  .llm = llm,
  .parallel = FALSE,
  .progress = FALSE
)

count_calls <- function(result, field) {
  sum(vapply(result$metadata, function(metadata) {
    value <- metadata[[field]]
    if (is.null(value)) 0L else as.integer(value)
  }, integer(1)))
}

comparison <- data.frame(
  program = c("baseline", "optimized"),
  accuracy = c(before$mean_score, after$mean_score),
  predictor_calls = c(
    count_calls(before, "predictor_calls"),
    count_calls(after, "predictor_calls")
  ),
  tool_calls = c(
    count_calls(before, "tool_calls"),
    count_calls(after, "tool_calls")
  )
)
comparison

The package’s deterministic regressions fix the GEPA proposal and model responses. One compiles the baseline and selects the hybrid on disjoint training and validation rows; the other runs both reviewed programs on six held-out tickets:

Reviewed program	Holdout accuracy	Predictor calls	Tool calls
Baseline	1.00	6	0
Hybrid	1.00	3	3

That replay verifies proposal, selection, runtime, and call accounting; it is not a promise that every stochastic GEPA run will discover the same source. For a live run, accept the candidate only when held-out quality is at least as good and predictor calls fall:

stopifnot(
  after$mean_score >= before$mean_score,
  count_calls(after, "predictor_calls") <
    count_calls(before, "predictor_calls")
)

The reveal is simple: Flex did not find a better prompt. It found that half the tickets did not need one.

Read the program GEPA selected

Always inspect executable source before promoting it. A useful candidate for this task has a deterministic catalog path and a predictor fallback:

optimized$module_src

The deterministic replay selects this shape:

fallback <- Predict(
  "$outer",
  instructions = "Route tickets without a known incident code."
)
forward <- function(ticket) {
  if (grepl("INC-", ticket, fixed = TRUE)) {
    hit <- lookup_incident(ticket = ticket)
    if (isTRUE(hit$found)) {
      return(Prediction(queue = hit$queue))
    }
  }
  fallback(ticket = ticket)
}

module_src is the program Flex optimized—not an explanation of what happened. The runtime trace supplies the evidence: predictor calls, tool calls, tokens, and the exact source used for each row. GEPA also records the winning candidate ID and validation score:

optimized$config$optimizer$best_candidate_id
optimized$config$optimizer$best_scores
optimized$config$optimizer$per_val_instance_best_candidates

Persist callable tools and factories by registry name rather than embedding them in the artifact:

board <- pins::board_folder("pins")
router_registry <- list(
  lookup_incident = lookup_incident,
  sandbox_factory = sandbox_factory
)

pin_module_config(
  board,
  "issue-router-flex",
  optimized,
  registry = router_registry
)

restored <- pins::pin_read(board, "issue-router-flex") |>
  restore_module_config(registry = router_registry)

Use the same registry names when restoring. Saving and restoring the artifact does not invoke the factory. The registry supplies executable authority, so version and review its definitions as code. Record the model, package versions, data split, metric, and seed beside the pin; pass the model again when running the restored fallback path.

Choose the smaller source language that fits

Flex has two source modes because not every structural question needs code.

Mode	Use it when	Execution boundary
JSON (default)	GEPA may vary a bounded graph of Predict and Chain-of-Thought steps	Parsed and type-checked as data; no code interpreter
R (opt-in)	Candidates need branches, deterministic computation, dynamic predictors, or named tools	Fresh runner from `interpreter_factory`; enforced sandbox required by default

JSON sources use backward references between ordered steps and map their final values to the outer signature. See flex() for the schema and limits. Start there unless the optimization question truly needs R control flow or tools.

Executable Flex exposes a small set of predictor constructors plus the named tools you supply. Those tools execute on the host and keep their closure environments, so treat each one as a privileged capability.

max_predictor_calls limits predictor invocations across the bridge, and max_tool_calls limits direct host-tool requests. A single agentic predictor such as ReAct, CodeAct, or RLM can perform additional work internally; configure that primitive’s own limits as well.

The guest source runs again from the beginning after each bridged request while recorded host responses are replayed. Keep guest-side computation pure, avoid external side effects, and bound loops explicitly.

Generated R must run in an enforced sandbox. r_code_runner() isolates a subprocess but is for source you already trust; do not disable require_sandbox for optimizer-authored code. Executable Flex is synchronous, so evaluate it with .parallel = FALSE; specialized token streaming is not available.

When Flex is the wrong tool

Do not use Flex merely because a task contains more than one step. Prefer a regular module when prompt or demonstration optimization is enough. Prefer an explicit pipeline when people should design and review the workflow. If a decision is entirely deterministic and already known, write an R function.

Use Flex for the unresolved middle: the outcome is measurable, several implementation strategies are plausible, and the choice of model calls, code, or tools is itself what you want to optimize.

For algorithm details, see Advanced Optimization. For intentional API differences and remaining gaps, see dsprrr vs. DSPy.