Flex: optimize a whole program • dsprrr

flex() is experimental: its source format and behavior may change.

Most dsprrr optimizers improve the instructions or examples inside a program whose shape you chose. flex() makes the implementation itself optimizable: GEPA can change how many predictors run, add a deterministic branch, or call a tool you supplied. This page builds a support-ticket router, sets up a search that rewards skipping the model when a lookup table already knows the answer, and shows how to judge and keep the result.

That freedom helps when you can score the outcome but do not yet know the best decomposition. It is unnecessary when the workflow is already clear.

What should change?	Use
Instructions or examples inside a known module	A regular module and teleprompter
A workflow whose steps your team should own	An explicit pipeline
The number or order of Predict and Chain-of-Thought steps	JSON Flex
The choice among R logic, predictors, and selected tools	Executable Flex

A router that learns when not to call a model

Suppose a support team routes tickets to four queues. Some tickets contain an incident code with a known owner. Others need language-model judgment.

A conventional predictor sends every ticket to the model. The Flex objective is more specific: keep routing accuracy, but avoid a predictor call when the incident catalog already contains the answer.

The example runs optimizer-written R code inside Posit’s mcp-repl operating-system sandbox. Install the R client and the executable first:

Rscript -e 'install.packages("mcptools")'
uv tool install posit-mcp-repl   # or: pipx install posit-mcp-repl

Define the task and its evidence

Use separate training and validation rows during search, then reserve a holdout set for the final comparison:

library(dsprrr)

route_signature <- signature(
  "ticket -> queue: enum('database', 'security', 'payments', 'identity')",
  instructions = "Route each support ticket to its owning queue."
)

train <- data.frame(
  ticket = c(
    "[INC-DB-17] checkout failures in us-east",
    "Password reset links expire immediately",
    "Duplicate charge after a plan upgrade",
    "Queries time out after a schema migration"
  ),
  queue = c(
    "database", "identity", "payments", "database"
  )
)

validation <- data.frame(
  ticket = c(
    "[INC-SEC-9] unusual-login burst",
    "Card charged twice for one invoice"
  ),
  queue = c("security", "payments")
)

holdout <- data.frame(
  ticket = c(
    "[INC-PAY-4] capture queue is growing",
    "[INC-DB-17] replicas unavailable",
    "[INC-SEC-9] login spray detected",
    "Reset email arrives after its token expires",
    "Renewal created two invoices",
    "Read traffic fails after failover"
  ),
  queue = c(
    "payments", "database", "security", "identity", "payments", "database"
  )
)

The catalog is an ordinary R closure wrapped in an ellmer tool definition. Its description tells GEPA what the tool returns; supplying it to Flex gives generated code one named capability rather than the rest of the host session.

lookup_incident_fn <- local({
  catalog <- c(
    "INC-DB-17" = "database",
    "INC-SEC-9" = "security",
    "INC-PAY-4" = "payments"
  )

  function(ticket) {
    hit <- names(catalog)[vapply(
      names(catalog),
      grepl,
      logical(1),
      x = ticket,
      fixed = TRUE
    )]
    if (!length(hit)) {
      return(list(found = FALSE))
    }
    list(found = TRUE, queue = unname(catalog[[hit[[1L]]]]))
  }
})

lookup_incident <- ellmer::tool(
  lookup_incident_fn,
  name = "lookup_incident",
  description = paste(
    "Find the owner of an incident code in a support ticket.",
    "Returns list(found = FALSE) when no known code appears; otherwise",
    "returns list(found = TRUE, queue = <owning queue>)."
  ),
  arguments = list(
    ticket = ellmer::type_string(
      description = "Complete support-ticket text."
    )
  )
)

lookup_incident_fn("[INC-DB-17] replicas unavailable")
#> $found
#> [1] TRUE
#> 
#> $queue
#> [1] "database"
lookup_incident_fn("Reset email arrives after its token expires")
#> $found
#> [1] FALSE

Start with one predictor

Executable Flex source defines a top-level forward() function. This baseline always calls one predictor, even for a ticket whose incident code is already in the catalog.

baseline_source <- paste(
  "router <- Predict(\"$outer\", instructions = \"Choose the owning queue.\")",
  "forward <- function(ticket) router(ticket = ticket)",
  sep = "\n"
)

sandbox_factory <- function() mcp_repl_runner(timeout = 45)

baseline <- flex(
  route_signature,
  module_src = baseline_source,
  tools = list(lookup_incident = lookup_incident),
  interpreter_factory = sandbox_factory,
  source_format = "r",
  max_predictor_calls = 1L,
  max_tool_calls = 1L
)

sandbox_factory gives every call a fresh sandboxed R session; mcp-repl’s sandbox is what stands between optimizer-written source and your machine.

Score the answer and the work

An exact-match metric alone cannot distinguish two correct programs. A trace-aware metric can. Here, a correct route scores 1; a correct route that needed a predictor scores 0.9; a wrong route scores 0.

route_metric <- metric_with_trace(
  function(prediction, expected, program_trace) {
    observed <- as.character(prediction$queue)
    wanted <- as.character(expected$queue)
    correct <- identical(observed, wanted)

    calls <- program_trace$metadata$predictor_calls
    if (is.null(calls)) {
      calls <- 0L
    }

    list(
      score = if (!correct) 0 else if (calls == 0L) 1 else 0.9,
      feedback = if (!correct) {
        sprintf("Expected '%s'; got '%s'.", wanted, observed)
      } else if (calls > 0L) {
        paste(
          "Correct, but a predictor ran.",
          "Prefer the incident catalog when it covers the ticket."
        )
      } else {
        "Correct and deterministic."
      }
    )
  },
  field = "queue"
)

Run the search

GEPA proposes complete module_src candidates. Training rows generate feedback; validation rows decide which candidate survives. Keep the first run small and bounded:

llm <- ellmer::chat_openai(model = "gpt-6-luna")

optimized <- compile(
  baseline,
  GEPA(
    metric = route_metric,
    metric_threshold = 0.95,
    population_size = 2L,
    generations = 1L,
    selection = "current_best",
    seed = 20260810L,
    track_best_outputs = TRUE,
    verbose = FALSE
  ),
  trainset = train,
  valset = validation,
  .llm = llm,
  control = optimizer_control(
    max_metric_calls = 30L,
    max_provider_calls = 30L,
    max_elapsed_seconds = 300,
    num_threads = 1L,
    progress = FALSE
  )
)

metric_threshold = 0.95 decides which training rows GEPA shows its proposer as failures: every row scoring below 0.95. With this metric that includes correct routes that needed a predictor (0.9), so their feedback (“Prefer the incident catalog…”) reaches the proposer.

Model output varies, so a seed does not guarantee identical source. Judge the result on held-out behavior, not on whether it reproduces one program string. The limits stop the search and return its best partial result after 30 metric calls, 30 provider calls, or five minutes.

Compare quality and calls

Report ordinary accuracy separately from the efficiency objective:

accuracy <- metric_exact_match(field = "queue")

before <- evaluate(
  baseline,
  holdout,
  metric = accuracy,
  .llm = llm,
  .progress = FALSE
)
after <- evaluate(
  optimized,
  holdout,
  metric = accuracy,
  .llm = llm,
  .progress = FALSE
)

count_calls <- function(result, field) {
  sum(vapply(result$metadata, function(metadata) {
    value <- metadata[[field]]
    if (is.null(value)) 0L else as.integer(value)
  }, integer(1)))
}

comparison <- data.frame(
  program = c("baseline", "optimized"),
  accuracy = c(before$mean_score, after$mean_score),
  predictor_calls = c(
    count_calls(before, "predictor_calls"),
    count_calls(after, "predictor_calls")
  ),
  tool_calls = c(
    count_calls(before, "tool_calls"),
    count_calls(after, "tool_calls")
  )
)
comparison

This page shows no live results. dsprrr’s own tests run this workflow with a mock model that routes every ticket correctly and, when GEPA asks for a new program, returns a hybrid written by hand (shown in the next section). They check that GEPA selects that program on the validation rows and that Flex runs and counts it correctly; they say nothing about whether GEPA would write the hybrid itself. On the six holdout tickets the tests record:

Program in the test	Holdout accuracy	Predictor calls	Tool calls
Baseline	1.00	6	0
Hand-written hybrid	1.00	3	3

For a live run, accept the candidate only when held-out quality is at least as good and predictor calls fall:

stopifnot(
  after$mean_score >= before$mean_score,
  count_calls(after, "predictor_calls") <
    count_calls(before, "predictor_calls")
)

Read the program GEPA selected

Always read executable source before you use it:

optimized$module_src

A useful candidate for this task has a deterministic catalog path and a predictor fallback, like the hand-written hybrid that the package tests feed to GEPA:

fallback <- Predict(
  "$outer",
  instructions = "Route tickets without a known incident code."
)
forward <- function(ticket) {
  if (grepl("INC-", ticket, fixed = TRUE)) {
    hit <- lookup_incident(ticket = ticket)
    if (isTRUE(hit$found)) {
      return(Prediction(queue = hit$queue))
    }
  }
  fallback(ticket = ticket)
}

module_src is the program itself, not an explanation of what happened. The evidence is in each row’s run metadata, which records the module_src it ran along with predictor calls, tool calls and tokens. GEPA also records the winning candidate ID and validation score:

optimization_result(optimized)$lineage$best_candidate_id
optimization_result(optimized)$extensions$gepa$best_scores
optimization_result(optimized)$extensions$gepa$per_val_instance_best_candidates

Keep the selected program

Save the selected source with your code, and rebuild the module from it where you deploy. The tool and the sandbox factory stay in your code, where you can review and version them:

writeLines(optimized$module_src, "ticket-router.R")

router <- flex(
  route_signature,
  module_src = paste(readLines("ticket-router.R"), collapse = "\n"),
  tools = list(lookup_incident = lookup_incident),
  interpreter_factory = sandbox_factory,
  source_format = "r",
  max_predictor_calls = 1L,
  max_tool_calls = 1L
)

The saved file is Flex source for the sandbox, not a script to source(). Record the model, package versions, data split, metric and seed next to it.

Choose a source format

Flex has two source formats because not every structural question needs code.

Mode	Use it when	Execution boundary
JSON (default)	GEPA may vary a bounded graph of Predict and Chain-of-Thought steps	Parsed and type-checked as data; no code interpreter
R (opt-in)	Candidates need branches, deterministic computation, dynamic predictors, or named tools	Fresh runner from `interpreter_factory`; enforced sandbox required by default

JSON sources use backward references between ordered steps and map their final values to the outer signature; flex() documents the schema. Start with JSON unless the optimization question needs R control flow or tools.

Executable Flex exposes a small set of predictor constructors plus the named tools you supply. Those tools execute on the host and keep their closure environments, so treat each one as a privileged capability.

max_predictor_calls limits predictor invocations across the bridge, and max_tool_calls limits direct host-tool requests. A single agentic predictor such as ReAct, CodeAct, or RLM can perform additional work internally; configure that primitive’s own limits as well.

The guest source runs again from the beginning after each bridged request while recorded host responses are replayed. Keep guest-side computation pure, avoid external side effects, and bound loops explicitly.

Generated R must run in an enforced sandbox. r_code_runner() isolates a subprocess but is for source you already trust; do not disable require_sandbox for optimizer-authored code. Each sandboxed call has size limits, listed in How RLM works. Executable Flex runs rows one at a time and does not stream tokens.

When Flex is the wrong tool

Do not use Flex merely because a task contains more than one step. Prefer a regular module when prompt or demonstration optimization is enough. Prefer an explicit pipeline when people should design and review the workflow. If a decision is entirely deterministic and already known, write an R function.

Use Flex for the unresolved middle: the outcome is measurable, several implementation strategies are plausible, and the choice of model calls, code, or tools is itself what you want to optimize.

To compare Flex with the other optimizers, see Choose an optimizer.