Agentic optimization harnesses • dsprrr

Most dsprrr teleprompters own both candidate generation and selection. AutoResearch() and MetaHarness() split those jobs: an LLM agent proposes experiments, while a trusted R evaluator applies validated edits, runs the program, enforces budgets, and keeps the best result. This page shows how to run both harnesses on a two-step pipeline, keep the agent’s code in a sandbox, budget and resume a run, and read what happened.

Both harnesses can jointly edit the instructions and templates of every optimizable module in a program. That makes them useful when the problem is not one weak prompt but the contract between several pipeline stages. They do not change the program’s structure or run optimizer-written module source. To let an optimizer change the implementation itself (predictor structure, deterministic R, or tools), use the experimental flex(); see Flex: optimize a whole program.

Choose the search owner

Harness	Who owns the loop	Agent memory	Candidates
`AutoResearch()`	One research agent	Persistent across experiments	One per `propose` action
`MetaHarness()`	The R outer loop	Fresh proposer each iteration	A batch per iteration

AutoResearch() fits when one researcher should pursue a line of inquiry, use earlier results as working memory, branch, and decide when to stop. MetaHarness() fits when you want independent proposals, explicit frontier selection, and less dependence on one conversation’s hidden state.

With either harness, dsprrr:

evaluates the unmodified program as a baseline;
gives every candidate a content fingerprint and a parent id;
rejects unknown module paths and unsupported fields;
skips candidates identical to one already evaluated;
charges every evaluation to the optimizer_control() budget; and
returns the best scored program, including when a budget stops the run.

The example program

The examples optimize a two-step pipeline that answers questions from a short context. The first step quotes evidence, the second answers from it, so the two instructions have to agree on a format. The data has eight rows, which is enough to show the shape; a real run needs many more.

library(dsprrr)

find_evidence <- module(signature(
  "context, question -> evidence",
  instructions = "Restate the question, then quote the context sentences that answer it."
))
answer_from_evidence <- module(signature(
  "evidence -> answer",
  instructions = "Answer the restated question from the quoted sentences only."
))
qa_program <- pipeline(find_evidence, answer_from_evidence)

examples <- tibble::tribble(
  ~context, ~question, ~answer,
  "The Oslo office opened in 2019 and has 42 staff.", "When did the Oslo office open?", "2019",
  "Refunds are processed within 5 business days.", "How many business days do refunds take?", "5",
  "The Lisbon office has 18 staff.", "When did the Lisbon office open?", "unknown",
  "The API allows 600 requests per minute.", "How many requests per minute does the API allow?", "600",
  "Maria Lopez leads the data platform team.", "Who leads the data platform team?", "Maria Lopez",
  "Backups run nightly at 02:00 UTC.", "How long are backups kept?", "unknown",
  "Support tickets are answered within 4 hours.", "How fast are support tickets answered?", "4 hours",
  "The Berlin office moved to Mitte in 2021.", "How many staff work in the Berlin office?", "unknown"
)
trainset <- examples[1:4, ]
valset <- examples[5:8, ]

Keep code execution sandboxed

The agent can ask to run R code to analyze the program, the frontier, and the training rows. The harnesses require an operating-system sandbox for that code by default. mcp_repl_runner() provides one by launching Posit’s open-source mcp-repl, a persistent R session behind one MCP tool. Install the executable and the R client:

uv tool install posit-mcp-repl   # or: pipx install posit-mcp-repl
Rscript -e 'install.packages("mcptools")'

Then create the runner:

runner <- mcp_repl_runner(
  sandbox = "workspace-write",
  timeout = 30
)

The managed sandbox disables network access and limits writes to the workspace and temporary paths. The session stays alive across calls, so an agent can load a package or build an analysis once and keep working with it; runner$reset() starts a clean session. Code, results and context have size limits, listed in How RLM works.

A runner built around your own repl = function(input, timeout_ms) is different: dsprrr did not launch that server and cannot vouch for its isolation, so the runner reports sandboxed = FALSE and the harnesses reject it. r_code_runner() is rejected for the same reason: its callr process keeps your user’s permissions and is not a security boundary.

Set sandbox = FALSE on a harness only to remove the sandbox action entirely. Any supplied runner is then ignored; the agent can still propose edits but cannot run code.

AutoResearch: one persistent research session

Use separate chats for the program under test and the research agent. That keeps task traces, model choice, and cost attribution apart.

library(ellmer)

task_chat <- chat_openai(model = "gpt-6-luna")
research_chat <- chat_anthropic(model = "claude-sonnet-4-5")

metric <- metric_with_feedback(
  function(prediction, expected) {
    correct <- identical(prediction$answer, expected$answer)
    list(
      score = as.numeric(correct),
      feedback = if (correct) {
        "Correct."
      } else {
        paste0(
          "Expected '", expected$answer,
          "' but got '", prediction$answer, "'."
        )
      }
    )
  },
  field = "answer"
)

research <- AutoResearch(
  metric = metric,
  max_iterations = 16L,
  patience = 5L,
  target_score = 0.95,
  max_context_examples = 20L,
  max_feedback_examples = 8L,
  seed = 42L
)

compiled <- compile(
  qa_program,
  research,
  trainset,
  valset = valset,
  .llm = task_chat,
  .agent_llm = research_chat,
  runner = runner,
  objective = paste(
    "Answer from supplied evidence.",
    "Do not invent facts when the context is insufficient."
  ),
  .cache = FALSE
)

The harnesses also accept metric_with_trace() metrics, which can score a row from its status, ordered events, and module metadata. Candidate selection stays in the trusted R evaluator either way.

On each turn the agent must choose one action:

Action	What the agent does
`sandbox`	Runs R code against a `.context` holding the objective, the editable program manifest, visible training rows, and the current frontier
`propose`	Branches from a known candidate and submits one candidate: complete replacement instructions or templates at one or more module paths
`finish`	Stops, when another experiment is unlikely to help

The agent decides the research policy, but it cannot write into the module or the evaluator directly. dsprrr re-validates and applies every candidate.

Meta-Harness: fresh proposers, trusted selection

Meta-Harness creates a fresh proposer chat for each outer iteration, so the proposer reasons from the persisted evidence rather than from earlier conversation. Pass an ellmer Chat; dsprrr clones it and clears its turns before each iteration:

harness <- MetaHarness(
  metric = metric,
  max_iterations = 8L,
  max_candidates_per_iteration = 4L,
  frontier_size = 8L,
  patience = 3L,
  seed = 42L
)

compiled <- compile(
  qa_program,
  harness,
  trainset,
  valset = valset,
  .llm = task_chat,
  .agent_llm = research_chat,
  runner = runner,
  objective = "Improve factual accuracy without increasing abstention errors."
)

Meta-Harness rejects agents that are not ellmer chats, or that cannot be cloned, because reusing them could carry hidden conversation state across iterations.

Within an iteration, the proposer may request a few sandbox steps before it submits its batch. The outer loop then:

builds each proposal from its declared parent;
rejects invalid or duplicate candidates;
evaluates every unique candidate that fits the remaining budget;
updates the scored frontier; and
checkpoints the new lineage before starting a fresh proposer.

The agent can inspect and hypothesize, but the scorer, the budget, and the accepted state stay with the host.

Joint pipeline edits

Candidates address modules by the paths that named_parameters() reports. They stay stable within one program:

names(named_parameters(qa_program))
#> [1] "$/steps/1" "$/steps/2"

One candidate may replace fields at several paths. Here, a candidate can change the evidence step to write NONE when the context does not answer the question, and change the answer step to reply unknown when it sees NONE. The candidate ledger stores the complete resulting snapshot, not only the patch, so fingerprints, resumes, and comparisons are exact.

Compiled and frozen modules are boundaries: the harnesses do not edit them or anything inside them. To let the agent edit an inner component, change the program before compiling rather than asking the agent to get around the boundary.

Budget every kind of work

The shared optimizer ledger counts baseline and candidate evaluations as trials, and tracks program calls and tokens during evaluation, proposer calls and tokens, metric calls, cost, elapsed time, and errors:

control <- optimizer_control(
  max_trials = 25L,
  max_metric_calls = 200L,
  max_provider_calls = 250L,
  max_total_tokens = 500000L,
  max_cost = 10,
  max_elapsed_seconds = 1800,
  checkpoint_path = "meta-harness-checkpoint.rds"
)

compiled <- compile(
  qa_program,
  harness,
  trainset,
  valset = valset,
  .llm = task_chat,
  .agent_llm = research_chat,
  runner = runner,
  control = control
)

A finite provider, token, or cost cap stops the run when a chat cannot report its usage, rather than running unmetered. A stopped run returns its best partial program and says why it stopped:

optimization_result(compiled)$budget
optimization_result(compiled)$stop_reason
optimization_result(compiled)$status

Resume and audit a run

Both harnesses support optimizer checkpoints. Resume with the same program, data, metric, and harness configuration; resource limits may be raised:

resumed <- compile(
  qa_program,
  harness,
  trainset,
  valset = valset,
  .llm = task_chat,
  .agent_llm = research_chat,
  runner = runner,
  control = optimizer_control(
    max_trials = 40L,
    max_cost = 20,
    checkpoint_path = "meta-harness-checkpoint.rds",
    resume = TRUE
  )
)

The candidate ledger is attached to the returned module:

candidates <- optimization_result(resumed)$trials
candidates[, c(
  "id",
  "parent_id",
  "name",
  "iteration",
  "score",
  "selected",
  "status"
)]

# Complete per-candidate program state and evaluation feedback:
candidates$snapshot[[1]]
candidates$feedback[[1]]

optimization_result(resumed)$extensions$meta_harness$frontier_ids
optimization_result(resumed)$extensions$meta_harness$events
optimization_result(resumed)$stop_reason

With log_dir in the teleprompter or optimizer_control(), evaluations are also written through TrialLog.

Resuming restores candidate lineage, evaluation evidence, budgets, and random-number state. It does not restore the agent’s conversation or the mcp-repl session; the resumed run uses the chat and runner you pass.

Compose harnesses with Omni

The harnesses are ordinary teleprompters, so Omni() can run them as explorers. Pass their agent and runner through the per-explorer arguments:

omni <- Omni(
  metric = metric,
  explorers = list(
    research = AutoResearch(metric = metric, max_iterations = 6L),
    meta = MetaHarness(metric = metric, max_iterations = 4L)
  ),
  continuation = BootstrapFewShotWithRandomSearch(metric = metric)
)

compiled <- compile(
  qa_program,
  omni,
  trainset,
  valset = valset,
  .llm = task_chat,
  explorer_compile_args = list(
    research = list(.agent_llm = research_chat, runner = runner),
    meta = list(.agent_llm = research_chat, runner = runner)
  )
)

Do not use a harness as the continuation. Unless the seed wins, the continuation receives an explorer’s compiled program, which the harnesses refuse to edit; Omni then records the error and keeps the exploration winner. Keep parallel = FALSE (the default) with harness explorers: parallel mode copies explorer arguments into separate worker processes, and a runner’s mcp-repl connection belongs to your session. See Compose optimizers with Omni for the rest of the Omni workflow.

Inspiration and differences

The design credits four open projects:

Andrej Karpathy’s autoresearch, for the explicit baseline, experiment, score, keep-or-revert loop;
the AutoResearch and Meta-Harness engines of GEPA optimize-anything, for a common trusted evaluation boundary and optimizer composition;
the Meta-Harness paper, for the fresh-proposer outer loop; and
Posit’s mcp-repl, for persistent R execution with OS-level sandboxing.

dsprrr does not launch a coding-agent CLI, expose an HTTP evaluator, or ask an agent to edit arbitrary files. It uses provider-neutral ellmer chats, typed actions, complete module snapshots, the package’s optimizer ledger, and checkpoints. Benchmark these harnesses on your own task and budget; results reported by the source projects do not automatically transfer to a different model, metric, or program.