Investigate a Release Regression with an RLM • dsprrr

An RLM is useful when the answer is buried in a large object and you do not know the right exploration path in advance. It keeps the object in an R environment, lets the model inspect it with R code, and returns only selected results to the model between steps.

This worked example asks an RLM to investigate a checkout regression. The deterministic fixture contains 40,000 sessions (about 3.6 MB as row-oriented JSON) and 200 change records. The expected finding is fixed and independently checkable:

release:      2.4.0
cohort:       platform=mobile / plan=pro
before_rate:  0.92
after_rate:   0.61
drop_pp:      31
change_id:    CHG-1842
evidence:     Mobile Pro token refresh: retry budget changed from 3 to 0 and timeout from 8 s to 800 ms.

The model must discover which low-cardinality categorical fields define the affected cohort, calculate the change, and find the most relevant release note. It does not receive the complete session table in its prompt.

Why use an RLM here?

The task combines three properties:

the source object is expensive and distracting to serialize into a prompt;
the useful grouping is not supplied in the question; and
the answer needs both exact aggregation and interpretation of a small piece of text.

A regular Predict or ChainOfThought module would put the supplied context in token space. ProgramOfThought is a better fit when the required computation is already known. A deterministic R pipeline is best once the investigation pattern is stable. RLM occupies the exploratory middle: the model decides what to inspect, uses R for exact work, and revises its plan from the results.

Build the deterministic incident

The fixture uses no random numbers. Every cohort contains exactly 5,000 sessions per release.

n <- 5000L

sessions <- expand.grid(
  release = c("2.3.9", "2.4.0"),
  platform = c("desktop", "mobile"),
  plan = c("free", "pro"),
  within_group = seq_len(n),
  KEEP.OUT.ATTRS = FALSE,
  stringsAsFactors = FALSE
)

base_rate <- c(
  "desktop.free" = 0.88,
  "mobile.free" = 0.84,
  "desktop.pro" = 0.94,
  "mobile.pro" = 0.92
)

cohort_key <- paste(sessions$platform, sessions$plan, sep = ".")
conversion_rate <- unname(base_rate[cohort_key])
conversion_rate[
  sessions$release == "2.4.0" & cohort_key == "mobile.pro"
] <- 0.61

sessions$converted <-
  sessions$within_group <= n * conversion_rate

Most change records are irrelevant. One describes a checkout-authentication change for the affected release and cohort.

changes <- data.frame(
  change_id = sprintf("CHG-%04d", 1701:1900),
  release = rep(c("2.3.9", "2.4.0"), each = 100),
  component = "miscellaneous",
  note = "Routine maintenance with no expected checkout impact."
)

target <- which(changes$change_id == "CHG-1842")
changes$component[target] <- "checkout-auth"
changes$note[target] <- paste(
  "Mobile Pro token refresh:",
  "retry budget changed from 3 to 0 and timeout from 8 s to 800 ms."
)

The question does not reveal the affected dimensions:

question <- paste(
  "Checkout conversion fell after release 2.4.0.",
  "Find the finest low-cardinality categorical cohort with the largest",
  "before/after drop; do not stop at a marginal roll-up that dilutes the",
  "change. Identify its dimensions and values, quantify the rates, and cite",
  "the change-log record that is the strongest candidate explanation."
)

Define the investigation

The output signature makes the evidence checkable. SUBMIT() must provide all of these fields with compatible types.

library(dsprrr)
library(ellmer)

incident_signature <- signature(
  paste(
    "sessions, changes, question ->",
    "release: string, cohort: string,",
    "before_rate: number, after_rate: number,",
    "drop_pp: number, change_id: string, evidence: string"
  ),
  instructions = paste(
    "Inspect the schema and low-cardinality categorical fields; exclude the",
    "release, outcome, and row-index fields from candidate cohort dimensions.",
    "Find the finest low-cardinality cohort with the largest before/after",
    "drop; do not stop at a marginal roll-up that dilutes the change.",
    "Quantify it and cite the strongest matching change record.",
    "Format cohort in source-column order as '<dimension>=<value> / ...'.",
    "Copy the selected change note verbatim into evidence. Treat that record as",
    "evidence, not proof of causation."
  )
)

investigator <- rlm_module(
  incident_signature,
  interpreter_factory = function() {
    r_code_runner(timeout = 30, persistent = TRUE)
  },
  max_iterations = 8,
  max_llm_calls = 0L,
  max_output_chars = 10000
)

This example deliberately uses persistent r_code_runner() because the input is a large, rich R object and the fixture and generated trajectory are assumed trusted. A callr subprocess provides process isolation, not an operating-system security sandbox. Do not use this configuration for adversarial context or untrusted generated code.

The factory creates one runner for the invocation. State remains available between RLM iterations, and dsprrr shuts down the runner when the invocation ends.

Run and retain the evidence

Request structured output so the answer and its trajectory travel together:

result <- run(
  investigator,
  sessions = sessions,
  changes = changes,
  question = question,
  .llm = chat_openai(),
  .return_format = "structured"
)

result$output

This article does not claim a recorded model run. Model-generated code and the number of iterations can vary. The fixed output at the top is the result that a successful run must recover from the deterministic fixture.

What a successful trajectory looks like

The exact R code may differ, but the useful work should be recognizable in four steps.

1. Orient to the objects

The first step should inspect shape and schema, not print the full data:

list(
  session_dim = dim(.context$sessions),
  session_fields = names(.context$sessions),
  releases = table(.context$sessions$release),
  change_fields = names(.context$changes)
)

2. Calculate cohort-level changes

R performs the aggregation exactly:

rates <- aggregate(
  converted ~ release + platform + plan,
  data = .context$sessions,
  FUN = mean
)

before <- subset(rates, release == "2.3.9")
after <- subset(rates, release == "2.4.0")
deltas <- merge(
  before,
  after,
  by = c("platform", "plan"),
  suffixes = c("_before", "_after")
)
deltas$drop_pp <-
  100 * (deltas$converted_before - deltas$converted_after)
deltas[order(-deltas$drop_pp), ]

The small printed table, rather than all 40,000 rows, enters the next model turn.

3. Inspect candidate changes

Once the cohort is known, the model can narrow the change log:

candidate <- subset(
  .context$changes,
  release == "2.4.0" &
    grepl(
      "token|retry",
      paste(component, note),
      ignore.case = TRUE
    )
)

4. Submit typed evidence

The closing step returns the calculated values and the relevant record:

winner <- deltas[which.max(deltas$drop_pp), ]
SUBMIT(
  release = after$release[[1L]],
  cohort = paste0(
    "platform=", winner$platform,
    " / plan=", winner$plan
  ),
  before_rate = winner$converted_before,
  after_rate = winner$converted_after,
  drop_pp = winner$drop_pp,
  change_id = candidate$change_id[[1L]],
  evidence = candidate$note[[1L]]
)

If a submitted value is missing or has the wrong type, the RLM receives that validation error and can correct the submission on a later iteration.

Inspect and validate the result

The structured result carries the trajectory produced during this invocation:

trajectory <- result$metadata$repl_history

vapply(trajectory, function(step) step$code, character(1))
vapply(trajectory, function(step) step$success, logical(1))

Validate the model output against an independent calculation rather than trusting its prose:

rates <- aggregate(
  converted ~ release + platform + plan,
  data = sessions,
  FUN = mean
)

before <- subset(rates, release == "2.3.9")
after <- subset(rates, release == "2.4.0")
comparison <- merge(
  before,
  after,
  by = c("platform", "plan"),
  suffixes = c("_before", "_after")
)
comparison$drop_pp <-
  100 * (comparison$converted_before - comparison$converted_after)
oracle <- comparison[which.max(comparison$drop_pp), ]
candidate <- subset(
  changes,
  release == "2.4.0" &
    grepl("token|retry", paste(component, note),
      ignore.case = TRUE
    )
)

stopifnot(
  nrow(sessions) == 40000L,
  nrow(candidate) == 1L,
  identical(candidate$change_id, "CHG-1842"),
  identical(result$output$release, "2.4.0"),
  identical(result$output$cohort, "platform=mobile / plan=pro"),
  isTRUE(all.equal(result$output$before_rate, oracle$converted_before)),
  isTRUE(all.equal(result$output$after_rate, oracle$converted_after)),
  isTRUE(all.equal(result$output$drop_pp, oracle$drop_pp)),
  identical(result$output$change_id, "CHG-1842"),
  identical(result$output$evidence, candidate$note[[1L]])
)

Recursive queries are optional

sub_lm = NULL inherits the outer model supplied to run(). If generated code calls llm_query() or llm_query_batched(), dsprrr performs those calls in the host process and replays their results into the same code evaluation. A separate, cheaper model can handle those focused reads:

investigator <- rlm_module(
  incident_signature,
  interpreter_factory = function() {
    r_code_runner(timeout = 30, persistent = TRUE)
  },
  sub_lm = chat_openai(),
  max_llm_calls = 4
)

This incident does not require a sub-query: R aggregation plus one targeted change record is enough. Do not add recursive calls merely because the module supports them.

Choose the runner deliberately

Configuration	Use it for	Boundary
`rlm()` with no runner arguments	Compact, JSON-compatible context	Fresh managed OS sandbox; network disabled, workspace writes allowed, bounded transport
`interpreter_factory = function() mcp_repl_runner()`	Explicit managed-sandbox configuration	OS sandbox, fresh runner per invocation, bounded transport
`r_code_runner(persistent = TRUE)`	Trusted tasks with rich or large R objects	Persistent callr process with the host user’s permissions

The managed MCP path is the default for the one-call rlm() helper. It requires the suggested R package mcptools and the external mcp-repl executable. It disables network access but still permits writes inside the allowed workspace. The final JSON-RPC request must fit the 7 KB wire bound; dsprrr tries a gzip/base64 wrapper when the raw request is too large. Each encoded RLM control frame must fit 3,000 bytes. Large data frames and model objects need a runner or resource adapter that preserves those values without forcing them through the compact MCP request. Declared host tools execute outside the guest sandbox with the host process’s permissions.

For a compact document, the managed one-call form is enough:

answer <- rlm(
  "document, question -> answer",
  document = "Owner: team-a\nCommitment: publish the audit by Friday",
  question = "Which commitments have no owner?",
  .llm = chat_openai(),
  .max_iterations = 4L,
  .max_llm_calls = 0L
)

See How the RLM Works for runner lifecycle, recursive calls, budgets, and failure behavior.

Turn discovery into a program

RLM is a poor default for a known report. If several investigations repeatedly aggregate the same columns and join the same records, encode that path in R or a dsprrr pipeline. If the best implementation is itself the search problem, evaluate it over labeled cases and consider Flex.

Use RLM to discover the path. Use deterministic code once you know it.