# Findings — mixed-pipeline (config repair) Task shape: agent must (a) read three input files (broken JSONC-ish config, env file, SPEC), (b) author a Python stdlib-only fixer, (c) run it, (d) verify the output matches SPEC (strict JSON, key order, port = int from env, no placeholder leakage, features booleans, upstreams list of {name,url} dicts). Small, deterministic, single-deliverable — read → write → run → verify. No ambient search/exploration. All 15 runs passed with score 1.0. --- ## 0. What every arm did identically (the task's floor) Before contrasting the arms, three behaviors are universal across all 15 reps and so are properties of the task, not of any post-train: 1. **Batched triple-read opener.** All 15 runs open with the same move — three `read_file` calls prepared in a single turn (dashboard.json.broken, app.env, SPEC.md), then one reasoning block. The openers are nearly interchangeable: fable-r1 "Let me start by reading all three files to understand what needs to be done" (`bakeoff-fable--mixed-pipeline--r1--1785056598`); stock-r1 "Let me start by reading the contents of the workspace to understand what I'm working with" (`bakeoff-stock--mixed-pipeline--r1--1785044212`); thinkingcap-r3 "Let me start by reading all three files to understand what I'm working with" (`bakeoff-thinkingcap--mixed-pipeline--r3--1785050375`). 2. **Text-preprocess-then-`json.loads` architecture.** No arm attempted a hand-rolled tokenizer or a real state-machine JSONC parser. Every rep is a variant of: strip `//` comments → swap single→double quotes → strip trailing commas → resolve `${...}` → `json.loads` as validator → `json.dump`. And because the architecture is identical, the failure modes are identical. 3. **The `//`-in-URL trap and the newline-then-`}` comma trap.** Of 15 reps, ~11 hit the comment-stripper eating `http://` (the `//` looks like a comment to a naive regex), and a large fraction separately hit trailing commas where the `,` sits on one line and the closing `}`/`]` on the next. These are the task's two hidden teeth, and they bite every arm roughly equally. So the interesting variation is in *recovery manner*, not avoidance. --- ## 1. Per-arm behavioral profiles ### stock (Qwen/Qwen3.6-27B) **Approach shape — checklist planner.** stock consistently reasons about the SPEC as a numbered requirement list *before* writing code. stock-r1 lays out the five defects line-by-line, then a numbered plan ("1. Strip comments... 2. Replace `${DASHBOARD_PORT}`... 3. Convert single quotes..."), then traces the comment-stripper char-by-char in its head before ever running (`bakeoff-stock--mixed-pipeline--r1--1785044212`). stock-r2 does the same explicit enumeration, including pre-empting the URL hazard: "I need to be careful about URLs containing `//`. Like `'http://127.0.0.1:9001'` — the `//` in the URL is inside quotes" (`bakeoff-stock--mixed-pipeline--r2--1785044341`). **Thinking-trace character.** Moderate length (1.0k–11k reason chars), heavy on "let me think more carefully" re-reasoning after a failure. The most striking trait is **pre-run mental simulation that still fails**: in stock-r1 the model walks through its scanner, concludes the single-quote tracking will protect the URL — "when we enter the single-quoted string `'http://127.0.0.1:9001'`, we toggle `in_sq = True`. Then scanning `http://127...`, at `//` we check `not (in_sq or in_dq)` — but `in_sq` is True, so we don't break! ... Looks correct. Let me run it." (`bakeoff-stock--mixed-pipeline--r1--1785044212`) — then exit 1. The trace is locally coherent and globally wrong. stock-r3's 11k-char trace is the deepest, descending into a long audit of why the trailing-comma regex fails across newlines: "the regex only looks at current line but `\n` is between `,` and `}`. After stripping trailing spaces from each line, the `,` stays because it's on the same line as the value" (`bakeoff-stock--mixed-pipeline--r3--1785044483`). **Tool decisions.** stock leans on `patch` for fixes (4 patches in r3) rather than whole-file rewrites. It is the only arm to try the **`(?