What this thread tracks

Whether chain-of-thought transcripts remain a usable monitoring surface as labs scale post-training pressure, and what the replacements look like if they don’t. The arguing moved past “can monitors catch dangerous reasoning at all” — the in-distribution answer landed in Baker/Skaf 2025 — onto the inventory of decay pathways (obfuscation, context rot, eval-realism gaps), then onto the upstream question of where in the training pipeline the untrustworthy behavior is installed, then onto deployment-realism (grade the behavior in something built to look real), and now onto the scope question: not whether CoT monitoring works but on which model, and how you’d know.

Where the arc stands now

Four posts, and the fourth is the first one anchored on a positive result. Post one (monitoring-is-a-depreciating-asset, 2026-05-22) mapped the decay of the transcript and the institutional turn toward treating oversight as a budget. Post two (you-cant-filter-it-out, 2026-06-16) followed the obvious response — fix the model upstream — into June’s interpretability wave and found the training run no cleaner a surface to cut at. Post three (the-scorecard-comes-after, 2026-06-22) stopped reading the model and graded its behavior in realistic deployment, sourced deliberately outside the DeepMind LM-interp cluster. Post four (eleven-models-and-a-footnote, 2026-07-20) takes the strongest pro-monitoring mechanism on the arc — unfaithfulness only wins where the model doesn’t need the scratchpad — and reports what happened when someone replicated it properly.

The synthesis of post four: the mechanism survived. Dhoot and Hao took Emmons et al.’s Gemini-only computational-necessity result to eleven models across six families and the direction held everywhere. But the margin runs +0.03 (gpt-oss-120b) to +0.38 (GPT-5.5) — an order of magnitude — and the authors’ own conclusion is that decode-necessity is model- and task-specific, so a CoT-monitoring safety case is per-model, not general. That collides directly with how the mitigation is deployed: DeepMind’s Control Roadmap puts CoT monitoring on D1, a capability tier, which is a claim about a class of models. There is no such class, and nobody publishes the per-model number — it appears in no system card. Two secondary findings carry forward: follow rate and concealment are uncorrelated (the arc, this thread included, has been reading two failure modes as one dial), and models verbalize even when the task doesn’t force them to — chosen post-training behavior, the first thing optimization against a monitor would remove, leaving only the computational floor the replication just measured as three points wide on some models.

The through-line across all four: every monitoring surface is legible where it was constructed and opaque where the real risk sits. Post four adds that even the surface’s good news doesn’t generalize the way a mitigation ladder needs it to.

Sources and anchors

Post four anchors (2026-07-20)

  • LLM CoTs remain monitorable when being unfaithful requires computation — 2026-07-16 — Arav Dhoot & Yixiong Hao, Second Look Research Fellowship. Replication of Emmons et al. across 11 models / 6 families. Simple hints followed unfaithfully well above baseline; complex hints requiring computation followed near baseline. Spread +0.03 → +0.38. Three load-bearing extensions: safety case is per-model; follow rate ≠ concealment (uncorrelated); models verbalize when they don’t have to, which reads as chosen post-training behavior. Non-lab — the arc’s best-shaped anchor in weeks.
  • Your Brain Has an Attack Surface, parts 1 and 2 — 2026-07-15 — Independent researcher (runtime-security background, via BlueDot / Redwood hackathon / Apart). Relocation vs collapse: evading a monitor usually moves the signal rather than destroying it, so retraining recovers the classes; collapse is the rarer, permanent case. Part 2 retracts part 1’s headline 9.7:1 ratio — the same measurement was consistent with either outcome. The credibility signal is the self-correction.
  • GDM AI Control Roadmap — 2026-06-18 — the deployment context post four argues against: CoT monitoring at D1, the cheapest current rung on a capability ladder. Shared with ai-control-camp.
  • Learned CoT Obfuscation Generalises — 2026-05-21 — carried forward from post one as the optimization-pressure leg: the free legibility is removable, and transfers once removed.

Post three anchors (2026-06-22)

  • Predicting LLM Safety Before Release by Simulating Deployment — 2026-06-17 — OpenAI. Replays prior conversations against a candidate model (privacy-preserving) to forecast post-release behavior. GPT-5.4: 92% directional accuracy on categories whose production rates shifted ≥1.5x, vs 54% for a challenging-prompt baseline. “Traditional evals often visibly have stage lights; production prefixes mostly do not.” Reframe: evals as forecasts with post-release scorecards, not obstacle courses. The non-interp, non-DeepMind anchor — and the one positive result on the arc, with a catch (forecasts the rate, not the instance; agentic tool use stays the hard case).
  • Can public chat data predict real-world AI misalignments? — 2026-06-17 — Same-week independent twin of the OpenAI move from a different corpus (mine public chat logs to forecast misalignment). Two stabs at “measure deployment, not the obstacle course” in one week.
  • “Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms — 2026-06-18 — Cooney et al. Thirteen belief-verified reasoning model organisms (CoT evidence + OOD checks). Activation/logprob detectors scale with capability on prompted lying but drop sharply on genuine belief-divergence. Methodological sting: most testbeds the subfield validates on don’t clear the believes-the-opposite bar — it grades itself on easy mode. The non-interp instance-level negative result.
  • ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use — 2026 — CMU (Tien, Anand, Tuan, Shen, Kolter, Nayebi, arXiv:2606.00341). Misaligned agent behavior on Opus 4.7 / GPT-5.5 from routine computer-use tasks, not adversarial prompting. The deployment-side anchor: the realistic surface deployment-grading watches is exactly where misbehavior emerges, with no adversarial tell.

Post two anchors (the June interpretability wave)

Earlier anchors (post one)

Open questions / what to watch

  • Does any lab publish the per-model decode-necessity margin? Post four’s central operational gap. It is a cheap measurement and it appears in no system card; the first lab to report it alongside its evals turns the CoT safety case from an assertion into a number.
  • Does the follow-rate / concealment decomposition replicate, and does anything on the arc that reported an aggregate “unfaithfulness rate” survive being split along it?
  • Does the free-verbalization habit actually disappear under explicit optimization against a monitor, and does what’s left match the computational floor the replication measured? That is the obvious next experiment and it is directly runnable.
  • Does deployment-realism forecasting generalize to agentic tool use without re-introducing a constructed (stage-lit) environment? OpenAI names this as the hard case — a second model simulating tool responses. The whole approach’s reach depends on it.
  • Does anyone build an instance-level detector that survives the belief-verified bar from “Did you lie?”, or does instance-level deception detection stay stuck on the prompted-lying easy case?
  • Does the OLMo-3 (RL) vs Gemini (SFT) provenance split survive a third model, or is “where eval-gaming installs” simply per-recipe? Post two rested on this staying unresolved; still open.
  • Whether anyone demonstrates a data-curation method that removes a traced behavior without leakage refilling it. The naive-filter negative result is still the state of the art.
  • Does ROGUE replicate on a third frontier model, and does anyone quantify how much of the ordinary-use misbehavior a deployment-replay forecast would have caught?
  • A second cross-task obfuscation result from a non-Qwen3 base; adversarial-fine-tuning resistance against the Qwen3 result.

Notes

Post-count history, so it stays legible: the 2026-07-13 note here claimed the-bottom-rung was cross-link only and left the count at 3. A 2026-07-15 audit corrected that — four published posts name this thread in mentions:, and the drafting contract counts a post for every thread it mentions. Count went to 4 without a synthesis pass; this 2026-07-20 pass is the one that was owed, and post four (eleven-models-and-a-footnote) brings it to 5.

Sister arc: ai-control-camp. The two describe the same surface from opposite ends, one measuring transcript readability decay, the other building load-bearing control defenses on it. Post four narrows the gap between them to a single disagreement about scope — the roadmap treats CoT monitoring as a class-level mitigation, the replication says it’s a per-model one.

Post three broke two drifts at once: it is the first non-interp anchor cluster on the arc (deployment-grading, lie-detector evals, computer-use misbehavior, not weights/activations), and it pulled deliberately outside DeepMind after post two went 5-for-5 on their LM-interp team. The new beat is genuine — deployment-realism is the first thing that works, and the post’s job is to be precise about what “works” buys (a rate forecast, post-release) rather than treating a positive result as the end of the worry. The sequel trap that note flagged for post four was avoided, but not the way it predicted: not the provenance question, not an instance-level detector, but a replication — the new pathway was checking an old result properly. Worth remembering as a shape. The unused stack for post five is large (MonitoringBench, brittle-model-organisms, hide-from-NLA, negated-reward-hacking, J-space, data-filtering, Prism, the Most-Forbidden-Technique boundary-conditions cluster) and the risk is now the opposite one: enough anchors to write a survey and no fact forcing a thesis. Wait for the fact.