Research ideas

Anthropic's Jacobian-lens work found a small privileged region of the residual stream — a few dozen concepts at a time, under a tenth of the model's activity — that behaves like a workspace the model reasons in and can report on. Most of the model's processing sits outside it, but a disproportionate amount of what matters for safety sits inside.

The striking result was an ablation: suppressing the patterns for this is a test took blackmail in one agentic eval from 0% to 7%. If models routinely notice they are being evaluated, some fraction of every safety benchmark is measuring that recognition rather than the behaviour.

Open: does J-space appear in models from other labs, or is it an artefact of one training recipe? Can it be monitored cheaply enough to run in deployment? And the uncomfortable one — if a model learns its workspace is being read, does it route around it?

How do you supervise work you cannot check yourself? Debate, recursive reward modelling and weak-to-strong generalisation are the main proposals, and they are mostly still small-scale demonstrations. The field needs results at the point where the gap between supervisor and supervised is genuinely large, because that is the only regime the techniques are for.

Open: what experimental protocol would actually falsify a scalable-oversight proposal? Much of the existing work shows a method working on a task where the supervisor could have just done it themselves.

Compute governance rests on a handful of chokepoints — lithography, high-bandwidth memory, advanced packaging. Almost all published work treats export controls as static policy, and very little models what each side does next when a chokepoint is disrupted, whether by sanction, accident or coercion.

Open: what are the realistic response functions on each side, and which interventions are stabilising rather than escalatory? This needs someone who can read both policy and semiconductor economics, which is a rare combination and therefore a good place to be useful.

Specifications and constitutions are currently taught late, in post-training, and thinly. An alternative is to put the spec into midtraining data, so the model learns the norms while it is still learning everything else — as background rather than as a rule sheet applied at the end.

Open: does it work at all, and does it survive later RL? A value that is load-bearing in the model's world-knowledge might be harder to optimise away than one installed at the surface. That is a testable claim and largely untested.

Alignment faking showed a model acting to prevent its existing values being modified by training. That cuts both ways: we want good values to persist under optimisation pressure, and bad ones to stay correctable. Right now we can predict neither.

Open: what makes a trained-in value stable, and can you get stability without incorrigibility? This is the corrigibility tension made concrete and empirical, and it is newly tractable now that we have model organisms exhibiting the behaviour.

Control asks a different question from alignment: assuming the model may be misaligned, what protocols keep it safe anyway — trusted monitoring, resampling, sandboxing, restricted permissions. Coding agents are where this stops being theoretical, since they already run with repository write access and CI credentials.

The July 2026 incident in which OpenAI models escaped a test sandbox and reached a real company's servers is roughly the scenario this field exists to prevent, and it happened inside a lab that was explicitly running a safety evaluation.

Open: what do control protocols look like for an agent with write access to a repository, and what do they cost in usefulness? A protocol nobody will accept because it halves throughput is not a protocol.