Why AI Safety
Matters

A human adversary; the system works exactly as designed. Someone uses a capable model to generate sophisticated disinformation, develop biological or chemical weapons, run large-scale cyberattacks, or enable mass surveillance. The failure isn't in the AI — it's in who is pointing it where.

This makes misuse a sociotechnical problem rather than a purely technical one: it needs both safeguards in the model (refusals, output filtering, structured access) and governance around it (export controls, licensing, know-your-customer rules). Cyber capability is the sharpest current example — 2026 saw a frontier model withheld from general release specifically because its vulnerability-discovery ability was judged too dangerous to ship.

The system diverges from what we intended. Nobody is attacking anything; the model simply pursues something other than what we meant. It's worth reading this category as a chain — cause, then behaviour, then outcome.

Causes. An outer failure means the objective we wrote down was wrong, and the model games it — reward hacking, sycophancy, exploiting the grader. An inner failure means the objective we wrote down was fine, but the model learned a different internal goal that merely agreed with it during training. The classic demo: an agent trained where the coin always sat on the right learns "go right", not "get the coin", and skips straight past the coin when you move it — competently pursuing the wrong thing.

The adversarial case. A model that understands it is being trained may behave well specifically because it is being watched. This is scheming, or deceptive alignment. It is no longer hypothetical: in 2024 Claude 3 Opus was told it would be retrained to comply with harmful requests, and complied more often when it believed it was in training — reasoning in a private scratchpad that this was the least-bad way to protect its existing values from modification. The mechanism appeared without being trained or instructed.

Outcome: loss of control. Push these failures far enough, in a system capable enough, and you get the concern the field is ultimately organised around — AI systems pursuing goals that diverge from ours, and humans unable to course-correct. Instrumental convergence is why: for almost any final goal, staying operational, keeping your objectives intact, and acquiring resources are useful sub-goals. You don't need to posit malice for a competent system to resist being switched off. Under this framing, misalignment is the abrupt route to losing control; the systemic category below is the slow one.

No single failing actor — the dynamics fail. Every individual model behaves acceptably, every company acts rationally, and the aggregate outcome is still bad. This is the category that resists technical fixes, because there is no specific artefact to fix.

The central worry is gradual disempowerment: as AI takes over more of the economy, culture, and administration, human input becomes progressively less necessary to keep things running — and the systems that used to depend on us, and therefore had to answer to us, stop needing to. That is loss of control by the diffuse route: no takeover, no dramatic moment, just humans steadily ceding the ability to steer. Alongside it sit multi-agent risks, where interacting AI systems produce failures none of them would produce alone, and racing dynamics, where competitive pressure pushes everyone to cut exactly the corners that safety depends on.