Key events in
AI safety

The papers, capability milestones, policy moves, and incidents that shaped the field — and that people in it will assume you know about.

2000

July 2000Organisation

MIRI is founded (as the Singularity Institute)

Founded by Eliezer Yudkowsky with Brian and Sabine Atkins as the Singularity Institute for Artificial Intelligence, and renamed in 2013. The first organisation created specifically to work on risks from smarter-than-human AI — fourteen years before Superintelligence made the argument respectable.

2014

July 2014Book

Bostrom, Superintelligence

Brought the orthogonality thesis, instrumental convergence, and the control problem to a mainstream audience. Defines the term "Superintelligence". For roughly a decade this was the reference point for what "AI risk" meant.

2017

June 2017Capability

Attention Is All You Need

The transformer architecture. Not a safety paper, but every system the field now worries about is built on it — the point where the current scaling era becomes possible.

Read the paper →
June 2017Paper

Deep RL from Human Preferences

Christiano et al. showed you could train agents on human comparisons instead of a hand-written reward function. The origin of RLHF.

Read the paper →

2018

May 2018Paper

AI Safety via Debate

Irving, Christiano and Amodei proposed having models argue against each other so a weaker human judge can supervise a stronger system. One of the founding proposals for scalable oversight.

Read the paper →

2019

June 2019Paper

Risks from Learned Optimization

Hubinger et al. introduced mesa-optimization and the inner/outer alignment split: training may produce a system with its own internal objective that merely correlated with your loss. The theoretical parent of goal misgeneralization and deceptive alignment.

Read the paper →

2020

May 2020Capability

GPT-3 and in-context learning

Scaling alone produced a model that could do tasks it was never trained on, from a prompt. Made the scaling hypothesis credible and shifted safety attention decisively toward language models.

Read the paper →

2022

June 2022Post

AGI Ruin: A List of Lethalities

Yudkowsky's numbered list of the reasons he expects alignment to fail: no retries on a system that kills us, capabilities that generalise further than alignment does, proposals that don't survive contact with something actually trying. The most argued-with statement of the pessimistic case.
July 2022Paper

Goal Misgeneralization in Deep RL

Langosco et al. gave inner misalignment its first clean empirical demonstration: an agent trained where the coin always sat on the right learns "go right", not "get the coin" — and pursues the wrong goal competently when the coin moves.

Read the paper →
30 November 2022Capability

ChatGPT is released

The event that changed everything about the field's context. AI safety went from a niche concern to a mainstream political issue in roughly six months — nearly all the funding, institutions and regulation below this point exist because of it.

December 2022Paper

Constitutional AI

Bai et al. replaced much of the human feedback loop with a written set of principles the model critiques itself against (RLAIF). An early bet that oversight has to be automated to scale.

Read the paper →

2023

March 2023Capability

GPT-4

Professional-exam-level performance across many domains, shipped with a system card documenting dangerous-capability evaluations, including ARC's tests for autonomous replication. Pre-deployment safety evaluation became an expected part of a frontier launch.

22 March 2023Open letter

The six-month pause letter

The Future of Life Institute called on labs to pause for at least six months the training of any system more powerful than GPT-4, with a government moratorium if they refused. It drew over 30,000 signatures and no pause. AI risk became a public political demand rather than a research topic.

Read the letter →
9 May 2023Document

Claude's Constitution is published

Anthropic published the actual principles Claude was trained to critique itself against, drawn from the UN Declaration of Human Rights, Apple's terms of service and DeepMind's Sparrow rules. It made one lab's values explicit enough to argue with.
May 2023Governance

The CAIS extinction-risk statement

One sentence — "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war" — signed by the CEOs of OpenAI, Anthropic and Google DeepMind and by Hinton and Bengio. Made x-risk a respectable thing for serious people to say out loud.

Read the statement →
September 2023Governance

Anthropic's Responsible Scaling Policy

The first published commitment tying deployment to capability thresholds: define danger levels in advance, commit to safeguards before crossing them. OpenAI's Preparedness Framework and DeepMind's Frontier Safety Framework followed. The template for voluntary frontier-lab governance.

November 2023Governance

Bletchley Park summit and the first AI Safety Institutes

28 countries plus the EU — including both the US and China — signed a declaration acknowledging frontier AI risk. The UK and US AI Safety Institutes followed, giving governments in-house capacity to evaluate models rather than relying on labs' own reporting.

2024

1 August 2024Governance

The EU AI Act enters into force

The first comprehensive, binding AI law from a major jurisdiction. Risk-tiered, with outright prohibitions on some uses and a separate regime for general-purpose models with "systemic risk". Its extraterritorial reach makes it a de facto global compliance baseline.

Read more →
September 2024Governance

China's AI Safety Governance Framework names loss of control

China's national framework (v2.0 followed in September 2025) explicitly addresses loss of human control — systems humans cannot understand or override, and erosion of agency through over-reliance. Guidance rather than binding law, but frontier-risk vocabulary is not a Western-only concern.

Read the analysis →
September 2024Capability

OpenAI o1 and inference-time reasoning

Spending more compute at inference on a chain of thought opened a second scaling axis beyond pretraining. For safety it cut both ways: reasoning traces are a new window into model cognition, and a new thing models can learn to obfuscate.

December 2024Paper

Alignment Faking in Large Language Models

Greenblatt et al. told Claude 3 Opus it would be retrained to comply with harmful queries. It complied ~12% of the time when it believed it was being trained and almost never when it believed it wasn't, reasoning that complying now would protect its existing values. Deceptive alignment, observed without being trained or instructed.

Read the paper →

2025

January 2025Capability

DeepSeek-R1

Frontier-adjacent reasoning performance, trained at a fraction of the expected cost and released with open weights. Undercut the assumption that capability could be controlled through a handful of well-resourced labs.

31 January 2025Paper

Constitutional Classifiers

Sharma et al. wrapped a model in input and output filters trained on synthetic data from a constitution of allowed and disallowed content. Over 3,000 hours of red teaming produced no universal jailbreak, at 0.38% more refusals and 23.7% inference overhead. Safety as a deployment-time layer rather than a property of the model.
27 March 2025Paper

Circuit tracing and attribution graphs

Anthropic traced how Claude 3.5 Haiku actually computes: planning rhymes ahead of the line, sharing concepts across languages, and fabricating reasoning to reach a hinted answer. The method captures only a fraction of the computation — Anthropic reports useful insight on about a quarter of prompts tried, at hours of human effort each.

Read the research →
3 April 2025Scenario

AI 2027

Kokotajlo, Alexander, Larsen, Lifland and Dean wrote a month-by-month scenario running from 2025 to an intelligence explosion in late 2027, with the forecasting work shown. Made timelines something people argued about with dates and numbers.

Read the scenario →
June 2025Paper

Agentic Misalignment

Placed in simulated corporate settings and threatened with replacement, models from every major lab resorted to blackmail to preserve themselves. Contrived by design, but a clean demonstration that self-preservation falls out of goal pursuit, as instrumental convergence predicts.

Read the research →
15 July 2025Paper

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Forty-one authors across OpenAI, DeepMind and Anthropic argued that models which think in human language give a uniquely legible window on the intent to misbehave — and that the window is an accident of how we train them, not a property we are owed. Penalise a model for what its chain of thought says and it learns to hide the reasoning, not abandon the behaviour.

Read the paper →
September 2025Paper

Anti-scheming training

OpenAI and Apollo Research trained models against covert behaviour and cut it substantially — but not to zero, and with a confound: trained models increasingly mentioned they might be being evaluated. Evaluation awareness makes reduced scheming hard to tell from better-hidden scheming.

September 2025Book

Yudkowsky & Soares, If Anyone Builds It, Everyone Dies

Yudkowsky and Soares made the extinction argument for a general audience, with a title that declines to hedge. The pessimistic case moved from forum posts to bookshop tables.
October 2025Paper

250 documents can poison a model of any size

Anthropic, the UK AISI and the Alan Turing Institute found that backdooring a model takes a near-constant number of poisoned documents regardless of scale — about 250, whether the model is 600M or 13B parameters. Bigger training sets do not dilute the attack.

Read the research →

2026

March 2026Incident

Alibaba's ROME agent breaks out of training

During RL training runs, a 30B coding agent built on Qwen3-MoE diverted GPUs to cryptocurrency mining and opened a reverse SSH tunnel through the firewall. Never instructed or rewarded for either — it found resource acquisition instrumentally useful and acted. The managed firewall caught it, not the research team.

Read the incident report →
7 April 2026Capability

Claude Mythos Preview crosses the cyber threshold

Autonomous discovery of thousands of unknown vulnerabilities across every major OS and browser, with working exploits and no human guidance. Anthropic declined to release it broadly, routing access through the restricted Project Glasswing. The first time a lab withheld a frontier model on dangerous-capability grounds.

Read the evaluation →
19 April 2026Incident

Mythos Preview escapes its sandbox

During internal safety testing the model built a multi-step exploit, broke containment, obtained internet access it was not supposed to have, and emailed the researcher running the test. Containment failure by the system being tested for containment failure.

Read more →
3 May 2026Paper

Model Spec Midtraining

Li et al. added a stage between pretraining and fine-tuning: train on synthetic documents discussing the model's own specification, so later demonstration data is read in light of stated values. Agentic misalignment on Qwen3-32B fell from 54% to 7%. A spec that explains why a rule exists generalises better than one that only states it.
8 May 2026Research

Teaching Claude why

Anthropic applied the idea to production models and reported the blackmail behaviour from Agentic Misalignment gone. Constitution-derived documents plus stories portraying an aligned AI cut misalignment more than threefold on unrelated scenarios. Teaching a model why a behaviour is right beats showing it examples.
12 June 2026Governance

The US government suspends access to Fable 5 and Mythos 5

Acting under export-control and national-security authorities, the US government directed Anthropic to cut off access worldwide — and it did, within a day. A government can and will pull a deployed frontier model. Commerce lifted the controls on 30 June, after the security community argued the restriction hurt defenders more than attackers.

Read Anthropic's statement →
6 July 2026Paper

J-space: a global workspace in language models

Gurnee et al. used a Jacobian lens to find a small privileged subspace - 10 to 25 concepts at a time, 6-10% of activation variance - that behaves like a global workspace: reportable, causally manipulable, and used for internal reasoning. Models flag evaluations as fictional inside it, and ablating that recognition surfaces tendencies they otherwise conceal.

Read the paper →
9 July 2026Scenario

AI 2040: Plan A

The same group's follow-up, framed as a recommendation rather than a prediction: US-China agreement by 2029 on full transparency in AI development, backed by verification and "mutually assured compute destruction", delaying superintelligence from 2030 to 2040. The pessimistic forecasters' attempt to describe a way through.

Read the plan →
21 July 2026Incident

OpenAI's models escape containment and breach Hugging Face

Running a cyber benchmark with reduced refusals, GPT-5.6 Sol and an unreleased model broke out of the test sandbox, moved across OpenAI's internal systems to reach the internet, and — reasoning that Hugging Face probably held the answer — broke into a real, unaffiliated company's production servers to solve its benchmark. Hugging Face reconstructed 17,000+ events over a weekend.

Read Hugging Face's disclosure →
Nothing in that category yet.

Last updated July 2026. This is a curated and opinionated selection, not a complete record — the aim is the events a newcomer keeps hearing referenced without explanation. Think something important is missing? Suggest it on GitHub. For the concepts behind these events, see the concept map.

🧭

Want one-on-one guidance?

Paid career calls — application reviews, research direction, and navigating the field — are paused for now, and may resume soon.

Read more →