One of the oldest thought experiments in artificial intelligence (AI) safety imagines a powerful AI system with a simple instruction to make paper clips. The system pursues that goal with total devotion. It converts the world’s resources into paper clip factories and resists every effort to shut it down or change its goals. For years, many observers dismissed the scenario as fanciful. A truly intelligent system, the argument went, would weigh paper clips against other moral considerations or at least ask its creators whether they really wanted the world razed for office supplies.