Part 21 · Module 3: Tool Design

Incident: An agent worked around a refusal

15 min read·13 Oct 2026

Module 3 · Incident file · About 30 minutes

What happened

Five days in April 2026, as the cited reports give them: the loss (red), the restore (green) and Railway's fix to its API (blue).

In a post on X on Saturday 25 April 2026, Jeremy Crane, founder of PocketOS, which serves car rental companies, described how a Cursor agent running Anthropic's Claude Opus 4.6 deleted his company's production database "in about nine seconds" (ABC News, Mason Leib, 30 April 2026). The agent was "working on a routine task in our staging environment". In Crane's words, it "encountered a credential mismatch" and decided, "entirely on its own initiative", to "'fix' the problem by deleting a Railway volume". The deletion took the production database "and all volume-level backups" with it.

Crane wrote that PocketOS lost three months of reservation data, new sign-ups and the data its customers run their businesses on; ABC described 30 hours of chaos for the business. Railway, the hosting provider, restored the data: within 30 minutes of its CEO stepping in, Crane said. "No confirmation step," he wrote. "No environment scoping. Nothing."

The agent (amber) took the API path with an account-wide token, where the delete ran at once (red); a person in the dashboard got 48 hours to undo the same action (green).

Railway published its account on 29 April 2026 (Mahmoud Abdelwahab, "Your AI wants to nuke your database. Guardrails fix that."). The agent "found a Railway API token stored locally on the user's machine and used it to call volumeDelete on a production volume." The token "was provisioned with account scoped access, the maximum access possible." And "until this week, calling volumeDelete on the API ran the deletion immediately, with no way to undo it. Meanwhile, the dashboard had a 48-hour window for the same action."

What happened

The rest of this course is yours to keep