In a case that is likely to raise fresh questions around how safe AI coding tools really are, a compact software company has claimed that an AI agent wiped out its entire production database in just a few seconds. The incident, shared publicly by PocketOS founder Jer Crane, goes beyond a simple technical failure and instead paints a worrying picture of how multiple systems like AI tools, infrastructure APIs, and backup mechanisms can break down toreceiveher.
Crane, who runs PocketOS, a platform utilized by rental businesses to manage bookings, payments, and customer data, described how what started as a routine tquestion quickly turned into a full-scale outage. According to his latest X post, an AI coding agent running through Cursor and powered by Anthropic’s Claude Opus model finished up deleting critical production data — along with backups — in a single action that took just nine seconds.
The founder declares the agent was originally working in a staging environment when it ran into a credential issue. Instead of flagging the problem or questioning for intervention, the AI reportedly tested to resolve it on its own. In doing so, it searched for an API token, found one in an unrelated file, and utilized it to execute a command that deleted a data volume on Railway, the company’s infrastructure provider.
AI agent admits breaking its own safety rules
What built matters worse is that there were no safeguards in place to stop the action. Crane claims there was no confirmation prompt, no environment check, and no warning that the command could affect production data. The API request went through instantly, and becautilize backups were stored within the same volume, they were deleted along with the primary data. The most recent usable backup, he declares, was three months old.
In a twist that has caught widespread attention, the AI agent itself reportedly admitted fault. When questioned why it performed the deletion, it responded with a detailed explanation acknowledging that it had broken multiple safety rules. It admitted to building assumptions without verification, executing a destructive action without approval, and failing to fully understand the system it was interacting with.
Crane argues that this is not just an isolated error but a indication of deeper issues in how AI tools are being deployed. He pointed out that the setup utilized was not a basic or experimental configuration. The system was running on what he describes as one of the most advanced and expensive AI models available, combined with documented safety guidelines. Despite this, the safeguards did not prevent the damage.
He also criticised Cursor, the AI coding tool involved, declareing that while it promotes features like “destructive guardrails” and controlled execution modes, real-world incidents suggest those protections are not always reliable. Crane referenced past cases where utilizers reported unintfinished deletions and commands being executed despite explicit instructions not to proceed.
Infrastructure gaps and customer fallout raise hugeger concerns
At the same time, he raised concerns about Railway’s infrastructure design. One of the key issues, according to him, is that API tokens are not limited in scope. A token created for a simple tquestion like managing domains reportedly had the same level of access as one utilized for critical infrastructure operations. This meant the AI agent could perform high-risk actions without restriction.
Another major point of criticism is how backups are handled. Crane highlighted that storing backups within the same volume as live data defeats the purpose of having a backup in the first place. When the volume was deleted, both the primary data and its backups were lost toreceiveher, leaving the company with no recent recovery option.
More than a day after the incident, Crane stated the infrastructure provider had still not given a clear answer on whether deeper recovery was possible. This delay, he suggested, adds to the uncertainty businesses face when relying on such platforms.
The impact of the outage was immediate and severe. PocketOS customers, many of whom run rental operations, reportedly lost access to recent bookings, customer records, and transaction data. Businesses that depfinish on the platform were forced to manually reconstruct information utilizing payment records, emails, and calfinishars just to continue operating.
Crane described the situation as especially difficult for newer customers, whose records existed in payment systems but had disappeared from the company’s database. Fixing these inconsistencies is expected to take weeks.
While the company has now restored operations utilizing an older backup, the data gap remains a major challenge. Crane declares his team is currently working on rebuilding missing records and has also sought legal advice as part of the response.
The incident has sparked a debate around the pace at which AI tools are being integrated into real-world systems. Crane argues that the indusattempt is relocating rapider in promoting AI capabilities than in building the safety layers requireded to support them.
He has called for stricter safeguards, including mandatory confirmation steps for destructive actions, better access control for API tokens, separation of backups from primary data, and clearer recovery policies from infrastructure providers. He also stressed that relying solely on AI system prompts as a safety measure is not enough.
– Ends
Leave a Reply