TrustFall Machine release spawning failure

Summary

TrustFall opened at 19:00 UTC, but every attempt to start the machine through Release Arena failed. Approximately 37,000 attempts were made by 246 users, with none succeeding.

During the incident, around 2,400 virtual machines (VMs) remained running after their associated attempts had failed. This placed increasing pressure on shared infrastructure. VIP+ dedicated servers were affected by the resulting load and were significantly degraded, but users were still able to access TrustFall through them.

TrustFall was removed from the competitive release schedule. There was no release on Saturday, 22 August 2026. The next machine release is scheduled for Saturday, 29 August 2026, and the season schedule is moved back by two weeks.

What happened

A bug in the platform code caused both of TrustFall's flags to be classified as root flags, instead of identifying one user flag and one root flag. Because the system could not identify a user flag, every attempt to start TrustFall through Release Arena failed. Each attempt that had already been allocated a VM left it running, and further attempts allocated more. Thousands of VMs accumulated.

The affected code was introduced approximately four months earlier, as a fix to how the platform decides which of a machine's flags is the user flag and which is the root flag. That fix resolved an earlier problem where the two could be attributed the wrong way round, but it identified the root flag by looking for certain words anywhere in the flag's file path rather than in its filename. TrustFall was the first release where both flags sat inside a directory whose name contained one of those words.

For several hours, we investigated the machine, tried to remove the affected VMs, and tried to save the release. The VMs left behind by failed attempts could not be identified from the platform side, because the failure had removed the platform's own record of them, so they could not be removed individually. We also could not cancel work already in progress. This added to the delay.

To protect the rest of the platform, we stopped Release Arena activity for TrustFall three hours after it opened, and withdrew it from the competitive schedule six hours after it opened. Because VIP+ users had accessed TrustFall while free users could not, continuing the competitive release would not have been fair.

Why the impact continued to grow

The platform allocated a VM from a pool of ready machines before completing the operation that failed. When the operation was rolled back, its local record was removed, but the VM was not deleted and continued running.

The same rollback removed the information used to prevent another VM from being allocated. Each new attempt could therefore take another one. Retrying a failed operation was expected user behaviour; the platform should have handled those retries safely.

Deployments already in progress could not be cancelled, allowing abandoned work to continue consuming resources. The VMs used for Release Arena ran on the same underlying virtualisation infrastructure as some VIP+ dedicated servers. As the number of Release Arena VMs grew, they consumed shared compute and storage capacity, degrading those dedicated servers.

TrustFall itself had higher CPU and disk usage during approximately the first 10 minutes of startup. This was expected for a nested-virtualisation machine and settled during testing. It was manageable at the intended scale, but thousands of simultaneous deployments multiplied that startup load and accelerated infrastructure saturation.

The machine's resource profile amplified the incident. It did not cause the runaway deployments or the failure to delete them.

Impact on VIP+

The defect only affected the Release Arena workflow used to start machines. VIP+ dedicated servers did not encounter the same bug. Those servers were affected by the increased load on shared infrastructure, which significantly reduced their performance and reliability, and a substantial proportion of dedicated machine deployments failed. Users were nonetheless able to access TrustFall through them, while Release Arena users could not access it at all.

Because free users could not access TrustFall through Release Arena while VIP+ users could access it through dedicated servers, free users did not have an equal competitive starting point. TrustFall was therefore removed and will not return as a competitive machine.

Why our checks did not catch it sooner

Pre-release testing verified the machine's behaviour and confirmed that the Release Arena pool could be provisioned. The higher startup CPU and disk usage was observed, but it settled as expected.

Pool provisioning and starting a machine use different workflows. Our checks did not exercise the complete user operation that failed when the release opened.

During the first several hours of the incident, our investigation focused on the machine itself. Pre-release testing had shown its higher CPU usage settling after startup as expected, but in production CPU usage remained at its limit for hours.

We therefore investigated whether the machine's nested-virtualisation design was responsible. During that time, we also tried to clean up the affected VMs and determine whether the release could be saved. In-progress deployments could not be cancelled and there was no single control to remove everything at once. These efforts added to the delay, and TrustFall was ultimately withdrawn six hours after it opened.

The completed investigation showed that the sustained production CPU usage was a consequence of the unexpectedly large number of VMs running simultaneously and the resulting pressure on shared infrastructure. Pre-release checks ran at the intended scale and did not reproduce those conditions. This difference initially made the machine appear to be the cause rather than an amplifier of the platform failure.

Infrastructure monitoring detected the resulting load, but the original application error was not surfaced correctly. This initially directed the response toward infrastructure capacity rather than the failed machine-start process.

Response timeline

• 19:00 UTC: TrustFall opened and all attempts to start the machine through Release Arena began failing.
• 22:12 UTC: New Release Arena activity was stopped.
• Around 01:00 UTC: The accumulated VMs were deleted and TrustFall was withdrawn from the competitive release schedule.
• Around 04:00 UTC: Infrastructure returned to normal.
• 18 August: The underlying platform defect was confirmed.
  Stopping new activity prevented further accumulation, but it did not remove the machines already running. Infrastructure began recovering only after those machines were deleted.

What we are changing

• Correcting the flag-classification logic so directory or account names cannot cause flags to be misclassified.
• Testing the complete user workflow for starting a machine before every Release Arena opening.
• Ensuring VMs created during a failed attempt are automatically deleted.
• Changing retry protection so it cannot be removed by a failed operation.
• Adding the ability to cancel deployments already in progress.
• Splitting VM operations by region and isolating Release Arena operations from other workloads.
• Alerting directly on machine-start success rates and ensuring underlying application errors reach responders.
• Shipping an already-approved cleanup safeguard that had not previously reached production. This would not have prevented the initial failure, but it would have prevented the wider infrastructure impact.
• We have introduced a firm rule for Saturday releases: if a critical release issue cannot be identified and safely fixed within two hours, the release will be cancelled.

Further improvements

• Adding earlier validation of user and root flag information.
• Building an operator control that can stop a release and reclaim its resources in one action.
• Improving how critical reliability changes are tracked through deployment.