Service Outage

At approximately 10:40 AM EST on 7/29, one of our server admins was making WorldEdit changes around spawn. Unfortunately, due to operator error, a larger than expected selection was made during the WorldEdit session that went unnoticed, which included a large portion of the server spawn.  After attempting to change the larger than expected number of blocks, the server became unresponsive and crashed.  The server was quickly restarted, however, due to the crash, the change was not recorded in the admin logs and was not able to rollback the edit.

In the best interests of the players, the Minecraft admin team the decision to put the server into whitelist only mode to prevent other players from logging in and notified me we would need to restore the world map from backups.  Coincidentally, around this time, the Mojang authentication servers were offline, however, this was unrelated to the incident.  Around 4:50 PM EST, the server map was successfully recovered from backups generated at 10:00 AM.  As a result, only about 40 minutes worth of playtime was rolled back, and the total server outage lasted for approximately 6 hours.

In order to prevent these issues in the future, we are taking steps to prevent this kind of incident from happening again in the future.  Our admin team has taken the time to ensure they have WorldEdit CUI client mods installed to assist when using WorldEdit.  Additionally, we have implemented a server-side restriction limiting the number of WorldEdit blocks that may be modified at a time.

Thank you for your patience during this outage, and we hope the steps we're taking will prevent these events from happening again in the future.

Please feel to discuss these events on our forums.

Subscribe to RSS - outage