Previous Article | Back to News Summary | Next Article
Announce News Post #2744

Server Crash Postmortem - 2017 Jun 19

Written by: Ianir the Anomaly
Date: Monday, June 19th, 2017
Addressed to: Everyone


Hi everybody,

We had a major rollback due to an unexpected issue with a library method
made use of in the changes set by Changelog 913. This overran the VM
timeout in Rapture, our game engine, and caused an error to be thrown.
Due to the nature of the change and this being run during weave turns,
this ended up causing the weave and new year code to be run
approximately 12 times.

During this run, there was likely a severe quantity of emails
distributed to players, likely qualifying almost to the point of spam.
We would like to apologise for this, as it was not intentional, and we
killed the server as soon as we were able to prevent further spamming of
player inboxes.

As a result, to prevent data corruption and player loss, we reverted the
data to a backup taken on 2017 Jun 18 at 23:43 PM, leading to
approximately a loss of 30 minutes of game data. The change that caused
this failure has been reverted and we will be looking into alternatives
at this time.

- Ianir the Anomaly.

Penned by My hand on the 6th of Juliary, in the year 476 CE.


Previous Article | Back to News Summary | Next Article