Host-Failure Recovery - Automatic Server Recreation

How It Works

If your server’s host stops answering for 30 minutes, your server is rebuilt on a healthy host from its most recent disaster-recovery snapshot. Snapshots are taken every 2 hours, so anything since the last one is lost. This is on for every server.

GameGrid checks every host every 5 minutes. A host is treated as failed when it has not sent a heartbeat for more than 30 minutes and its agent also fails a direct health check. A host that only fails a ping is not treated as failed. A host that is still reachable is not treated as failed, and neither is a host our staff have taken out of service on purpose for maintenance.

Before a server on a failed host is rebuilt, GameGrid asks the server itself whether it is still answering players, the way a server browser does, several times over a few seconds. If it answers, the host has only lost touch with GameGrid and your players are still on it, so the server is not rebuilt: it keeps running where it is, our staff are alerted, and GameGrid asks again every few minutes. Only a server that no longer answers is rebuilt.

Each server on a failed host is rebuilt on a healthy host in the same region, from that server’s own most recent disaster-recovery snapshot. If one of your own backups is newer than that snapshot, the backup is used instead, so you lose less. A server that was running is started again. A server that was stopped is rebuilt and left stopped.

Snapshots and What Can Be Lost

Disaster-recovery snapshots are taken every 2 hours for every server on a host, running or stopped, because a stopped server still holds save files, configuration and mods. They are encrypted and kept in off-site object storage.

Anything saved after the most recent snapshot is lost when a server is rebuilt, so the loss can be up to 2 hours of play. A snapshot is a plain copy of the server’s files: it does not run the game’s save command and does not pause the server, so the last moments before a snapshot may not be in it.

If a snapshot cannot be taken (for example because the running game holds a file open), the previous snapshot stays the most recent one, and the snapshot is tried again within minutes.

The host keeps your server’s recent snapshots, not only the newest. If the newest one cannot be opened when your server is rebuilt, GameGrid tries the one before it, and then one more before that. Each of them has to pass the same rules as the newest (the wipe rule and the one-day rule below), and each is further back in time, so more play is lost. If none of them opens, a recovery that our staff start uses your newest backup instead, and an automatic recovery does not rebuild the server: it stays where it is and our staff are alerted.

This is why snapshots are not a substitute for backups. Your own backups on the Backups tab run the game’s save step first and are the ones to restore from when you want a specific point in time.

How Long It Takes

Detection takes 30 minutes of silence plus up to 5 minutes until the next check. Rebuilding then depends on the size of your server’s files and the time the game takes to start. Most servers are back within minutes of the failure being detected; large worlds or heavy mod packs take longer.

No action is required from you.

When a Server Cannot Be Rebuilt Automatically

If a server has no disaster-recovery snapshot yet (for example, its host failed within minutes of the server being created, before its first snapshot was taken), it is rebuilt from your newest backup when that backup is recent; otherwise it is not rebuilt automatically. It stays where it is and our staff are alerted, so they can restore it for you from one of your backups.

A snapshot or backup taken before you last wiped the server, deleted its savegames, restored a backup or loaded a save from the Vault is never used automatically, because it would bring back what you removed or replaced. If the host fails before the next snapshot is taken, the server is not rebuilt automatically: it stays where it is and our staff are alerted, so they can decide with you what to restore.

The same happens when the newest snapshot or backup is more than a day older than the moment the host stopped answering, which means snapshots had been failing: GameGrid does not roll your server back that far without a person deciding.

If no healthy host has room for the server, or no healthy host can run your game, it stays where it is, our staff are alerted, and recovery is tried again every few minutes until a host can take it. If the only thing missing is the game itself, GameGrid installs your game on a healthy host first and then rebuilds your server there; a large game can add up to an hour and a half to the rebuild.

A server is never left half-moved: if any step of the rebuild does not complete, the server stays recorded on its original host and the rebuild is tried again.

After Your Server Has Been Rebuilt

We email your account’s contact with the server’s new address and the time of the snapshot or backup it was rebuilt from. If you use webhooks, a server.migrated event is sent as well.

If our mail provider rejects your contact address (it bounces, or the mail is reported as spam), GameGrid stops sending email to it, so this notice and our other emails do not reach you until the address works again. Keep your account’s email address current, and open a support ticket once it is fixed so we can turn email back on.

Your server keeps its name and its place in your bundle. It runs on a different machine, and its port can change too, so the address your players use changes. Check the Server Address row on the Overview tab and hand out the new one.

Your FTP and SFTP login is set up on the new host with the same username and password, so your file tools keep working once you point them at the new address.

When the failed host comes back, the old copy of your server on it is stopped and kept from starting. It is kept, not deleted and not merged into your rebuilt server. Open a support ticket if you need something from it.

On a DayZ server, mods you installed from the Steam Workshop are not part of the snapshot, so GameGrid installs them again on the new host before your server starts there, in the same order and with the same settings. Steam hands out each mod's current version, so a mod that was updated since you installed it comes back updated. If a mod cannot be installed again (it was removed from the Workshop, or you uploaded it by hand), your server starts without it and the Mods page marks it "Not on this host" with an "Install again" button. The same happens when our staff move your server to another host.

If you want a smaller loss window than 2 hours, take your own backup before anything you would not want to repeat. See How Backups Work.

Turning It Off

Recovery is on for every server and there is no switch for it in the panel. To turn it off for a server, open a support ticket and our staff will switch it off for that server. Open another ticket to turn it back on.

The Administration tab describes this behaviour under "Automatic Host-Failure Recovery".

Most customers should leave recovery on: it is the fastest way back online after a host failure.