Host-Failure Recovery - Automatic Server Recreation

How It Works

If your server’s host stops answering for 30 minutes, your server is rebuilt on a healthy host from its most recent disaster-recovery snapshot. Snapshots are taken every 2 hours, so anything since the last one is lost. This is on for every server.

GameGrid checks every host every 5 minutes. A host is treated as failed when it has not sent a heartbeat for more than 30 minutes and it also fails a direct ping or its agent health check. A host that is still reachable is not treated as failed, and neither is a host our staff have taken out of service on purpose for maintenance.

Each server on a failed host is rebuilt on a healthy host in the same region, from that server’s own most recent disaster-recovery snapshot. A server that was running is started again. A server that was stopped is rebuilt and left stopped.

Snapshots and What Can Be Lost

Disaster-recovery snapshots are taken every 2 hours for every server on a host, running or stopped, because a stopped server still holds save files, configuration and mods. They are encrypted and kept in off-site object storage.

Anything saved after the most recent snapshot is lost when a server is rebuilt, so the loss can be up to 2 hours of play. A snapshot is a plain copy of the server’s files: it does not run the game’s save command and does not pause the server, so the last moments before a snapshot may not be in it.

If a snapshot cannot be taken (for example because the running game holds a file open), the previous snapshot stays the most recent one, and the snapshot is tried again within minutes.

This is why snapshots are not a substitute for backups. Your own backups on the Backups tab run the game’s save step first and are the ones to restore from when you want a specific point in time.

How Long It Takes

Detection takes 30 minutes of silence plus up to 5 minutes until the next check. Rebuilding then depends on the size of your server’s files and the time the game takes to start. Most servers are back within minutes of the failure being detected; large worlds or heavy mod packs take longer.

No action is required from you.

When a Server Cannot Be Rebuilt Automatically

If a server has no disaster-recovery snapshot yet (for example, it was created less than 2 hours before its host failed), it is not rebuilt automatically. It stays where it is and our staff are alerted, so they can restore it for you from one of your backups.

If no healthy host has room for the server, it stays where it is, our staff are alerted, and recovery is tried again every few minutes until there is room.

A server is never left half-moved: if any step of the rebuild does not complete, the server stays recorded on its original host and the rebuild is tried again.

After Your Server Has Been Rebuilt

We email your account’s contact with the server’s new address and the time of the snapshot it was rebuilt from. If you use webhooks, a server.migrated event is sent as well.

Your server keeps its name and its place in your bundle. It runs on a different machine, and its port can change too, so the address your players use changes. Check the Server Address row on the Overview tab and hand out the new one.

When the failed host comes back, the old copy of your server on it is stopped and kept from starting, so two copies of your world never run at the same time.

If you want a smaller loss window than 2 hours, take your own backup before anything you would not want to repeat. See How Backups Work.

Turning It Off

Recovery is on for every server and there is no switch for it in the panel. To turn it off for a server, open a support ticket and our staff will switch it off for that server. Open another ticket to turn it back on.

The Administration tab describes this behaviour under "Automatic Host-Failure Recovery".

Most customers should leave recovery on: it is the fastest way back online after a host failure.