The Night One Command Deleted Production 😳

In January 2017, a GitLab engineer ran a delete command on the wrong database server. About 300 GB was gone in seconds, and the backups couldn't save it.

What saved it A copy an engineer had made by hand about six hours earlier.

Dramatised true story. Characters are fictional stand-ins. Source: about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/

Narration: Kokoro TTS (Apache-2.0). 3D animation, music sound: original.

Have you ever tested a full restore 👇