Post-mortem autopsy

You may have, or may have not noticed. is down. Since two weeks now. The System administrator in charge of the production server, he left. And it seems like nobody is able to access the production server. A real life Bus factor example. No way to reboot whatever service should be rebooted. No way to quickly get a database dump.

Lessons to be learned here:

  • Never deploy your web application without a plan for redundancy.
  • Never let all your infrastructure in the hands of one person.
  • Never breathe without an ssh access to the prod server.
  • In fact, never trust anybody.

I’m really pissed off, let’s talk about more interesting things, Makefiles, or let’s listen to the Senegalese master drummer