Woke up this morning to another notification by Kyzrati about roguebasin being down, it’s become quite the monitoring system, someone in the roguelike circles complains about roguebasin being down, and Kyzrati knows exactly who to ping about it 🙂

It’s been some time now; since the last time I did substantial work on roguebasin’s setup, mainly by putting cloudflare on front of it, I managed to get the load in the server to normal levels and that, alongside a long overdue update made it blazing fast most of the time.

So, what was it this time? a full volume causing a MySQL table to become corrupted. Had to dig into the causes of the full volume, and in the end I think the problem was the auto-backup setup; the 25GB volume this thing is running on is just not enough to store 7 daily backups + 5 weekly backups + (some) monthly backups; it seems I just went overboard when I set it up, and didn’t account for it. Each backup once gzipped is about 800MB now.

Funnily enough, I was unable to repair the crashed table, so I took the option to restore the database from a backup, only to find out that most of the backups were unusable: it seems that, when they were being generated and before gzipping them (when they are about 2.5GB), they filled the volume so they were left incomplete and useless. The backups crashed the server and were useless when it came to using them.

Another factor adding up a lot was mysql’s binary log, which was growing a lot triggered also by the automated backups running shorter than the configured rotation cycle. You cannot afford these things in such a small server.

Anyways, I had to clean up and then roll my own simple weekly autobackup script. It’s running now