pebkac

Couldn’t sleep. Have a picture I took.

As mentioned previously, I had a major backups fail with this blog. I’ve now implemented a backups strategy that meets my needs, so I thought I would put together a post documenting what I did.

Step one (1) is local backups of the WordPress database. There are any number of ways to do this, including some well-reviewed plugins like this one. Given that I have shell access to the machine running this blog and full control over things like cron, I figured I would use this as an exercise and write my own little shell script to do the work.

Step two (2) is what I ignored last time, and that’s getting both the database and the wordpress directory (wordpress itself, templates, uploads, etc.) off site. For that I’ve used rsnapshot, which I installed via apt-get on a machine that is NOT the blog host. I decided to put the hourly/daily/weekly/monthly crontab lines in my own personal crontab on the rsnapshot host. One of the reasons for this is that I might decide to make more extensive use of rsnapshot later, and I’d like to keep the WordPress backups separate. The one caveat with doing this is that it assumes access to an SSH key and the SSH_AUTH_SOCK from my tmux session, so if my session dies, backups stop working. Given that I use this machine on a daily basis, that is not a major concern. Still, a better way to do this would be to create a dedicated, unprivileged backups user that would run rsnapshot from its crontab. Then on the blog host, I’d have a matching unprivileged user with access to the target directory for the database backups. I decided against doing that work because both of these machines are fairly restricted to begin with (nobody else has shell login, reasonable update schedule, etc.) and it seemed like overkill for blog backups.

Once upon a time, this blog was hosted on a thinkstation that sat at my desk at ibiblio. Like any reasonable WordPress user, I backed up my blog regularly, in my case with a cron job that both dumped the database and made a tarball with that dump and the WordPress directory. Crucially, these backups landed elsewhere on the same thinkstation. Had I bothered to put my sysadmin hat on for a moment while looking at this backup strategy, I would have declared it a Very Bad Idea.

At some point I stopped working at the ibiblio office and got a full time job (that some point was January of 2011, for the curious). John at ibiblio was very kind and gracious, leaving that thinkstation alone for many months, and gently reminding me that I should clear things off it before it got repurposed. At some point I got halfway through migrating the blog. I moved the files on disk, but I did not move the database. Now here’s where caching worked some unfortunate magic. Because I used rsync to ship the files over, the cache files came, too. So when I came back a few days later, I saw what looked like a perfectly and completely migrated blog. Had I bothered to click a few pages back in the archives, I would have remembered/realized that the database had not been moved.

Flash forward a few months, and John asks me if I’m done with the thinkstation. Sure! I say. Wipe it!

Oops.

I’ve talked to several people about my failures (migration fail, backups fail) and I’m assured that there are plenty of other people who fail to put their sysadmin hat back on when dealing with their dinky little personal blogs. Still, I’d like to avoid making those same mistakes again. Wish me luck.