Ubuntu
Rash decisions always feel like sure bets before you act on them: I recently went “you know what, upgrading this server’s Ubuntu LTS won’t be a big deal — I’ve done all these package updates for the past few years with nary a hiccup… I think I’ll just purge my PPA repo (since ffmpeg is officially back in 16.04.1), open a screen session, and get going! And, for a multitasking win, I’ll do it in between seeing patients!”
That did… not go well.
Of COURSE I ran into a package bug that caused MySQL to fall over during the upgrade. Normally, MySQL dying wouldn’t be an issue, since my site is served from either an APCu or memcache cache, but HEY what’s this, PHP moved over to 7.0 and none of the extensions it wanted had been installed yet. So, the entire site was down.
Suspecting anything and everything was borked up, I did some diagnostics (aka panicked which commands to see what was still installed) and found it wasn’t too bad. The upgrade had completed with one exception, and had some unintended, un-researched complications. So, it wasn’t as bad as when I tried to upgrade my Joyent Accelerator years ago (Solaris… and the last time I’ll pick the obscure OS option) and was left with only the shell session I was currently running, trying to figure out how to get wget back on the machine so I could get gcc so I could START to reinstall things. Leaving my laptop powered up, connected to that one SSH session, praying the internet didn’t hiccup while I went to dental school classes, then rushing back home to debug some more—THAT was truly un-fun. So this was better than that.
But wait, the DSA (ssh-dss) keys I generated years ago were now disabled by default. Which, of course, that’s the last thing I would ever think of when trying to debug ssh problems. In this case, LISH saved me as I was able to generate new keys, add them to the server, and once again establish new SSH connections.
Then it turned out my stats program Mint was totally disabled. It was written like 12 years ago, last updated some years ago, by a great programmer who is now a busy father and has moved on to other things. It was time to dig in and fix someone else’s PHP, only to learn that MySQL’s default sql_mode is now far more restrictive than it used to be, which caused queries that used to run perfectly to fail. That took a little while to figure out.
The easiest were the fixes to my own code, because I know where (many, not all) of the gremlins live. Sometime in the middle of it all, John left a comment on this site, so that was a good sign. Luckily, it must not have required him to answer the security image challenge, because I found out that was broken, too. Fixed!
Looks like I won’t have to upgrade Ubuntu until 2021. If I do it before then, I’ll make darn sure to research it first.