I’m not sure how, but performing some fairly standard WordPress operations ended up effectively destroying several tables on the main blog which runs the front page operations. I’m restoring them from backups, though AWS’ backups strategy involves creating a whole new DB instance so it’ll take a little longer than I’d like. Luckily, all the individual blogs are fine while the front page is a mess.
*Update* All clear. Front page’s mangled tables restored from backup. Still can’t find the exact reason they corrupted to begin with, but nightly backups are working just fine, so that’s good at least.
*Update 2* Wonderful, there were side effects that deleted many images from the S3 storage. We know what happened, and how to prevent it from happening again, but in the meantime, fixing this particular mess involves re-uploading the now-missing images. Everyone affected knows about it and will be fixing as soon as they can. Shaking out the bugs, just shaking out the bugs…