Here is a quick saturday morning update. Simon and I are still working on this. Simon has verified that the database is intact so that is good news. We have multiple offline copies of the database.
The area of concern is that the server file system got moderately scrambled (I didn't think this sort of failure scenario was possible on a virtual machine when you don't have to worry about physical disk errors, but apparently it can still happen ...) Running "fsck" (similar to dos chkdisk) is the only option, and that found a ton of problems and had to do a lot of things to bring the file system back into consistency. If anyone cares about the gory details, quite a few files were dumped into /lost+found and given #inode_number names.
The other tricky thing is that when the volume got corrupted enough system files were affected so that the machine can no longer successfully boot, so it looks like we'll end up wiping the machine and reloading everything from scratch.
So what we are looking at is:
1. reload the server operating system from scratch
2. restore the needed software (apache, mysql, mediawiki, etc.)
3. do various server and software setup/config steps.
4. restore the mysql database for the wiki (which is intact) ... this gives us all the wiki text content.
5. restore the tree of attachment files (figures and pictures and other media the wiki references.) This tree does have some loss. Most of the lost files are in lost+found, but may now have non-descript file names or directory names.
6. We will then try to work through the lost+found to recover as much of the missing pictures and figures as we can, and some of this may need to be on a case-by-case basis as we discover broken image links. In some cases the whole directory was moved and the file names are intact so when we do see a busted link, we may be able to quickly find the file, and move that whole directory back where it should go.
7. Very recently (like in the last couple weeks) I have increased the space/capacity of my personal backup system, so I will investigate automated backups of the wiki (and the forum).
Yep, and many of us have worked as sys admins in our careers so we know all the comments ahead of time, no need to be repetitive. It's always an issue of available time and energy, less an issue with intent or knowledge.
