wiki.flightgear.org scheduled downtime

[curt]

The FlightGear wiki: http://wiki.flightgear.org will be offline for about 1 hour starting at 19:00 UTC (today, Wed 21 Jan) so our ISP can migrate the server to new hardware. Thanks for remaining calm and staying shiny.

Curt.

[Johan G]

Thanks for the heads up.

Edit: It is a bit close, but I think I will add a site notice in case someone is editing etc.

Edit2: Done, using Template:Due date to hide the notice after 20:00 UTC

[www2]

HI it's now 22:13UTC and the wiki is not yet up.

[curt]

They indicated the maintenance is progressing but taking longer than anticipated. That was a couple hours ago so I am trying to contact them for a status check to make sure everything is still progressing. Sorry for the longer than expected outage.

[curt]

Now the issue (whatever it is, not explained to me) has been escalated ... please stand by.

[www2]

Any updates for now?

[chriscalef]

I always cringe whenever anybody says something will be done in an hour. Been there so many times.

Thanks for all your hard work!

[Johan G]

Still down 19:00 UTC the following day.

Good thing this is not a business; this could be worse.

[www2]

So far as i know (only from the user site) the database is down and the file system is Read-only.
Edit the wiki serve is current offline.

[curt]

This has turned out to be a degenerate mess, sorry everyone! The ISP claimed it would be a quick hour off line and back to everything being good. They just emailed me again so know I'm going to run off and try to address another issue they claim to have discovered and we'll go from there. Again, sorry, I was promised a quick hour offline and it would be done.

[Hooray]

I just hope that someone managed to create backups before the whole thing went haywire - the wiki has easily become one of the most useful and most popular web based FlightGear related resources meanwhile - so any kind of data loss would be a pretty serious issues (think avsim debacle a few years ago)

[curt]

Here is a quick saturday morning update. Simon and I are still working on this. Simon has verified that the database is intact so that is good news. We have multiple offline copies of the database.

The area of concern is that the server file system got moderately scrambled (I didn't think this sort of failure scenario was possible on a virtual machine when you don't have to worry about physical disk errors, but apparently it can still happen ...) Running "fsck" (similar to dos chkdisk) is the only option, and that found a ton of problems and had to do a lot of things to bring the file system back into consistency. If anyone cares about the gory details, quite a few files were dumped into /lost+found and given #inode_number names.

The other tricky thing is that when the volume got corrupted enough system files were affected so that the machine can no longer successfully boot, so it looks like we'll end up wiping the machine and reloading everything from scratch.

So what we are looking at is:

1. reload the server operating system from scratch
2. restore the needed software (apache, mysql, mediawiki, etc.)
3. do various server and software setup/config steps.
4. restore the mysql database for the wiki (which is intact) ... this gives us all the wiki text content.
5. restore the tree of attachment files (figures and pictures and other media the wiki references.) This tree does have some loss. Most of the lost files are in lost+found, but may now have non-descript file names or directory names.
6. We will then try to work through the lost+found to recover as much of the missing pictures and figures as we can, and some of this may need to be on a case-by-case basis as we discover broken image links. In some cases the whole directory was moved and the file names are intact so when we do see a busted link, we may be able to quickly find the file, and move that whole directory back where it should go.
7. Very recently (like in the last couple weeks) I have increased the space/capacity of my personal backup system, so I will investigate automated backups of the wiki (and the forum).

Yep, and many of us have worked as sys admins in our careers so we know all the comments ahead of time, no need to be repetitive. It's always an issue of available time and energy, less an issue with intent or knowledge.

[patstan]

Do you not have backups of the media content which includes things like images?

[elgaton]

Should you need help recovering the images (e.g. because the lost+found directory does not have all of them), an option might be getting them from our browser caches - see this Pro Webmasters question for a description of the approach.

[Johan G]

@ curt: Thanks for the update on what is going on.

Simon has verified that the database is intact so that is good news. We have multiple offline copies of the database.

Those are both good news. (Lets just hope that the backups are recent. )

6. We will then try to work through the lost+found to recover as much of the missing pictures and figures as we can, and some of this may need to be on a case-by-case basis as we discover broken image links. In some cases the whole directory was moved and the file names are intact so when we do see a busted link, we may be able to quickly find the file, and move that whole directory back where it should go.

I do not really understand what that will look like in the wiki, but if I got it correct they would show up as broken wiki links.

I suspect that these links will most probably show up in the Special:WantedFiles list, as pages will link to them.