Board index FlightGear The FlightGear project

wiki.flightgear.org scheduled downtime

Questions about the FlightGear organisation, website, wiki etc.

wiki.flightgear.org scheduled downtime

Postby curt » Wed Jan 21, 2015 5:06 pm

The FlightGear wiki: http://wiki.flightgear.org will be offline for about 1 hour starting at 19:00 UTC (today, Wed 21 Jan) so our ISP can migrate the server to new hardware. Thanks for remaining calm and staying shiny. :-)

Curt.
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: wiki.flightgear.org scheduled downtime

Postby Johan G » Wed Jan 21, 2015 6:37 pm

Thanks for the heads up. :)

Edit: It is a bit close, but I think I will add a site notice in case someone is editing etc.

Edit2: Done, using Template:Due date to hide the notice after 20:00 UTC
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5546
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: wiki.flightgear.org scheduled downtime

Postby www2 » Wed Jan 21, 2015 10:14 pm

HI it's now 22:13UTC and the wiki is not yet up.
www2
 
Posts: 261
Joined: Thu Apr 16, 2009 1:58 pm

Re: wiki.flightgear.org scheduled downtime

Postby curt » Thu Jan 22, 2015 12:18 am

They indicated the maintenance is progressing but taking longer than anticipated. That was a couple hours ago so I am trying to contact them for a status check to make sure everything is still progressing. Sorry for the longer than expected outage. :-(
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: wiki.flightgear.org scheduled downtime

Postby curt » Thu Jan 22, 2015 1:16 am

Now the issue (whatever it is, not explained to me) has been escalated ... please stand by.
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: wiki.flightgear.org scheduled downtime

Postby www2 » Thu Jan 22, 2015 6:31 pm

Any updates for now?
www2
 
Posts: 261
Joined: Thu Apr 16, 2009 1:58 pm

Re: wiki.flightgear.org scheduled downtime

Postby chriscalef » Thu Jan 22, 2015 6:39 pm

I always cringe whenever anybody says something will be done in an hour. Been there so many times. :-)

Thanks for all your hard work!
chriscalef
 
Posts: 247
Joined: Wed Feb 20, 2013 9:28 pm

Re: wiki.flightgear.org scheduled downtime

Postby Johan G » Thu Jan 22, 2015 7:05 pm

Still down 19:00 UTC the following day. :|

Good thing this is not a business; this could be worse. ;)
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5546
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: wiki.flightgear.org scheduled downtime

Postby www2 » Thu Jan 22, 2015 11:10 pm

So far as i know (only from the user site) the database is down and the file system is Read-only.
Edit the wiki serve is current offline.
www2
 
Posts: 261
Joined: Thu Apr 16, 2009 1:58 pm

Re: wiki.flightgear.org scheduled downtime

Postby curt » Fri Jan 23, 2015 4:47 pm

This has turned out to be a degenerate mess, sorry everyone! The ISP claimed it would be a quick hour off line and back to everything being good. They just emailed me again so know I'm going to run off and try to address another issue they claim to have discovered and we'll go from there. Again, sorry, I was promised a quick hour offline and it would be done.
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: wiki.flightgear.org scheduled downtime

Postby Hooray » Fri Jan 23, 2015 4:52 pm

I just hope that someone managed to create backups before the whole thing went haywire - the wiki has easily become one of the most useful and most popular web based FlightGear related resources meanwhile - so any kind of data loss would be a pretty serious issues (think avsim debacle a few years ago)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11469
Joined: Tue Mar 25, 2008 8:40 am

Re: wiki.flightgear.org scheduled downtime

Postby curt » Sat Jan 24, 2015 2:38 pm

Here is a quick saturday morning update. Simon and I are still working on this. Simon has verified that the database is intact so that is good news. We have multiple offline copies of the database.

The area of concern is that the server file system got moderately scrambled (I didn't think this sort of failure scenario was possible on a virtual machine when you don't have to worry about physical disk errors, but apparently it can still happen ...) Running "fsck" (similar to dos chkdisk) is the only option, and that found a ton of problems and had to do a lot of things to bring the file system back into consistency. If anyone cares about the gory details, quite a few files were dumped into /lost+found and given #inode_number names.

The other tricky thing is that when the volume got corrupted enough system files were affected so that the machine can no longer successfully boot, so it looks like we'll end up wiping the machine and reloading everything from scratch.

So what we are looking at is:

1. reload the server operating system from scratch
2. restore the needed software (apache, mysql, mediawiki, etc.)
3. do various server and software setup/config steps.
4. restore the mysql database for the wiki (which is intact) ... this gives us all the wiki text content.
5. restore the tree of attachment files (figures and pictures and other media the wiki references.) This tree does have some loss. Most of the lost files are in lost+found, but may now have non-descript file names or directory names.
6. We will then try to work through the lost+found to recover as much of the missing pictures and figures as we can, and some of this may need to be on a case-by-case basis as we discover broken image links. In some cases the whole directory was moved and the file names are intact so when we do see a busted link, we may be able to quickly find the file, and move that whole directory back where it should go.
7. Very recently (like in the last couple weeks) I have increased the space/capacity of my personal backup system, so I will investigate automated backups of the wiki (and the forum).

Yep, and many of us have worked as sys admins in our careers so we know all the comments ahead of time, no need to be repetitive. It's always an issue of available time and energy, less an issue with intent or knowledge. :-)
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: wiki.flightgear.org scheduled downtime

Postby patstan » Sat Jan 24, 2015 3:17 pm

Do you not have backups of the media content which includes things like images?
patstan
 
Posts: 117
Joined: Wed Jul 16, 2014 1:27 pm
Callsign: patstan
Version: 3.0
OS: Windows 8

Re: wiki.flightgear.org scheduled downtime

Postby elgaton » Sat Jan 24, 2015 4:29 pm

Should you need help recovering the images (e.g. because the lost+found directory does not have all of them), an option might be getting them from our browser caches - see this Pro Webmasters question for a description of the approach.
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1107
Joined: Tue Mar 19, 2013 4:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: wiki.flightgear.org scheduled downtime

Postby Johan G » Sat Jan 24, 2015 5:27 pm

@ curt: Thanks for the update on what is going on.

curt wrote in Sat Jan 24, 2015 2:38 pm:Simon has verified that the database is intact so that is good news. We have multiple offline copies of the database.

Those are both good news. :) (Lets just hope that the backups are recent. ;) )

curt wrote in Sat Jan 24, 2015 2:38 pm:6. We will then try to work through the lost+found to recover as much of the missing pictures and figures as we can, and some of this may need to be on a case-by-case basis as we discover broken image links. In some cases the whole directory was moved and the file names are intact so when we do see a busted link, we may be able to quickly find the file, and move that whole directory back where it should go.

I do not really understand what that will look like in the wiki, but if I got it correct they would show up as broken wiki links.

I suspect that these links will most probably show up in the Special:WantedFiles list, as pages will link to them.
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5546
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Next

Return to The FlightGear project

Who is online

Users browsing this forum: No registered users and 1 guest