Board index FlightGear Development Scenery

Next World Scenery build

Questions and discussion about enhancing and populating the FlightGear world.

Re: Next World Scenery build

Postby wkitty42 » Mon Sep 17, 2018 5:29 pm

perhaps this post from the developers' mailing list helps further the understanding?

torsten dreyer wrote:status update:
I can now export the scenemodels-db data to a terrasync master from within a docker container running on an AWS instance.
For some reason, this currently only works if I run the container from the command line but not as a scheduled task from the AWS "elastic container service" on that exact same virtual machine that happily accepts the cmdline execution. Looks like there are still some network access restrictions in place I have overseen.
Currently, only https://dream.t3r.de/fgscenery contains the latest output (incl. Oahu and Jan Mayen custom scenery). Other mirrors will follow soon.

Why does this all take so long?
First, there was the core database holding the scenery models. It's postgresql with postgis extension. It used to have several other extensions, too and it took some time to get rid of all those unsupported (by AWS) extensions.
Then there was the Apache + php(cgi) based web frontend that I had to wrap into containers, divide the Apache service and convert the php part into a separate fast-cgi (php-fpm) service.
There were a few supporting tasks like database cleanup and statistics update that had to be wrapped into containers.
The most interesting and tricky part is the scenery was (and still is) the database exporter to the terrasync master. This consists of a couple of stored SQL procedures executed by python wrappers. This thing writes all database entities to the well-known file structure found in the /Objects and /Models folders by using very clever SQL statements invented by Martin Spott years ago. Before those scripts can start, they have to calculate the ground elevation for new object positions. This is where the the fgelev tool from FlightGear comes into place.
First challenge was to get fgelev running in a headless envirionment on a virtual machine somewhere in Amazon's datacenter (Yes, it needs SimGear and OSG).
Next, it needs access to the full terrain data. This one really drove me nuts. It's impossible to bake 80GB of terrain data into a docker image. On the other hand, it is not affordable for us to have that amount data within a persistent volume inside the AWS cloud. Neither was terrasyncing the entire set into an ephemeral container from our terrasync master an option as we have to pay for bandwith.
The current solution is to simply mount the directory from the terrasync master with sshfs and make it look like a local volume. (sshfs is a wrapper for ssh + sftp and the fuse file system). This pulls in only the files actually needed by fgelev. Doing a few thousands of elevation probes over the network is fast enough and the amount of data transferred over the wire is acceptable. The drawback is, fgelev+sshfs require up to 2GB of RAM plus a privileged docker container and hence a virtual server instance to run at.

Next - and hopefully final tasks are:
- get the scheduled tasks running
- automatically provision the virtual server only when it's needed and discard it afterwards
- reduce operation costs by using "spot instances" (a.k.a: run when machines are cheap, not at a specific time)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 4762
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Next World Scenery build

Postby Müller » Tue Sep 18, 2018 2:16 pm

I do not know if you wrote this for me, but if you did, thank you, although I can have this information on the devel list.
And when I mentioned it is a difficult process, please read it as "Next World Scenery build". Because it involves many resources, time, people, access to data, equipment, and especially great knowledge, that is, it is a very complex and time-consuming job
Müller
 
Posts: 410
Joined: Mon Feb 20, 2012 12:37 am

Re: Next World Scenery build

Postby Catalanoic » Tue Sep 18, 2018 3:37 pm

all you do for FG makes sense for us, please go ahead!
User avatar
Catalanoic
 
Posts: 1014
Joined: Mon Mar 05, 2012 12:33 am
Location: Barcelona (LEBL)
Callsign: Catalanoic
Version: 2017.3
OS: Lubuntu/Windows 7

Re: Next World Scenery build

Postby wkitty42 » Fri Sep 21, 2018 9:36 pm

Müller wrote in Tue Sep 18, 2018 2:16 pm:I do not know if you wrote this for me, but if you did, thank you,

yes, it was for you and you are quite welcome...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 4762
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Previous

Return to Scenery

Who is online

Users browsing this forum: No registered users and 7 guests