Scenery building server - anyone interested?

[MyName]

I recently replaced my slow AMD Sempron 3000+ CPU with a AMD Athlon II X4... It really made a difference in the games I have, as the processor was bottlenecking the system, but I think I don't play enough to make good use of this CPU. I already know about F@H, SETI@HOME, BOINC, etc. but I wanted to do something to this project as well. I thought that scenery compilation was demanding, so I thought of setting up a VM on this system with scenery compilation tools. I don't know the system that you(scenery developers) use, but this might come in handy if your system is slow. If there is enough interest I'll look into this and try and implement it.
(I currently have 2 GB of RAM, but I'm obviously going to increase this before setting it up)

[Avionyx]

Certainly sounds interesting, would mean I could build scenery while away from home which would be a huge bonus for me, as I spend most my time on my laptop!

[Hooray]

That's indeed a good idea, a while ago, the forum user reeed had exactly the same idea, see here: viewtopic.php?f=5&t=8735

So there's certainly interest in this sort of thing, and I am pretty sure you'd get to see a number of feature requests for such a thing.

Back in 2010, I exchanged a bunch of eMails with reeed regarding this TerraGear/VM setup - the neat thing about a VM image (i.e. VirtualBox-based) would be that it could not only be provided as a server, but that it would also be possible to create a bootable "live CD/DVD" - so that people could easily run and install this things locally.

I will check if I can still find those messages and forward them to you.

Personally, I think that using virtualization is an excellent opportunity here to simplify participating in FG development, if we had "live CDs" for most criticial FG processes, we'd surely have more people able to contribute.

Though, I am really not sure if starting with TerraGear is the simplest approach to pursue, I think there are several potential pitfalls - especially given that it is currently being worked on, but also due it not really being the most intuitive piece of software in the FG eco system

I consider it pretty challenging.

If you're just looking to get started, you might just as well look into creating VM images to set up a multiplayer server (fgms), this is well documented in the wiki.Even today, many people find it not so easy to set up everything properly. So having a VM image to easily run fgms might be a nice practice to get started?

And you'd surely gather valuable experience for your TerraGear image, too.

The neat thing about this whole VM-based approach is that everything would be neatly encapsulated - so that would also make it possible to run several VMs at once (i.e. a fgms & terragear servers at once) while still controlling everything from the central host system, this could even be done using a simple GUI to easily configure and control all VMs.

Another thing worth keeping in mind is compatibility, i.e. 32 bit vs. 64 bit OS. In the case of TG, 64 bit architectures offer theoretical benefits, regarding memory use - OTOH, many people these days are still running 32 bit OS.

If you manage to get this working, it would surely be a significant contribution - especially once people actually start supporting it to simplify things even further.
For example, the "TerraGear GUI" could be changed to support a "remote mode" where a build server can be used, so that scenery is built remotely. In the same sense, a web frontend could also be provided to allow people to easily use the server.

A GIS/GRASS plugin might be another interesting option.

The folks currently involved in doing scenery-related coding are psadro_gm and papillon81

Regarding scenery work in general, Martin is generally considered to be the head of the scenery department, he also has access to server resources and is running the landcover DB/mapserver.

[Hooray]

Also note that when we looked into this last year, we found a number of Linux distributions that are specifically designed to support "re-mastering", i.e. you can create a custom Linux distribution just by setting everything up as needed, and then create a bootable live CD automatically (i.e. scripted or GUI based) 100% based on the running "live session".

Basically, this would allow you to download such a distro's ISO image, then install it into a VM - set up everything as needed, and create a completely new ISO/VM image based on your customizations.

Note that you can do all of this easily using the free SuseSTUDIO service provided by OpenSUSE. This is a completely web-driven wizard to create OpenSUSE-based live CD/DVD images, completely set up and customized to your own needs.

You can even test and run the images remotely, but you can obviously also download the images once things are working.
It is also possible to set up "projects" - so that several people can work together.

[f-ojac]

Yes, that would indeed be interesting, with a small web interface to interact with the user.

Always wanted to do it, never found the time...

[ot-666]

Just an idea…
A way to distribute work to a bunch of vm scenery bulding servers would be nice.
Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.

[statto]

Actually, running TerraGear from the command line is rather easy - you put everything in data and then wait for everything in output

The main trouble is in compiling TerraGear and the fact it doesn't always play nicely. I think the fact the tutorial on the wiki is segmented doesn't help either - it's far easier to put all your elevation data in one folder, all your shapefiles in another folder, your airport data in another folder and then start running the process.

[Hooray]

ust an idea…
A way to distribute work to a bunch of vm scenery bulding servers would be nice.

Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.

I asked reeed to check if he can still find our conversations from a year ago, so that we can post everything here.

[MyName]

I think I'll need to postpone this idea, it seems TerraGear is not very "stable" and exactly easy to get running at the moment... Don't think it's the best thing for a clueless user who's never installed it before to use it.

[statto]

Probably a good decision. Lots of changes going on with the software.

If this is a project which still interests you however perhaps you could still get some of the ground work with who has access to the server down.

[Hooray]

just trying the concept by coming up with a fully virtualized setup of fgms might be another option, that should be much easier to accomplish, but would be a nice exercise to see how things can be deployed, configured and run - sort of like a "beta test" to gather more experience with virtualizing FG related software.

[reeed]

PM exchange in 2010 between Hooray and reeed.

See also the wiki page.

I really think you've had an excellent idea here:

Subject: A scenery build server?

Many people (esp Windows users) have reported problems getting TerraGear tools to run. This is probably a significant factor limiting community contribution to FG custom scenery.

A 'build server' is one solution to this problem -- a server that accepts public-contributed shapefiles, builds the tile in an automated process (using the latest bugfixed TerraGear tools), then provides a way for the user to download the resulting .btg files to check his work.

Let's pool our know-how and resources together and aim to setup a scenery build server!

It's the most sensible, most logical and most efficient thing to do.

Ideally, we would only set up the whole thing once on the server side and then provide an API that could be remotely leveraged with GUI frontends on the client side. This would have a whole load of advantages, server-side processing I mean.

I think it would be cool if we could come up with a list of requires steps, such as:
1) get a server with shell access
2) set up terragear
3) download all required data sources
4) provide a custom API that can be remotely used for building scenery on the server

That would really be a great addition if we could get that going.
Ideally, we could work together with the custom-scenery.org folks (mapserver) and the scenery objects database (stockill).

Integrating all the existing infrastructure and tools in that fashion would be a great step forward.

It would also directly address all issues related to locally setting up terragear stuff, but also updating everything would be much easier. We only need a fairly capable server. But maybe there are ways to get a good shell server for open source projects?

I could imagine getting in touch with a number of virtual (or even dedicated) hosting providers to ask them if they'd be willing to support flightgear by donating a server.

If you are still interested in setting this up, I would love to help you doing this.
I think one important step would be planning this through:

1) find a server for development/testing purposes (could be done locally, using localhost and NAT for public exposure/beta testing)
2) set up terragear and all dependencies (there are automated build scripts, which really should be be used and adjusted where necessary, to have a completely repeatable setup strategy (i.e. to have the option to set up other scenery build servers at some point)
3) integrate with custom scenery, mapserver and scenery db
4) find a way to create a robust remote API (something like XML/SOAP, or JSON based)

Looking at the recent contributions (i.e. FGSignMaker), it seems we have some pretty capable web developers here, so it might even be possible to directly use a web API, so that users could use the scenery build server with their favorite server.

On the other hand, in the long run it would be REALLY neat, if we could integrate such a tool directly with the GRASS toolsuite, i.e. as a custom plugin.

yes you are absolutely right in the steps you outlined. The drawback, however, is that users still need to fiddle with sftp / scp, and then mess with the commandline (the options for shape-decode and fgfs-construct are no joke), download the tiles with sftp / scp, etc.

I was actually thinking of integrating the ssh functionality directly into the GUI, so that this would be done transparently ...
Users would not necessarily need to be aware of remotely using a server.

I would suggest, in addition, a web interface to handle the up- and downloads, and offer compilation options.

Yes, that might be another good option to pursue!

(Re: Gij's TerraGearGUI)
The commands will remain the same, regardless of where they're being run (build server vs user's machine).
That's what I am thinking - so, Gijs' GUI will remain useful.
It would only need to be enhanced with SSH functionality.
Depending on the programming language he uses, there are probably various libraries/options available here.

So the command strings that his GUI builds would then merely be appended to an SSH command stream.

(Re: finding a provider or host)
Another option that could be pursued: getting in touch with various "big" open source hosters to ask for support.

But for the time being, the easiest thing would be to write down our proposal and then get in touch with the right flightgear people. In this case, Gijs (for the GUI part) and martin (for the scenery part).

I would imagine, that it's possible to host some of this on the existing mapserver infrastructure!?

A few reasons why i'm hesitant:

• the response on the forum was lukewarm
• I don't have the know-how to setup a server on my own
• Gijs has released the TerraGearGUI which promises to be the messiah for Windows users

Tha said, I sense momentum building up in scenery creation of late. If ever a need arises for a site to host compiled .btg files, perhaps we can revive this build server idea statto invites users to submit compiled .btg files to him for hosting. But really, the folks at custom-scenery.org ought to be doing more work more visibly.

I had some time to actually look into this, and I am now more than ever convinced that your idea is the way to go, let me clarify:

I am not sure if you are familiar with Linux or not, but basically having a "terragear server" could be as simple as setting up terragear -once and for all- on ONE Linux box, then set up a new "terragear" user with a private home directory (user:terragear and password: terragear) on the same box.

Then it would only be a matter of accessing the CLI (command line interface of the terragear box) by using a standard SSH client, which would make it possible to run commands on the server.

In other words, there would be no need to have any custom protocol or anything else.

It would just be a matter of:

• get access to a (powerful) Linux box with plenty of RAM and hard disk space (virtual or dedicated)
• build terragear and its depedencies on that box (i.e. requires compiler)
• install all required data files for compiling scenery (shapefiles etc)
• add a new 'terragear' user
• provide ssh access to the box

The next step would be to properly integrate this with the terragear GUI that Gijs is working on, which should be doable because it will also just be a wrapper for the shell environment. So the GUI will merely need a way to SSH into the "terrragear build server" and run its commands there.

[f-ojac]

On the devel list, Martin just announced he was close to completion to be able to build www scenery - and on the fly - daily basis.

To be read here : http://sourceforge.net/mailarchive/foru ... gear-devel

[Johan G]

That's exciting news.

[Hooray]

Just for future reference, because this seems hard to find according to requests I received via PM, and because the idea keeps popping up every few months: http://wiki.flightgear.org/TerraGear_sc ... ild_server