Board index FlightGear Development Scenery

Scenery building server - anyone interested?

Questions and discussion about enhancing and populating the FlightGear world.

Re: Scenery building server - anyone interested?

Postby D-YETI » Sat Feb 15, 2014 12:47 am

By the way I personally would give some computing time and IO /Diskspace, at the server listed below, too, because I hope much more pople will benefit from it than running it for me alone. Since Terragear is not so easy to install and even with the gui doesn't always give the result I was expecting.

Intel® Core™ i5-4570 CPU @ 3.20GHz × 4
32GB RAM
SSD + RAID
but rather slow Internet connection. (600kbit/s dn / 128kbit/s up)
D-YETI
 
Posts: 19
Joined: Mon Jul 19, 2010 12:14 pm
Location: Germany
Version: 2.6.0
OS: Ubuntu 12.04

Re: Scenery building server - anyone interested?

Postby Hooray » Sat Feb 15, 2014 4:03 am

The first step would probably be finding someone familiar with installing/setting up TG on Linux, and then also someone familiar with virtualization, so that a Linux distro like TurnkeyLinux can be set up with TerraGear - from that point on, everything should be fairly straightforward actually, assuming that we have someone willing to work on a web-interface -or API- to expose TerraGear as a web service, e.g. using Python, Php, Ruby or Perl - in fact, one could even use Nasal for this :-)

I am not going to work on this anytime soon, but I am willing to help work out the details to get this started.

One of the most straightforward ways to proceed here, would be just setting up TerraGear and using SSH/SCP to interact with the service - obviously, that would require either setting up accounts for each user, or setting up a shared/trusted account - in turn, this could then be supported by Gijs' TerraGear GUI - i.e. kinda like a "remote operating" mode that works over SSH

Qt has various APIs to handle SSH, e.g. see: http://doc.qt.digia.com/qtcreator-exten ... ction.html

EDIT: I now have this working: SimGear & TerraGear built inside a VirtualBox VM running TurnKeylinux (Debian wheezy 64bit) - anybody interested in playing with this, please get in touch.
Image

The only thing missing now is to modify TerraGear GUI to support working over SSH (local vs. remote mode), for example by adding a new tab like this: Image
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby Gijs » Sat Feb 15, 2014 9:06 am

It'd be nicer/better to have a web interface. That'll allow you to keep the interface and backend in sync (without asking people to recompile/redownload the GUI om each change), save people from downloading yet another tool and make it work/look the same on all systems.

Cheers,
Gijs
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9364
Joined: Tue Jul 03, 2007 2:55 pm
Location: Amsterdam/Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Scenery building server - anyone interested?

Postby Hooray » Sat Feb 15, 2014 9:21 am

I am not sure about that - if there's anybody interested in working on a web interface that would obviously be great, but short of that, I would simply modify TerraGear GUI to support two modes local (default) and remote (build server) over SSH. I already looked at the code briefly and the main issue is the amount of Qt code that runs locally, i.e. to download stuff for example - if that could be moved out into external programs, preferably part of TerraGear itself, then it would be trivial to support both, local AND remote TG installations. Currently, there's just too much stuff directly done, without calling external tools - so that would need to be factored out.
Overall, the TerraGear GUI is not such a complex code base, so I am not sure if keeping the frontend/backends in sync is such a huge issue ?? I mean, people ALREADY need to have a matching TG/TGGUI versions after all ...
I am certainly not going to stop anybody from coming up with a webservice - it's just something that I am probably not going to work on anytime soon - while modifying TGGUI accordingly would just take a few hours and involve well-understood steps.
So I wasn't thinking of forking TGGUI, but more about extending it to support local AND remote installations - selected/configured during startup.

Obviously you are much more familiar with the code in question here - but based on what I've seen, it would not be such a difficult undertaking to turn TGGUI into a "remote control" for a TG build server setup.

If frontend changes are such an issue, I would actually suggest to get rid of the hardcoded GUI and use Qt's scripting support for those *.ui files instead - that would allow you to automatically download QtScript while booting.

Finally, coming up with a working web service is in my opinion more difficult than using SSH to teach TGGUI to run a handful of commands remotely and fetch the results via SFTP - so yeah, I am kinda lazy here :D
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby Soitanen » Sat Feb 15, 2014 10:18 am

Just my opinion: I don't need to have this remote Terragear server. I found very easy and problem-less way to compile TG (using Calculate Linux), and I use custom-tweaked TG to generate some country-specific features, so I think that server will not help me. Also, some tuning of shapes is needed after downloading.

Much more interesting for me will be possibility to add my custom scenery (not objects, terrain) to Terrasync server, but this is offtopic here.
Boeing 737-300. Reworked cockpit, FDM, autopilot and much more. WIP.
Boeing 737-800. WIP. Canvas PFD and ND.
Antonov An-24B. Made from scratch. Very good FDM. 3D model by Adrian. WIP.
Project Russia (some cities, based on OSM with custom objects).
Soitanen
 
Posts: 489
Joined: Sat Jun 16, 2012 6:50 am
Location: Saint-Petersburg, Russia
Version: git
OS: Linux Mint 17

Re: Scenery building server - anyone interested?

Postby psadro_gm » Sat Feb 15, 2014 2:10 pm

An issue you will may hit is that some terragear tools don't behave very well when run at the same time (genapt and ogrdecode, in particular). this may work with two separate work directories, but I have not tried it. If using the same work directory, You can hit some directory / file creation collisions. To recitify these, I added a Boost global mutex so one can write at a time. With multiple users, this may become a bottleneck.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 750
Joined: Thu Aug 25, 2011 2:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Scenery building server - anyone interested?

Postby Hooray » Sat Feb 15, 2014 2:48 pm

I was thinking along the lines of setting up TG system-wide but always adding a user (with quotas) - so that each TG run would be kept inside the user's $HOME - would that work, or should we look for some workaround (like chroot) ?
I don't quite foresee hundreds of people using such a system - I am more interested in setting up the infrastructure, to allow 5-8 people to use such a system without having to build/install TG, some of us pretty powerful servers available, with lots of space, RAM and CPU resources - so why not use these here ...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby Hooray » Tue Mar 11, 2014 9:56 pm

Image

As part of working on a long-standing feature request, namely having a central shared TerraGear scenery build server, saiarcot895 has created Linux packages for Debian/Ubuntu (deb/ppa) to install precompiled TerraGear binaries. This now greatly simplifies installing TerraGear, because people no longer need to manually set up a complete Linux build environment and all SimGear/TerraGear dependencies. Everything is now done in an automated fashion.
Next, we're hoping to use this to install TerraGear on a public server for which people can ask for remote shell access (SSH). People interested in exploring this, should be getting in touch via the forum or the wiki.
Obviously, users will still need to be familiar with TerraGear, but they may benefit from reduced bandwidth restrictions and/or more horsepower in comparison to running TerraGear locally (e.g. one user offered to contribute hosting on a 32gb RAM and 8-core server). So this could be a great opportunity for people to run scripted/unattended jobs, without having to go through the hassle of downloading/building/installing and configuring TerraGear. But so far, being familiar with TerraGear and Linux is going to be a prerequisite still.
Once that is working, we'll investigate making the setup reproducible by using TurnKeyLinux. Once we have a working TKL distro, we can install a full TG setup in just a few minutes by downloading an ISO file and installing it in a VM (VMWare/VirtualBox). This would basically allow people to easily download/install TerraGear locally, either installed next to their OS, or as a virtual machine.
The long term idea is to hook up TerraGear GUI to it, so that the GUI front-end talks to TerraGear across SSH.
This is currently still a use-case for which TG wasn't designed for, and the TG developers mentioned already on the forums that there may be some roadblocks ahead, so everything here is still highly experimental. But ultimately we hope to provide a front-end to a Linux-based TerraGear VM, either by reusing TerraGear GUI or by coming up with a custom web-based front-end (please get in touch if you can help with this!).
If you're interested in helping or learning more, please get in touch via the forum or via the wiki: http://wiki.flightgear.org/TerraGear_sc ... erver#News.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby Hooray » Thu Apr 03, 2014 2:07 pm

http://wiki.flightgear.org/FlightGear_N ... _F-JJTH.29
Image
http://wiki.flightgear.org/TerraGear_sc ... ild_server
F-JJTH wrote:
But maybe we could create a deb/apt package so that we can easily install this inside the VM ?

Setting the required environment is really simple:
1) Having a debian based OS
2) Install the terragear toolchain (I already providing a download_and_compile_tg.sh since some months and it works really fine)
3) Install a webserver

That's all, it take less than 10 min to setup the required environment.


I like your original tg build script, but I'd prefer to get the cmake files extended accordingly. that would be in line with James' superbuild/fgmeta efforts.
In general, build scripts should no longer be OS specific. If we could extend the superbuild/fgmeta stuff to also build TG, we could then also use cpack to create/update the linux packages, which would greatly simplify TG deployment. I am sure that saiarcot895 could help with the specifics here, so that we can get rid of any custom scripts, and just integrate everything with superbuild/fgmeta, so that this stuff could possibly be even run on the build server...

For the web service, I would prefer to also create a package, so that future updates can be easily shipped to all people running it.
At least, that should be a long-term goal - manually having to download, compile and install things is too fragile and doesn't scale - see the number of fgms servers we have for example ...


F-JJTH wrote:Well after some IRC discussion I have my answer:
- nobody is working on HGT file (TIFF format coming from http://www.viewfinderpanoramas.org
- our SHP are already really complex and we don't want that people add more complexity to our SHP. (We can imagine some funny guys adding all the swimming pool of their village !!! We surely don't want that )

That's why we don't need to give "easy-access" to our user to ogr-decode/hgtchop/tg-construct tools.

However we want detailed airport ! So it make sense to give easy-access to genapts850

Because this part is already implemented, I will simply improve this part and see if our scenemodels.flightgear.org guys are ready to host the project :)


psadro_gm already mentioned that some tools may have mutex issues when running "concurrently" - so if that still is a problem, we may need to use cron jobs to schedule things sequentially, where each finished/terminate job would re-schedule itself or the next job. Once that is no longer a bottleneck, we could also run multiple jobs in parallel, and merely re-nice each job to a lower priority - not sure if RAM may be an issue though ?

I think it would be a good idea to prepare support for multiple back-ends/TG servers.

My proof-of-concept is not oriented in this way, in fact I'm more and more thinking to limit the tool to genapts850. Why ?
Because I'm not sure we want that people improve our shapefiles, they are already really complex (we have seen a lot of report about scenery 2.0 eating too many RAM), also I'm not sure we have a lot of people interested in improving HGT (altitude) file.
So finaly, I think our user are mostly interested in improving their airports.

Any opinion about _only_ providing a genapts850-web-generator ?


It clearly is better than anything we have now - but personally, I'd try to keep it scalable - i.e. by coming up with a CLI tools wrapper that turns CLI profiles (program path, name, settings, files) into a web service (API). That way, we could also support other tools in the future, without having to rewrite tons of things.

That's basically the approach I experimented with when modifying Gijs' Qt GUI.
And power users would be able to create new "profiles" and chain them together with other programs.

When I played with this a few weeks ago, I even found a few tools that would create ncurses-based GUIs by using a profile for each tool - and there's another python lib for wrapping CLI tools and exposing them as a web service, which also handles uploads/downloads.

Having/supporting just a single server may inevitably become a bottleneck sooner or later.
For those reasons, I'd prefer to 1) support multiple servers (i.e. by running jobs over SSH), and 2) use a simple cron-based scheduler - or maybe even using Linux "batch" to schedule things directly based on overall system load.

But maybe this is too sophisticated for now - let's better postpone this discussion until we have someone showing up and willing to take over maintenance.
So what is involved framework-wise here, i.e. dependencies (php, perl ... etc ?)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby F-JJTH » Thu Apr 03, 2014 6:02 pm

Hooray wrote in Thu Apr 03, 2014 2:07 pm:So what is involved framework-wise here, i.e. dependencies (php, perl ... etc ?)

Almost no dependencies, just a web server and the TerraGear toolchain.
On a debian based-OS it's as simple as :
Code: Select all
sudo apt-get install apache2 php5 libapache2-mod-php5 php5-mcrypt
mkdir terragear
cd terragear
wget http://clement.delhamaide.free.fr/download_and_compile_tg.sh
chmod +x download_and_compile_tg.sh
./download_and_compile_tg.sh


Then put the website file (not yet available...) in his /var/www directory.

If someone has problem with the list of command line I listed just above I guess this guy is not the good person for hosting this tool :) They are all pretty basic command (create a directory, install some packages, execute a script...)


I will wait some feedback (is this project is interesting ?) before digging more in this tool.

Regards,
Clément
User avatar
F-JJTH
 
Posts: 697
Joined: Fri Sep 09, 2011 11:02 am

Re: Scenery building server - anyone interested?

Postby Hooray » Thu Apr 03, 2014 6:08 pm

ok, thanks for clarifying - so it's just using a standard LAMP stack ?
Are you using any library/framework for creating the wizard (prev/next) ?
Also, once you make the code for this available, I would like to check that this also works properly with the debian/ubuntu packages available for TG now (there should be no reason why it should not work) - because that should be more straightforward than building from source, and there won't be a complete C++ build environment required. Next, I would like to turn your code into a debian package and host it, so that people can easily install the whole thing, where it would then automatically pull in the TG packages.

Once that is working, I can provide a pre-created ISO image based on TurnKeyLinux/Debian Wheezy with everything included.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby F-JJTH » Thu Apr 03, 2014 6:22 pm

Hooray wrote in Thu Apr 03, 2014 6:08 pm:so it's just using a standard LAMP stack ?
Yep


Hooray wrote in Thu Apr 03, 2014 6:08 pm:Are you using any library/framework for creating the wizard (prev/next) ?
No, I created it "by hand", I'm only using Leaflet for the map, and jQuery (with UploadFile plugin) but none of these 2 frameworks require any installation.

Source code is here: (be sure the code is really ridiculous ! less than 800 lines...)
http://clement.delhamaide.free.fr/terragear-web-gui.zip

Take care to define $TG_BIN variable (command.php:3) who point to the run_*.sh script (in my case, feel you free to point directly to binary file instead of these "proxy" file executing the real binary as parameter)
Also the $_SESSION['workspace'] variable (index.php:26) but this one is universal and should work.

Regards,
Clément
User avatar
F-JJTH
 
Posts: 697
Joined: Fri Sep 09, 2011 11:02 am

Re: Scenery building server - anyone interested?

Postby Hooray » Sat Apr 05, 2014 3:14 pm

we may want to add a feature that allows people to create/save their own "profiles" with custom settings, so that people can more easily come up with working default settings, as per: viewtopic.php?f=5&t=22256&p=205111#p205108
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenery building server - anyone interested?

Postby F-JJTH » Sat Apr 05, 2014 5:39 pm

No need of "profiles settings", the problem reported in your link is not caused by wrong settings.
Also there is no "custom settings per user" to save anywhere. The web interface is done in a way that everything is really simple and easy.

Regards,
Clément
User avatar
F-JJTH
 
Posts: 697
Joined: Fri Sep 09, 2011 11:02 am

Re: Scenery building server - anyone interested?

Postby Hooray » Sat Apr 05, 2014 5:58 pm

Use elevation file from mapserver.flightgear.org (Not yet available)

reeed also mentioned that idea in his original proposal:
Image

From a coding point of view, the simplest solution would be using NFS to mount an existing location read-only, we could then set up a network share, that could be used (and cached) by all build servers. It would take only 5-10 minutes to set that up on the VM side of things, and it would be automatically available.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 7 guests