Board index FlightGear Development Scenery

Create a BOINC project for scenery generation?

Questions and discussion about enhancing and populating the FlightGear world.

Create a BOINC project for scenery generation?

Postby sgofferj » Fri Dec 09, 2011 2:20 pm

Hi,

I understand that generating the scenery from the raw data takes quite some CPU time.
How about creating a BOINC project which could harness the idle CPU-power of many users to assist with this job?
FG 3.1 GIT / Opensuse 12.3 / Phenom II X4 / GForce GTX560
Stefan's little Flightgear corner | The Finnish Weather Center | Wolves in Finland

Working on: EFTP
COM: IAX2:home.gofferje.net/stefan (MO-FR 0700-2000 UTC, SA+SU 0900-2000 UTC)
sgofferj
 
Posts: 789
Joined: Mon Dec 05, 2011 5:13 pm
Location: EFTP
Callsign: OH-SW
Version: 3.1 GIT
OS: Opensuse

Re: Create a BOINC project for scenery generation?

Postby longfly » Fri Dec 09, 2011 4:38 pm

Actually I dont have a clue about this (generating scenery) and I think there're many others like me out there. So a BOINC-Project would be a way to contribute to FG really ease! I would join this. :P

Regards Longfly
User avatar
longfly
 
Posts: 228
Joined: Tue Dec 07, 2010 5:41 pm
Location: EDDM (Germany)
Version: always new

Re: Create a BOINC project for scenery generation?

Postby curt » Fri Dec 09, 2011 9:12 pm

For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files. There is a computational aspect, but on the whole it's disk IO that will limit things.

Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time.
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: Create a BOINC project for scenery generation?

Postby sgofferj » Fri Dec 09, 2011 9:49 pm

Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^.
But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.

Maybe it actually would be a good idea for the Terragear people to concentrate on toolchain and workflow for a moment and come up with something that in basic can work fairly automatically. That would for sure be an investment into the future, because toolchains and workflows which need extensive manual intervention not only eat up resources unnecessarily but also include inherent risks for the whole project because these kinds of manual operations require specific knowledge which is incorporated in persons. If the persons leave the project for what reason ever, the whole project is in jeopardy.
FG 3.1 GIT / Opensuse 12.3 / Phenom II X4 / GForce GTX560
Stefan's little Flightgear corner | The Finnish Weather Center | Wolves in Finland

Working on: EFTP
COM: IAX2:home.gofferje.net/stefan (MO-FR 0700-2000 UTC, SA+SU 0900-2000 UTC)
sgofferj
 
Posts: 789
Joined: Mon Dec 05, 2011 5:13 pm
Location: EFTP
Callsign: OH-SW
Version: 3.1 GIT
OS: Opensuse

Re: Create a BOINC project for scenery generation?

Postby statto » Sat Dec 10, 2011 6:04 am

Compiling a detailed area of 3 square degrees across only takes a couple hours running on a not-very-powerful server. Admittedly that's not that large of an area, but I've never found the build process to be unreasonable, though I only work with small batches.

I also have a bash script I use for Terragear, which makes the scenery generation process very simple. I will share it when I update my website.

In my opinion the biggest issue is with the quality of the scenery data and not necessarily with building the scenery, but I also have a working copy of TerraGear, and there have been some problems with recent scenery builds. Honestly compiling TerraGear could be the biggest issue :)
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2106
Joined: Fri Jan 25, 2008 10:57 pm

Re: Create a BOINC project for scenery generation?

Postby ot-666 » Sat Dec 10, 2011 3:17 pm

I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery.
After the task is done the data would be uploaded back to the server.

The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave.


Oliver
Callsign: ot-666
Working on LOWI and other stuff - Custom Scenery Overlay Repo: http://gitorious.org/fgfs-custom-scenery/custom-scenery-overlay/
VMX22 - Osprey... sometimes in 2014
ot-666
 
Posts: 746
Joined: Sun Nov 08, 2009 6:14 pm
Location: Germany, Konstanz
Callsign: ot-666
IRC name: ot666
Version: GIT
OS: win7 64bit

Re: Create a BOINC project for scenery generation?

Postby Hooray » Fri Jan 31, 2014 2:02 am

Just for future reference, because this seems hard to find: http://wiki.flightgear.org/TerraGear_sc ... ild_server
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Create a BOINC project for scenery generation?

Postby Hooray » Thu Dec 03, 2015 2:26 pm

Subject: Scenery building server - anyone interested?
MyName wrote:I recently replaced my slow AMD Sempron 3000+ CPU with a AMD Athlon II X4... It really made a difference in the games I have, as the processor was bottlenecking the system, but I think I don't play enough to make good use of this CPU. I already know about F@H, SETI@HOME, BOINC, etc. but I wanted to do something to this project as well. I thought that scenery compilation was demanding, so I thought of setting up a VM on this system with scenery compilation tools. I don't know the system that you(scenery developers) use, but this might come in handy if your system is slow. If there is enough interest I'll look into this and try and implement it.
(I currently have 2 GB of RAM, but I'm obviously going to increase this before setting it up)


Subject: Google code colse there doors
elgaton wrote:Therefore, we could adopt an even simpler approach:
  1. improve the existing tools (and add new ones) to support regenerating a single tile without depending on the surrounding ones;
  2. expand the FlightGear Scenery Database webforms, if needed, to allow any FG user to submit improvements that can not be done via the Airport Gateway or OSM;
  3. periodically grab all improvements from OSM/the Airport Gateway via their APIs;
  4. regenerate the single tiles using a distributed system like BOINC (see this page on the wiki);
  5. publish the scenery via HTTP (from a single website or a network of mirrors).
Clients would thus only need to perform an HTTP request that includes an "If-Modified-Since" header to check whether the scenery has changed or not. (Any correctly configured HTTP server would thus serve the updated tile or issue a "Not modified" response). The only caveat is that, if we decide to use a network of mirrors rather than a single site, timestamps on files should be synchronized (otherwise, should the client decide to change the used mirror, it might be tricked into downloading the same scenery again when it's not necessary to do it).



This is currently getting some attention on the devel list:

Continous, Distributed Generation of Terrain
Torsten wrote:Hi all,

when our main scenery resource will be gone in a couple of weeks, we will
have a hard time to create a new world scenery. Generating the entire world
was a job that ran continuously for days - if not weeks, even on a powerful
machin with almost unlimited resources.

As we have to re-think the entire process anyway, I just had the idea to
use our users resources for that, probably by using something like
BOINC: https://boinc.berkeley.edu/trac/wiki/AppIntro
We sure need to do many tweaks within the tool-chain, but I can't think of
any reason why this should not work.

Does anybody have experience with BOINC or alike?

Torsten
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Create a BOINC project for scenery generation?

Postby psadro_gm » Fri Dec 04, 2015 4:09 pm

I am currently changing the intermediate file format for the toolchain ( as requested by Martin Spott ). It was a good idea, so I'll continue down this path even in his absence. But it's also a good time to look into how to share these files across boinc.

The two largest operations are ogr-decode, and tg-construct.

ogrdecode is already being broken into two - one utility for landclass polygons, and one utility for vector data. This was done because vector data interacts between many different 'shapefiles' . The OSM data has z_order which determines, for instance, if a primary road passes over a motorway, underneath. (or intersects). The geometry built changes based on this...

I've already designed these tools to be multi-threaded, so breaking the work up between several machines should be doable.

I will take a look at how data is passed between client / server and try to get a working experiment for teh decode utilities in the next few weeks.

Pete
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Create a BOINC project for scenery generation?

Postby f-ojac » Fri Dec 04, 2015 6:50 pm

Yep, tell us if you need some computing/disk/network power.
f-ojac
 
Posts: 1304
Joined: Fri Mar 07, 2008 10:50 am
Version: GIT
OS: GNU/Linux

Re: Create a BOINC project for scenery generation?

Postby elgaton » Fri Dec 04, 2015 9:06 pm

@psadro_gm - since I've had an in-depth look at the wiki (as I had the same idea, did not look much into it due to a lack of time on my part), I hope you don't mind if I provide a couple of tips.

  • To allow people to be able to run the TerraSync toolchain on all Linux distributions without incurring in library problems, the chain must be built statically on an old Linux distribution (see the Linux compilation page).
  • The basic API and network applications page might be helpful.
  • There is also a wrapper that could be used to run the toolchain without editing the source code - however, that would not allow the checkpointing feature to be used. (This means that, if for some reason a stage is interrupted, it must be restarted from the beginning, which can be quite painful if the task takes several hours to be completed, like tg-construct).
  • Once the server is set up (some distributions, like Debian, have a wrapper for this) and the toolchain binaries are signed, it suffices to split scenery generation in work units and generate them on the server.

Since BOINC allows input files to be kept once a task is finished (locality scheduling), I'd suggest:
  • introducing checkpoints in TerraSync (that would benefit all users, in my opinion, not only BOINC ones);
  • generate a work unit for each tile, using locality scheduling for the files that do not change often (e.g. SRTM elevation data);
  • repeat the process - this way, people who have already generated a tile will be preferably assigned the same one in the future, reducing bandwidth consumption.
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1106
Joined: Tue Mar 19, 2013 5:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: Create a BOINC project for scenery generation?

Postby psadro_gm » Sat Dec 05, 2015 3:57 pm

I spent this morning playing around with boinc. I created an instance on AWS ( from boinc tutorial here: http://boinc.berkeley.edu/wiki/Installing_BOINC_on_EC2 ) to run the server and connected from the client on fedora. It's pretty easy to set this example up. Now to actually integrate with vector_decode...

I also read up on checkpointing. I am unsure if we can use this, as if a single tile computation fails - we really need to start over. checkpointing seems to make sense if your input is processed in a serial way. read from file - perform action - write output - read the next item... For each stage, a work item will be a single tile, so if it fails - we'll just start the tile again.

as far as giving the same tile to a client, I am unsure about this as well. Here's what I see so far. There are 9 stages of scenery generation. (currently 8, tglod is future...)

1) hgtchop - input is a bucket id. files need to be pulled from viewfinderpanorama.
2) hgtfit - input is a bucket id. output is the arr.gz, and arr.fit.gz file
3 and 4) vector-decode / shapefile decode. The 'input file' is simply a bucket id. The client needs to read from the postGRE database the shapefiles that lie within the spatial query of this tile. It will then generate the intermediate shapefiles as output.
5) genapts. the input file is an ICAO. the client reads apt.dat and generates the shapefiles for the airport.
6, 7, and 8) tgconstruct. the input files are all of the shapefiles in the preceding 2 stages tgconstruct will actually be 3 runs. It may even be 3 different executables. first stage creates a triangular mesh. second stage matches nodes on tile boundaries ( so all tiles need to be complete from stage 1 ), and then calculates elevation for each node. third stage computes normals, as nodes on the edges need top know all of the adjacent face normals in neighbor tiles.
9) tglod : the final stage ( for a future flightgear ), would assign groups of buckets for lod simplification.
Last edited by psadro_gm on Sat Dec 05, 2015 9:32 pm, edited 1 time in total.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Create a BOINC project for scenery generation?

Postby Hooray » Sat Dec 05, 2015 5:12 pm

This could be pretty compelling, given that the FlightGear community is likely to consist of users with tons of horsepower (CPU cores, RAM, GPUs, VRAM), and because BOINC is going to support OpenCL: http://boincfaq.mundayweb.com/index.php ... 1&view=506
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Create a BOINC project for scenery generation?

Postby psadro_gm » Sat Dec 05, 2015 9:19 pm

Unfortunately, my current work with terragear is going in the opposite direction with respect to number formats. It now uses CGAL multiprecision math ( required for clipping, and adding nodes on colinear segments). Very little of the toolchain works under the floating point number types used in openCL. The good news is that multiprecision math is starting to get some (rudimentory) support in new Intel processors.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21


Return to Scenery

Who is online

Users browsing this forum: No registered users and 5 guests