Board index FlightGear Development Scenery

Tutorial: Building Europe scenery

Questions and discussion about enhancing and populating the FlightGear world.

Tutorial: Building Europe scenery

Postby adrian » Fri Oct 08, 2010 1:33 pm

Building Europe scenery with SRTM v4, Corine landclass and OpenStreetMap data
Edited on 10/10/2010 to add information on OSM roads
Updated on 05/12/2010 to add further info

This is intended to complete and add up-to-date information on the wiki tutorial.
http://wiki.flightgear.org/index.php/Using_TerraGear

I will assume you are using the Linux build of Terragear. For instructions on how to get it compiled and installed, see
http://wiki.flightgear.org/index.php/Bu ... _Ubuntu_64
Replace all paths where appropriate.

First step after you have finished compiling terragear is to go to the file BuildTiles/Main/main.cxx and make sure CPU rlimit is a large number. I suggest 600000, this will let fgfs-construct run for almost a week. If you don't perform this check fgfs-construct will exit with error after a short time.
Make sure you edit the right values, there are two sets one below another. The first ones are not used.

Acquiring data

I. Elevation
SRTM version 4 is the latest version. Download
http://srtm.geog.kcl.ac.uk/portal/srtm41/elevation.kmz and open it in Google Earth.
In Temporary Places select elevation and choose the 5x5 degrees tiles or the 1x1 degrees subtiles.
Make sure you download GEOTIFF files. I suggest doing 10x10 degrees areas, this means 4 files. It takes longer to compile scenery but on the whole the process is more efficient. Later you will see why.

II. Landmass
Get VMAP0 landmass from http://mapserver.flightgear.org/download.psp (v0_landmass) for the 10 by 10 degrees terrain area you selected. Alternatively use other sources for landmass areas (which ones?).

III. Corine landclass
Get the full Corine set from http://www.eea.europa.eu/data-and-maps/ ... ta-version

IV. Road, railroad and coastline data
Download shapefiles only for countries inside the 10x10 tile from http://download.geofabrik.de/osm/
This will give you roads and rairoads. For coastline, go to http://downloads.cloudmade.com/
Download the same files for your country but you will only use coastline for landmass adjustments.
Do not use road data from Cloudmade since it can bomb ogr-decode. Reason still to be reasearched.
Other datasources can be found here: http://wiki.openstreetmap.org/wiki/Planet.osm

Preparing data

I. Elevation
Assuming you are following the directory structure from http://wiki.flightgear.org/index.php/Using_TerraGear , run srtmchop on each TIF file from data/srtm.
There is a catch here: work/SRTM-3 has to be named exactly like that. Otherwise genapts won't find the elevation required for airport surfaces. This name is hardcoded inside the program.

I suggest to run srtmchop manually for each file to check for errors.
Ignore warnings generated, some header data cannot be decoded correctly but it won't afect the result.

On certain TIF files, srtmchop will terminate with a memory free() error. I have not figured yet what causes this error only on some files. Until a fix is commited to GIT, you can run srtmchop through valgrind:
Code: Select all
valgrind --leak-check=yes ./srtmchop <file> <output dir>

This will stop the program from exiting and will get the job done, albeit 10 times slower.

After you are finished with generating arr files, you can run terrafit.py:
Code: Select all
./terrafit.py -m 50 -x 4000000 -e 20 work/SRTM-3

I consider -e 20 sufficient for SRTM v4.

II. Airports
This is a tedious step. genapts has to be run for each airport inside your 10x10 degree area. Copy apt.dat.gz from fgdata/Airports to data/airports. Extract it. For a quick look of which airports are in your area, open this file in Google Earth and remove all other markers, also check view -> grid in menu.
http://bbs.keyhole.com/ubb/ubbthreads.p ... 0-ICAO.kmz
It is best to make a list and iterate through each of the elements. Make sure you have all .arr and .fit files inside work/SRTM-3.
Code: Select all
#!/bin/sh
for i in LROP LRBS LRSB
do
   ./genapts --input=./data/airports/apt.dat --work=./work --airport=$i
done


III. Landcover
First open the SRTM files as raster layers in QGis. Edit project properties to set on the fly CRS transformation.

SRTM will serve as a cropping aid, so you can crop the other layers to the size needed.
Edit properties for each SRTM layer and set transparency well above 50% so you can see other layers.
After that open each Corine shapefile and make sure the layer really has elements in your area. Not all layers may have. Use the select tool to select a rectangle around SRTM borders.
Right click on layer and choose Save Selection As.

After cropping all neeeded layers, open them again and reproject on WGS84. Perform this operation from the menu Vector -> Data management tools-> Export to new projection. This step may not be necessary as I could gather from the source code, but it's easier to work with all data on the same projection.

Next you will need to move all files for a layer to its own directory inside data/shapefiles. You can perform mapping between Corine classes this way too. For instance I have put all shapefiles for "Green urban areas" in "DeciduousBroadCover" so urban green areas are filled with trees. You can put several shapefiles inside a data/shapefile/*** directory and they will be processed as the same material. You can get creative with roads too, put Corine polygons for roads, railroads etc. inside GrassCover or DeciduousForest, so OSM roads will have a buffer around them made up of trees or grass.

One problem I have faced with Corine is that "Continuous urban fabric" which is mapped to Urban is simply not present in my area. To avoid having only Towns, you can create a new layer in QGis and open the "Discontinuous urban fabric" layer (112) at the same time. Select edit mode for both layers, go to options and set the program to only show vectors from selected area. Then, select cities you would like to be mapped to Urban by holding down CTRL. After you are done, cut the selected polys and paste them into the new layer. Now you have both Town and Urban.

You can then open roads and railroads from OSM and make sure they are ok and have WSG84 as CRS.
One big issue is that fgfs-construct can't handle big road networks. To eliminate them and reduce scene complexity, open the attribute table for the shapefile. Search for features of type "residential", "track", "path", "pedestrian", "footway", "service", "steps" and other unimportant features. Once selected, delete these lines to simplify the shapefile. The process of deleting these features could take a long type if the number is large. During this time QGis may appear to be frozen.

If you have to correct the coastline, open v0_landmass in QGis and the coastline shapefile from OSM.
In edit mode you can modify landmass shape to conform to OSM data.

When all shapefiles are arranged properly inside the directory structure, you can run ogr-decode and set materials. But first you may have to edit default_priorities.txt and usgsmap.txt in share/TerraGear to add missing landclases. Check with materials.xml and http://wiki.flightgear.org/index.php/CO ... ls_mapping

On normal polygon shapefiles ogr-decode takes the usual parameters:
Code: Select all
./ogr-decode --max-segment 500 --area-type Default work/Landmass data/shapefiles/v0_landmass

For roads and railroads, add a third option: --line-width 10
Streams will be covered by Corine polys so don't worry about that.
When building roads and/or railroads, ogr-decode may terminate with some error because elements do not have the correct coordinates. See SGBucketDiff. If it does, check the work directory, chances are most of the polys in your area have compiled fine and you don't need to worry about it.
Further info: to prevent bogus data from killing ogr-decode, you may patch the following file in two places:
terragear-cs/src/Lib/Polygon/chop-bin.cxx
Lines 222 and 375 replace with:
Code: Select all
if ( (dx > 2880) || (dy > 1440) )
        //throw sg_exception("something is really wrong in split_polygon()!!!!");
   return;

Bogus data will be skipped, ogr-decode will continue.
In order to use OSM road data at maximum potential, you can also use SQL-like queries on the dataset using the --where option.
Eg:
Code: Select all
 ./ogr-decode --max-segment 500 --line-width 12 --area-type Freeway --where "TYPE=trunk" work/Freeway data/shapefiles/road

 ./ogr-decode --max-segment 500 --line-width 14 --area-type Freeway --where "TYPE=motorway" work/Freeway data/shapefiles/road

 ./ogr-decode --max-segment 500 --line-width 10 --area-type Road --where "TYPE=primary" work/Road data/shapefiles/road

 ./ogr-decode --max-segment 500 --line-width 7 --area-type Road --where "TYPE!=primary AND TYPE!=motorway AND TYPE!=trunk" work/Road data/shapefiles/road

These are suggested road and freeway widths for SE Europe. Your local mileage may vary. At this time, Road and Freeway are mapped as the same material in materials.xml but that may change in the future so build your scenery with that in mind.
According to which lines have been tagged in OSM data, you could even extract bridge segments and asign them a specific landclass which can then have the physical model of a bridge (this may require some hardcoding to ensure proper orientation).

In order to proceed with your own custom landclass mappings, you should also edit the following files to insert your own, as pointed out by i4dnf:
install/terragear-cs/share/TerraGear/default_priorities.txt and
install/terragear-cs/share/TerraGear/usgsmap.txt
(local paths may vary according to your own installation)

If all data has been prepared without errors, check inside each directory in work/ and delete folders which do not fit your 10x10 degree tiles.
Time to run fgfs-construct and generate the scenery. This may take a very long time depending on your machine, so let it run over the night.

Code: Select all
./fgfs-construct --work-dir=./../work --output-dir=./../output --lon=XX --lat=XX --xdist=5 --ydist=5 \
 AirportArea AirportObj BarrenCover BuiltUpCover CropGrassCover \
 CropWoodCover DeciduousBroadCover DryCropPastureCover EvergreenBroadCover \
 GolfCourse GrassCover HerbTundraCover IrrCropPastureCover Lake Landmass Marsh MixedCropPastureCover\
 MixedForestCover OpenMining  \
 Road Railroad Rock Sand ShrubCover SRTM-3 Stream Town Urban Freeway


The whole process can take 2-3 full days to complete. Still, it is possible to have the whole Europe in one or two months.
If anyone wishes to add this info to the wiki, please do. I am not a friend of wiki markup language unfortunately.
Last edited by adrian on Mon Dec 06, 2010 2:03 pm, edited 6 times in total.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 3:15 pm

Re: Tutorial: Building Europe scenery

Postby statto » Fri Oct 08, 2010 6:46 pm

This should be added to the wiki. Good work.
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2106
Joined: Fri Jan 25, 2008 10:57 pm

Re: Tutorial: Building Europe scenery

Postby Sealbhach » Fri Oct 08, 2010 7:38 pm

This is very helpful.

.
Sealbhach
 
Posts: 934
Joined: Wed Jun 30, 2010 10:17 am

Re: Tutorial: Building Europe scenery

Postby Gijs » Fri Oct 08, 2010 8:50 pm

Very nice article! Feel free to merge it with the existing wiki article yourself! Else I will do so tomorrow ;)
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9544
Joined: Tue Jul 03, 2007 3:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Tutorial: Building Europe scenery

Postby adrian » Fri Oct 08, 2010 9:47 pm

Thanks. I'm still waiting for my second scenery generation batch to compile. I estimate it will take another 24 hours or so. Gijs, if you have the time, please do the wiki merging yourself. I'm still not very confident on my wiki markup skills to do it myself.

A couple of notes: I think I've traced the srtmchop problem to libtiff (old version). Unfortunately I only run Debian stable and can't confirm this yet.
The process of generating airports is the most cumbersome. The fastest way to do it is to look them up in Google Earth, but you may still have to grep apt.dat for small and military airstrips which are not present inside the KML file. This step can take a lot of improvements. For one, airports in apt.dat could be ordered by country.
The fact that ogr-decode crashes with a "there is something very wrong with a polygon" message is probably due to road line elements from the shapefile which get corrupted when they are converted from OSM xml format to shapefile format. There is no way to avoid it yet, only to use a different datasource which has less or no errors of this sort. Or pester the folks at OSM who write the plugin for conversion.
Using OSM road and railroad data will probably triple your build time, however having accurate roads and railroads contributes to a good VFR experience so it should be worth it. I have had to cut back on OSM data in order to build the second generation quicker. The third generation will probably have complete road and railroad coverage but will take a lot of time to build. Fortunately the last step requires no human interaction.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 3:15 pm

Re: Tutorial: Building Europe scenery

Postby Sealbhach » Thu Oct 14, 2010 9:58 pm

Hi yoyo,

That link for elevation data doesn't seem to work: http://srtm.geog.kcl.ac.uk/portal/srtm41/elevation.kmz

I was thinking too, about the coastline. It would make me crazy to have to hand edit the landmass shapefile using the coastline shapefile as a template. That would be fractally damaging to my OCD. :shock: It would be so nice if we could get that coastline shapefile, which is a line type, convert it to polygon type, fill in the empty space and use it as the default Landmass shape. Surely that could be done somehow? It was bad enough trying to edit the roads shapefile....

I've got my Ireland CORINE scenery almost done, there's still a few areas that came up glitchy, some terrain glitches and other weird things. I'll probably put it up somewhere so people can try it out if they want.

.
Sealbhach
 
Posts: 934
Joined: Wed Jun 30, 2010 10:17 am

Re: Tutorial: Building Europe scenery

Postby statto » Fri Oct 15, 2010 1:48 am

Why the hell do you need the coastline file?
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2106
Joined: Fri Jan 25, 2008 10:57 pm

Re: Tutorial: Building Europe scenery

Postby Sealbhach » Fri Oct 15, 2010 5:14 am

statto wrote:Why the hell do you need the coastline file?


Well, I'm just looking at yoyo's tutorial:

If you have to correct the coastline, open v0_landmass in QGis and the coastline shapefile from OSM.
In edit mode you can modify landmass shape to conform to OSM data.


If you compare them, the coastline shape from cloudmade is really detailed and pretty looking compared to the v0_landmass shape. I have some areas in my scenery where there is a lake from CORINE very near the coast, and the difference in detail is very striking. It would be nice to have the coastline in the same detail.

.
Sealbhach
 
Posts: 934
Joined: Wed Jun 30, 2010 10:17 am

Re: Tutorial: Building Europe scenery

Postby statto » Fri Oct 15, 2010 5:25 am

Ah, you're not using all the corine data then. If you have all of it for Ireland you could merge it into one shapefile and use that as the coastline?
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2106
Joined: Fri Jan 25, 2008 10:57 pm

Re: Tutorial: Building Europe scenery

Postby adrian » Fri Oct 15, 2010 7:51 am

Hi there. First off, statto, if you are saying to ignore v0_landmass by using the switch in fgfs-construct, I don't know if that will produce the result you want.
Second, Corine is not that accurate for coastline, OSM data is superior, especially if you want to have harbors (for future building placement).

Sealbhach: the server appears to be down indeed, but I'm sure it's just temporary. About the coastline, yeah, but I don't know a fast method to do that in QGis?
I only did it myself for a limited area, the rest I just left as v0_landmass. I can see your problem, Ireland being an island :)
I'll try to find out a method to quickly convert that data into polygons. Maybe GRASS can do that. My problem with GRASS is that I'm running Debian stable and can;t easily install the latest version.
The glitches indeed appear in my scenery too sometimes. It could be due to too many polygons or some other bug in fgfs-construct. I remember reading about that problem on a website. I'll check back with the link. Try to reduce the number of faces by terrafitting with a lower -e number and see what happens.
If it's limited to some areas, try to find out what types and number of shapes are in that area. Consider rebuilding only those tiles to save time.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 3:15 pm

Re: Tutorial: Building Europe scenery

Postby Sealbhach » Fri Oct 15, 2010 10:03 am

statto wrote:Ah, you're not using all the corine data then. If you have all of it for Ireland you could merge it into one shapefile and use that as the coastline?


I hadn't thought of that. I could give it a try. I know Qgis can really take a long time if it has a lot of shapes to deal with. It's not crucial, I was just wondering if there was a quick and easy way to use that nice coast shape.
.
Sealbhach
 
Posts: 934
Joined: Wed Jun 30, 2010 10:17 am

Re: Tutorial: Building Europe scenery

Postby adrian » Fri Oct 15, 2010 10:05 am

http://www.cullam.com/flightgear.htm

On this page, if you scroll down you will see the kind of errors that appear sometimes. We should be able to find out in what circumstances these appear, if we gather enough data.
On a brighter note we might have a lot more detailed terrain in 2014:
http://www.bbc.co.uk/news/science-environment-11543990
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 3:15 pm

Re: Tutorial: Building Europe scenery

Postby Sealbhach » Fri Oct 15, 2010 10:22 am

Yep, this is exactly the issue I'm having:

Image

Luckily it's mostly just a few small localized areas. It seems to happen near towns where there are densely packed roads which I haven't removed, along with complex river patterns. I'm just guessing, but I think that might be a factor.

.
Sealbhach
 
Posts: 934
Joined: Wed Jun 30, 2010 10:17 am

Re: Tutorial: Building Europe scenery

Postby adrian » Fri Oct 15, 2010 10:54 am

Same as in my case. I'm thinking very high poly density are bugging the triangulation routines. Try removing the road data and rebuilding just those tiles. If it doesn't do anything, terrafit -e 50. I'll try those myself and report back.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 3:15 pm

Re: Tutorial: Building Europe scenery

Postby Johan G » Sat Oct 16, 2010 8:47 am

yoyo wrote:On a brighter note we might have a lot more detailed terrain in 2014:
http://www.bbc.co.uk/news/science-environment-11543990

I'm afraid not. According to the BBC article
The German space agency owns the hardware; EADS Astrium builds it; and Infoterra GmbH has exclusive rights to commercialise the data.

And after browsing through Infoterra's Single user EULA (PDF) (end user license agreement) i find that the FlightGear community could get the data
END USER means either the person, acting in his own name, or the organization, which is supplied
with the product and accepts this EULA.

But that the elevation data is in no way compatible with the GNU GPL license.
...Digital Elevation Models (DEM) extracted from TerraSAR-X
data (in any form whatsoever, i.e. databases) will be considered as derived products and not as
Derivative Works.

The END USER recognizes INFOTERRA’s right to copyright protection and/or protection against
unauthorized use of the TerraSAR-X data and any derived products, in accordance with the copyright
laws of Germany and applicable international agreements.


Sorry to bring the bad news. :(

As a side note, since the data is not free, I do not believe that the TerraSAR-X elevetion data will be anyway near as much used as the NASA SRTM data. :twisted:
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Some YouTube videos
Johan G
Moderator
 
Posts: 6625
Joined: Fri Aug 06, 2010 6:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 2020.3.4
OS: Windows 10, 64 bit

Next

Return to Scenery

Who is online

Users browsing this forum: No registered users and 4 guests