Board index FlightGear Development Scenery

Is the Scenery Workflow broken?

Questions and discussion about enhancing and populating the FlightGear world.

Is the Scenery Workflow broken?

Postby statto » Mon Mar 21, 2016 5:15 am

I'm writing here because I want to help fix a problem but I'm not sure how.

This weekend I started creating more land cover scenery for FlightGear. I even cleaned vmap0 data from scratch in order to get the proper merge, since the mapserver's down.

However, that's as far as I was able to get.

I downloaded TerraGear on both Linux and Windows and could get neither version compiled and/or working (where I downloaded a pre-compiled version).

Furthermore, Terragear's difficult to use, even with the GUI, and hung after several hours of working on a single piece of land cover (though perhaps it just took forever and I got impatient, but I didn't see any progress being made).

And even if Terragear were easy to use, the current state of FlightGear scenery needing to be rebuilt every so often, and the official distribution channels mean any custom scenery I produce won't be widely distributed/be able to be placed in TerraSync.

Custom scenery aside, with Martin gone, there's currently no way for me to get what I've made synced into the global data set, so it -might- be distributed at some point.

We've made huge strides in the past few years with how FlightGear looks, but the land cover scenery really lags behind outside Europe.

I have several different pieces of fresh ready-to-be compiled land cover. I want them in the next world scenery update/in Terrasync, and as quickly as possible. I know we don't really know what's going on at the moment, but I'm wondering what I can do to make this a reality.
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2100
Joined: Fri Jan 25, 2008 9:57 pm

Re: Is the Scenery Workflow broken?

Postby wlbragg » Mon Mar 21, 2016 7:03 am

statto, what parts of the world did you do? I would really like to get a copy of the "original" vmap0 data that you have.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4622
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: Is the Scenery Workflow broken?

Postby pommesschranke » Mon Mar 21, 2016 9:04 am

I agree that terragear is too complicated to obtain.
I used download_and_compile.sh on ubuntu.
http://wiki.flightgear.org/Scripted_Com ... ian/Ubuntu

Did you try the ubuntu PPA package?
https://launchpad.net/~saiarcot895/+arc ... tgear-edge


use terragear branch scenery/ws2.0 and GDAL 2.0


viewtopic.php?f=45&t=29119&p=279887&hilit=terragear+branch#p279887
viewtopic.php?f=5&t=28383&p=272003&hilit=terragear+branch#p272003

I found some old v0 shapefiles here (=scenery 1.0?)
ftp://ftp.ihg.uni-duisburg.de/FlightGear/TGShapes/

The Workflow for getting terrain & airport layouts into terrsync is not existing - We have to design it, together with Torsten Dreyer. Here's a quote from the devel mailing list:


I understad that goal, but it has led to 3 years of stagnation of
airport layouts. The gateway(our upstream) is full of new and improved
airport layouts. WED artists of the FG community are sad to see that
their work is not integrated in the main FG scenery.


Agreed, lets work on a procedure how to integrate that into our toolchain.


In some cases we can just update the ICAO.btg.gz without changing
the terrain around it.


OK, tell me what those cases are and it should be worth a try to replace the terrain chunk.
It will most likely require a recalculation of object elevations for that area.

I think it would be nice to integrate D-ECHO's beautiful islands
and my OMDB to make it available for all users without the need for
extra custom scenery download and folders.


Those islands should be easy, indeed.


I also don't expect a significant mismatch of tiles in Asia (India,
China) when we all use the v0 shapefiles and viewfinderpanorams's
elevation data.

I know that Martin Spott wanted to avoid those little edges at all cost,
but what is the opinion of the other developers & users ?

* tolerate some tile edges far away from the airport to get a nicer,
up-to-date airport layout ?

yes or no ?


No (my vote). Not everybody is flying traffic patterns. How does it look from high altitude?

I am not agains a new scenery strategy but I hate to see a patchwork-style planet one day.

The vision of having a solid tool chain generating a world scenery from a detailed and common database is what I am after.
Lets avoid gaps, elevation steps, overlapping and incompatible scenery tiles and work together on a one-fits-all scenery.

Until we have that, here is my suggestion:
Somebody provides self-contained and tested chunks of Terrain to be merged into Terrasync. That "somebody" is responsible for the usability of that terrain chunk.
By "provide" I mean
- make it available by rsync, tarball or anything else to be fetched, unpacked into terrasync without human interaction.
- Work out a method to discover new or outdated chunks
- Work out a method how to tell scenemodels-db which objects need new elevations
- Somehow "sign" the chunk so we have somebody who gurantees GPL compatibility
- Work out a method how to avoid mutual exclusive changes (one person updating EDFE, another improving EDDF [both share a tile])
Not everything need to be ready in detail but I'd like at least have an idea how this should work.

What I will not accept is: Follow some dropbox / gg-drive links, unpack an archive, guess what it might be, remove all side effects, do some hours of test flying, resetting object elevations by guessing the area etc. etc.

And always remember - at one day we will have Martin's landcover db back and hopefully a working toolchain. Whenever that happens, we build a new "world scenery". What happens to all those manual updates? I fear they are lost.


Torsten
pommesschranke
 
Posts: 1104
Joined: Sat Apr 27, 2013 7:58 pm
Location: EDLM & LJCE
Callsign: d-laser
IRC name: laserman
Version: git
OS: Linux Lubuntu 18.04

Re: Is the Scenery Workflow broken?

Postby psadro_gm » Mon Mar 21, 2016 1:51 pm

Yes, terragear is difficult to use. I had made a couple brief attempts to make it more user friendly, but the highest priority was always robustness and fixing issues Martin happened upon.

If you download and compile, you'll need to use the Scenery/ws2.0 branch. That should be stable, as it was used to generate the entire last global scenery (without crashing)

A brief status update on what I am currently doing on the master branch:

1) It's been a year since I started working on LOD ( using Mathias' .SPT directory hierarchy ). This exposed a major weakness in the toolchain used for ws2.0 - It triangulated landclass polys individually instead of whole tile at a time. This was necessary, as triangulating the whole tile with OSM roads cut-in was very unstable. I couldn't get any library to succeed for more than a few hundred tiles. But when trying to simplify these tiles, large holes get generated as the mesh is non manifold.

2) I 'started from scratch' with a couple assumptions:
- all vector data will be draped over the terrain - although we may still cut in some smoothing polys
- airports will be draped over a smoothed terrain within the airport boundaries ( changing airport layout will not require new tg-construct run unless boundary is changed )
- tile mesh must be manifold
- all intermediate files can be visualized with GIS tools ( request from martin )

I estimate I'm about 1 month away from having tg-construct generating a base mesh.

Having a mesh to work with ( in CGAL ) gives me a couple feature for 'free'.
- face and point normals can be generated from CGAL
- local differentials can also be generated ( request from Thorsten Renk - terrain curvature )

I have a large 12x10 degree section of Europe shapefiles from mapserver. Once I have a scenery, I'll put chunks of it on dropbox to experiment with.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 750
Joined: Thu Aug 25, 2011 2:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Is the Scenery Workflow broken?

Postby statto » Tue Mar 22, 2016 4:54 pm

wlbragg wrote in Mon Mar 21, 2016 7:03 am:statto, what parts of the world did you do? I would really like to get a copy of the "original" vmap0 data that you have.


I've done Australia, because that's the part of the world where I'm currently working on scenery. You can download the original version online if you poke around, but note I had to do pre-processing in order to get it to the v0_ files we know and love.

pommesschranke wrote in Mon Mar 21, 2016 9:04 am:I agree that terragear is too complicated to obtain.
I used download_and_compile.sh on ubuntu.
http://wiki.flightgear.org/Scripted_Com ... ian/Ubuntu


Thank you for the link. I have SimGear successfully compiled now, but TerraGear fails on (I think) a radians conversion line.

psadro_gm wrote in Mon Mar 21, 2016 1:51 pm:I estimate I'm about 1 month away from having tg-construct generating a base mesh.


Very cool.

Still, the one big question in my mind:

I have land cover, and I have a merge process. How can I get what I do into Terrasync?
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2100
Joined: Fri Jan 25, 2008 9:57 pm

Re: Is the Scenery Workflow broken?

Postby wlbragg » Tue Mar 22, 2016 7:09 pm

I think I am confusing the name vmap0 data with cs_data. I think they are one and the same. I already have the entire world of vmap0 data used in ws2.0 gen. and it is available online with a link on one or more threads in this forum. Thanks anyway!
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4622
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: Is the Scenery Workflow broken?

Postby statto » Tue Mar 22, 2016 9:18 pm

wlbragg wrote in Tue Mar 22, 2016 7:09 pm:I think I am confusing the name vmap0 data with cs_data. I think they are one and the same. I already have the entire world of vmap0 data used in ws2.0 gen. and it is available online with a link on one or more threads in this forum. Thanks anyway!


vmap0 data is the "base" data whereas cs_data is the additional data. The vmap0 data however had been cleaned before creating the cs_data layer, so if I'm downloading the vmap0 files from scratch, I have to do the cleaning again. I don't have a version of vmap0 data that has been cleaned - you might, if you downloaded the one used in ws2.0 gen.

I -do- have raw custom scenery in bits and pieces - random degree chunks of the western USA (looks like just Colorado and San Francisco), degree chunks of Florida, degree chunks of southeastern Alaska, Massachusetts and Vermont from seven years ago now, Puerto Rico, and a couple parts of Australia.
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2100
Joined: Fri Jan 25, 2008 9:57 pm

Re: Is the Scenery Workflow broken?

Postby psadro_gm » Wed Mar 23, 2016 2:06 pm

Hi Statto,

Even I am unsure what the future scenery will be built from. Right now, Torsten is heading up the scenemodels part of the scenery. I am unsure if he has plans on taking on the shapefiles and terrain generation side of scenery as well.

I created an account on AWS, but I am unsure if I will pursue this much further. Truth of it is, we relied heavily on Martin for the scenery data, and now we'll need to find an alternative :(
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 750
Joined: Thu Aug 25, 2011 2:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Is the Scenery Workflow broken?

Postby Torsten » Wed Mar 23, 2016 2:45 pm

Hi,

after running the scenemodels for a while, I know that there is not much magic behind running a database server for shapefiles.
The machine we use for running FlightGear at FSweekend has enough storage and computing power to build a world scenery.
We seem to have a better and working toolchain in a not so distant future. Looks like the only thing missing is the huge dataset that Martin has somewhere on his private hardrive.

Given that all that was generated from public data, would it be worth the effort to restart from scratch and create a new landcover database?

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 632
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Is the Scenery Workflow broken?

Postby erik » Wed Mar 23, 2016 3:07 pm

Why not just ask Martin for it? I'm sure he is willing to provide it.

Erik
erik
 
Posts: 1516
Joined: Thu Nov 01, 2007 1:41 pm

Re: Is the Scenery Workflow broken?

Postby Torsten » Wed Mar 23, 2016 3:25 pm

Believe me, I did ask more than once.
He does not seem to be willing to provide the data.

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 632
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Is the Scenery Workflow broken?

Postby wkitty42 » Wed Mar 23, 2016 4:17 pm

if martin is unwilling to provide the data then building a new land cover database is the only other option, isn't it? if all the data came from public sources, then it should be easy enough to gather it all again... granted, it seems like it will be a huge amount of wheel reinventing but what other alternative is there??
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5293
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Is the Scenery Workflow broken?

Postby psadro_gm » Wed Mar 23, 2016 5:01 pm

I believe Martin did some processing on the corrine data - One of his biggest achievements was the NLCD vectorization. The last thing I know he was working on was combining landclass data with OSM coastlines ( filling in the gaps with appropriate landclasses )

These tasks should not be considered trivial by any means.

He has several of these scripts in the terragear repo under gitscripts/
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 750
Joined: Thu Aug 25, 2011 2:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Is the Scenery Workflow broken?

Postby wlbragg » Wed Mar 23, 2016 6:07 pm

One of his biggest achievements was the NLCD vectorization

Absolutely that was a great achievement. However, part of that data is extremely flawed, the water data, it was included as part of the rounded data and you just can't do that and have roads and rails from OSM look correct when they are near any lake shoreline or river. Some of the roads (maybe many) ended up in the water as there was no control at to how the water edge was rounded (as far as I know). I am going on memory of scenery I built with that data compared to the same NLCD data I pulled and used as-is to generate the same area.
In that same rounded NLCD data there was also a compromise made that I feel was unnecessary on the deciduous land class. In this part of the world all the creeks and many rivers are bordered with deciduous forest and are major land marks, most of that was not generated, only the large forests were included.
As far as the rest of the land classes in that data, while it was miles ahead of what we have now in North America, it was still rounded enough to change borders of land classes beyond reality and that was also disappointing to me. I would attempt to tone down the smoothing to lessen the distortion.

So in short, as far as the rounded NLCD data for North America is concerned, my vote would be to learn how to use that rounding script and modify it to not touch the water layer and do it over.

I've put together Kansas and Ohio and know enough about what it takes to put this data together, it would be a job to redo the world.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4622
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: Is the Scenery Workflow broken?

Postby statto » Wed Mar 23, 2016 6:21 pm

Torsten wrote in Wed Mar 23, 2016 2:45 pm:Hi,
Given that all that was generated from public data, would it be worth the effort to restart from scratch and create a new landcover database?


Here are the sources I know of which currently constitute the world scenery.

I know wlbragg has vmap0 data for the entire world and I would probably be able to get just the vmap0 shapefiles up and cleaned for use in FlightGear in a weekend's worth of work. It didn't take very long for Australia. I know Martin did some post-work on the vmap0 files to get them working properly and I believe I've already accomplished something similar with the Australia files.

I don't have a source for Corine data but I should be able to find one and clean it.

Same with OpenStreetMap - I don't have a worldwide source of data, nor know how to update that source, but again that should be easily found.

Now for custom scenery:

I actually wrote the NLCD vectorization script. Martin improved it and got it running on the server, and for getting it running successfully. It can easily be run again, though note it took about a month to complete running on a dedicated server.

I have a very small amount of custom scenery floating around on my web server and hard drive which I've listed above. This MAY include St. Maarten (I'm checking now; I created these files, when, 2008? wow) and surroundings but does NOT include Hawaii, nor does it include some other areas of custom scenery I've made such as Rio de Janeiro - I simply don't have those shapefiles anymore. They are, however, part of the cs_ layers, if anyone has this file (or even the cs_ change file, which exists somewhere).

I think other users improved Dubai (I think Gijs?) and Singapore as well.

I'm happy to take the lead with this. Just note while I have experience setting things up on server, I'm not very good with my server-side skills, which is why I've been developing geographies and not taking a more technical role.

Thanks
John
Custom Scenery available from http://www.stattosoftware.com/flightgear
statto
 
Posts: 2100
Joined: Fri Jan 25, 2008 9:57 pm

Next

Return to Scenery

Who is online

Users browsing this forum: tikibar and 0 guests