Board index FlightGear Development Scenery

Breaking down NLCD raster image

Questions and discussion about enhancing and populating the FlightGear world.

Re: Breaking down NLCD raster image

Postby psadro_gm » Wed May 11, 2016 10:10 pm

As long as the area you specify is larger than a single tile, you should be OK.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby wlbragg » Sat May 14, 2016 6:16 pm

A note to users of these tools that sometimes think the tools are not working or the job has froze. I switched my scenery job over to my Linux computer which is not as powerful as my Windows machine and I started the run 3 days ago, it's still chugging along. This job size is an area the size of Kansas.
One of my worries here is we have lots of severe weather this time of year and the power frequently takes a hit and goes out. If even for an instant I'm back to square one. That is one thing about tgconstruct specifically that I don't like. I wish there was a way to restart a run where it left off. If there is, I don't know how.

I'm anxious to see the results of this run. Maybe today!
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Breaking down NLCD raster image

Postby psadro_gm » Sun May 15, 2016 3:31 pm

I've added a stage==x in master so, for instance, stage1 is completed - you can restart with stage 2.

I can look into restarting with incomplete tiles as well. This will need to enforce starting with empty directories, though.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby wlbragg » Sun May 15, 2016 8:43 pm

psadro_gm wrote in Sun May 15, 2016 3:31 pm:you can restart with stage 2

How are the stages laid out?
Stage1
Step 1 : Parse
Step 2 : Clip
Step 3 : Clean
Stage2
Step 4 : Elevation
Stage3
Step 5 : Output

psadro_gm wrote in Sun May 15, 2016 3:31 pm:I can look into restarting with incomplete tiles as well. This will need to enforce starting with empty directories, though.

I think that is a small price to pay for the ability to restart, especially in huge jobs.

Looking at my tgconstruct log I am not sure that I am not stuck?
Code: Select all
"/home/wayne/FlightGear/next/install/terragear/bin/tg-construct" --priorities="/home/wayne/FlightGear/next/install/terragear/share/TerraGear/default_priorities.txt" --usgs-map="/home/wayne/FlightGear/next/install/terragear/share/TerraGear/usgsmap.txt" --work-dir="/home/wayne/FlightGear/scenery/kansas/work" --output-dir="/home/wayne/FlightGear/scenery/kansas/output/Terrain" --ignore-landmass --threads --min-lat=37 --max-lat=40 --min-lon=-102 --max-lon=-94 AirportArea AirportObj Cemetery DeciduousForest Dirt Dirtroad-textured DryCrop EvergreenForest FloodLand Freeway-textured GolfCourse Grassland Greenspace Highway-textured IrrCrop Lake Marsh MixedForest NaturalCrop OpenMining Railroad-textured River Road-textured Sand ScrubCover SRTM-3 Stream SubUrban Town Urban

tg-construct version 2.1.0


Output directory is /home/wayne/FlightGear/scenery/kansas/output/Terrain
Working directory is /home/wayne/FlightGear/scenery/kansas/work
Shared directory is /home/wayne/FlightGear/scenery/kansas/work/Shared
Match directory is
Longitude = -102:-94
Latitude = 37:40
Nudge is 0
Load directory: AirportArea
Load directory: AirportObj
Load directory: Cemetery
Load directory: DeciduousForest
Load directory: Dirt
Load directory: Dirtroad-textured
Load directory: DryCrop
Load directory: EvergreenForest
Load directory: FloodLand
Load directory: Freeway-textured
Load directory: GolfCourse
Load directory: Grassland
Load directory: Greenspace
Load directory: Highway-textured
Load directory: IrrCrop
Load directory: Lake
Load directory: Marsh
Load directory: MixedForest
Load directory: NaturalCrop
Load directory: OpenMining
Load directory: Railroad-textured
Load directory: River
Load directory: Road-textured
Load directory: Sand
Load directory: ScrubCover
Load directory: SRTM-3
Load directory: Stream
Load directory: SubUrban
Load directory: Town
Load directory: Urban
Priorities file is /home/wayne/FlightGear/next/install/terragear/share/TerraGear/default_priorities.txt
Building tile(s) within given bounding box
  construction area spans tile boundaries
1286080 - Construct in w110n30/w102n37 tile 1 of 825 using thread 140333980436224

1286080 - Loading landclass polys

1286080 - Clipping landclass polys

1286080 - Cleaning landclass polys

1286081 - Construct in w110n30/w102n37 tile 2 of 825 using thread 140333980436224

1286081 - Loading landclass polys

1286081 - Clipping landclass polys

1286081 - Cleaning landclass polys

1286082 - Construct in w110n30/w102n37 tile 3 of 825 using thread 140333980436224

1286082 - Loading landclass polys

1286082 - Clipping landclass polys

1286082 - Cleaning landclass polys

1286083 - Construct in w110n30/w102n37 tile 4 of 825 using thread 140333980436224

1286083 - Loading landclass polys

1286083 - Clipping landclass polys

1286083 - Cleaning landclass polys

1302464 - Construct in w110n30/w101n37 tile 5 of 825 using thread 140333980436224

1302464 - Loading landclass polys

1302464 - Clipping landclass polys

1302464 - Cleaning landclass polys

1302465 - Construct in w110n30/w101n37 tile 6 of 825 using thread 140333980436224

1302465 - Loading landclass polys

1302465 - Clipping landclass polys

Should I keep seeing this grow?

What would the tile ID be of the last "construct" (w110n30/w101n37 tile 6 of 825)?
6 give an entirely different area, it wouldn't accept w101n37, how do I use the "By Tile ID" in this case?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Breaking down NLCD raster image

Postby wlbragg » Sun May 15, 2016 8:47 pm

wlbragg wrote in Sun May 15, 2016 8:43 pm:What would the tile ID be of the last "construct" (w110n30/w101n37 tile 6 of 825)?
6 give an entirely different area, it wouldn't accept w101n37, how do I use the "By Tile ID" in this case?

Tell me this isn't the answer?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Breaking down NLCD raster image

Postby psadro_gm » Sun May 15, 2016 10:07 pm

The first number printed is the tile ID - yes the tile index scheme shows how tile ids are assigned.

1302465 - Construct in w110n30/w101n37 tile 6 of 825 using thread 140333980436224
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby wlbragg » Thu May 19, 2016 9:39 pm

Well this has been disappointing.
I have been trying to successfully compile the "stand alone" Woodland layer from the USGS NLCD. It appears there are issues with the shapefiles that tgconstruct doesn't like. I found several errors in the shapes that I corrected, simplified, then re-ran ogrdecode, "successfully", only to fail on tgconstruct. I have been able to do a smaller portion of the state successfully. So I guess I am going to have to break it up into sections until I find the problem areas. Very CPU time consuming.

I had to do this with Tiger line data (stream) before, one error can crash the build. :x
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Breaking down NLCD raster image

Postby psadro_gm » Thu May 19, 2016 10:47 pm

These are the issues I'm trying to prevent in the master branch. Using CGAL to provide exact constructions...

http://www.cgal.org/exact.html
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby alge » Mon May 23, 2016 7:18 pm

Hey Pete, is the master branch usable for mere mortals at the moment? If it is... how do you use it? I notice that ogr-decode is gone, poly-decode seems to have replaced it, and poly-decode is not very good at saying which options it wants or doesn't want. (=
alge
 
Posts: 77
Joined: Thu Aug 30, 2012 1:45 am

Re: Breaking down NLCD raster image

Postby psadro_gm » Mon May 23, 2016 8:43 pm

It won't get you .btg files yet. I'm currently working on tg-construct. - stage 2 - I have another issue with cgal that I'm awaiting an anwer to. - Just submitted yesterday.

ogr-decode has been split into poly-decode for landclass, and vector-decode for vector data.
here is how I run poly-decode:

--area-type Littoral ${WORKBASE} ${DATABASE}/cs_littoral.shp

That's pretty much it. note that max-segment size, etc have been removed because mesh cleanup happens during triangulation, not at decode time.

vector-decode, is a new 'unfinished' beast. It works fine if all road/rail widths are the same. It takes all OSM layers at once and does proper clipping so road intersections take OSM z-order into account.
It also generated texture coordinates for proper intersections, instead of just putting one rectangle 'on top of' another.

The idea for master will be as follows:

1) gdal-chop, hgtfit - same as before
2) genapt to generate airport 2d layout and smoothing region - save as shapefiles
3) vector-decode - generate 2d vector network - and flattening regions
4) poly-decode - generate 2d landclass tiles

5) tg-construct
( stage 1 ) - generate 2d refined triangulation mesh of landclass data
( stage 2 ) - match tile edge node locations, re-refine without generating steiner nodes ( any new nodes must split existing triangle - no nodes on constrained edges )
calc elevations - smooth airport regions, flatten road/rail/lakes, etc
calc face normals
( stage 3 ) - calc point normals with neigboring tiles - use adjacent face normals.
( stage 4 ) - calc curvature - with neighboring point normals, texture - output btg

6) drape tool - drape airport 2d onto smoother terrain - save as .btg
7) drape tool - drape vector data onto flattened terrain - save as .btg

8) lod tool - simplify terrain.btgs and combine for similar vertex count - i.e. merge 4 tiles, and simplify by factor .25 - .btg filesize will be similar as 1 tile, but cover 4x terrain.

So a tile will have the terrain .btg - which is just landclass, a vector data .btg for roads/rail/streams - which can be loaded via LOD, and airport .btgs which no longer fill holes in the terrain, but are draped on top, so modifications to airport layouts will no longer require regenerating the surrounding terrain, if the airport boundary doesn't change. - if the boundary does change, the airport will be draped onto unsmoothed terrain :)
Last edited by psadro_gm on Mon May 23, 2016 8:55 pm, edited 1 time in total.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby wlbragg » Mon May 23, 2016 8:46 pm

Thanks for the update Pete!
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Breaking down NLCD raster image

Postby alge » Tue May 24, 2016 11:10 am

That explains my findings very well. (=

Do you want random patches to clean up option lists and help messages and such, or is better to leave you alone until you're finished?
alge
 
Posts: 77
Joined: Thu Aug 30, 2012 1:45 am

Re: Breaking down NLCD raster image

Postby psadro_gm » Tue May 24, 2016 4:09 pm

Feel free to send merge requests. I am at somewhat of a standstill right now with CGAL. I was looking for a cross platform standard option parsing library / code we could use for all tarragear utils.
This would make it much easier to deal with.
8.50 airport parser, textured roads and streams...
psadro_gm
 
Posts: 751
Joined: Thu Aug 25, 2011 3:23 am
Location: Atlanta, GA USA
IRC name: psadro_*
Version: git
OS: Fedora 21

Re: Breaking down NLCD raster image

Postby bugman » Tue May 24, 2016 4:32 pm

Could you bundle the MIT licensed The Lean Mean C++ Option Parser? I have a lot of Python experience with option parsers, but I don't know what the best for C++ or the full set of terragear utils would be. The documentation looks like it is pretty simple to use, and it seems to be light weight and rather standard for an option parser.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: Breaking down NLCD raster image

Postby legoboyvdlp » Wed Nov 21, 2018 4:38 pm

Can it be confirmed NLCD (e.g. from https://www.mrlc.gov/nlcd11_data.php) is useable for GPL scenery?

Also, if anyone knows any other sources of landcover data for the US that can be used it would help :)


Edit: I got JP2 images from https://earthexplorer.usgs.gov/ - am I ok to trace over them? I assume they are also public domain?
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 9 guests