Board index FlightGear Development Scenery

WS3.0 and Docker requirement

Questions and discussion about enhancing and populating the FlightGear world.

WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 1:16 am

I was getting started with setting up a ws3.0 scenery development environment and step 2 on the wiki requires a dockerhub account. Why do we need to make yet another account to build scenery?
I assumed the docker portion was to simplify setting up the scenery build environment. Is having this account a necessity if I want to generate ws3.0 scenery?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 1:46 am

Also Docker appears to require yet another application called Hyper-V in order to run on my version of Windows. Is all this really necessary?

Are there any instructions for setting this up manually, piece by piece? Docker would be fine if it didn't require as much setup as the container I want to get with it.

The deal with Hyper-V was Docker failed to install and said I needed Server to be running in services. Well it is. So then another search said I also need this Hyper-V?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 2:23 am

Well sorry for some of the noise, it turns out I needed some services running that were not, somehow they got shutdown. All better and was able to install Docker. Maybe now this will go a little smoother.

For the next person down the road that has this issue it was a cascading services failure starting at SMB 1.0 File Sharing Service.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 3:09 am

If the instructions in the wiki https://wiki.flightgear.org/Howto:Create_WS3.0_terrain are meant for the casual scenery developer, they are missing details we need.
I am stuck at...

To generate terrain you need to run the Virtual Planet Builder tool within the container.

Firstly, get the container running from the directory containing your data and output directories:

docker run --rm --mount "type=bind,source=`pwd`/data,target=/home/flightgear/data,readonly" --mount "type=bind,source=`pwd`/output,target=/home/flightgear/output" -it flightgear/ws30-vbp-generator:latest /bin/bash
You should now find yourself in a bash shell within your container. You should see data and output directories with are linked to the directories you created earlier:


Firstly, get the container running from the directory containing your data and output directories:

Where are these data folders, I recognize the structure from ws2.0 but not how they apply to the docker image nor if the data in them are similar. I did notice it talks about the data farther down the wiki, but shouldn't that come first, not last? Am I supposed to surmise that I need to skip around the documentation in order to fill in any gaps at the start?

You should now find yourself in a bash shell within your container.

Nope, the run command failed.

Code: Select all
docker run --rm --mount "type=bind,source=`pwd`/data,target=/home/flightgear/data,readonly" --mount "type=bind,source=`pwd`/output,target=/home/flightgear/output" -it flightgear/ws30-vbp-generator:latest /bin/bash
docker: Error response from daemon: invalid mount config for type "bind": invalid mount path: '`pwd`/data' mount path must be absolute.
See 'docker run --help'.


You should see data and output directories with are linked to the directories you created earlier

What directories did I create earlier?

I think there is a big piece missing at this step. Maybe we are assuming the user has a certain level of knowledge at this point that we don't. I am not proficient in Linux, but I do recognize the data structure were attempting to use here. What I have absolutely no knowledge about is how to get around in the docker image, ie: make directories, run commands?
What and where are source=`pwd`/data and target=/home/flightgear/data supposed to be, how do we make or access them in docker?

I don't mean for this to sound negative in any way, I truly am totally stumped trying to follow this and I have successfully built 2.0 scenery. I feel like I am missing some very important information and/or steps. It could be I am just dense, I would be the first to admit this.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 3:25 am

Is this even supposed to work in Windows?
After reading this I am not certain at all...
https://wiki.flightgear.org/Virtual_Planet_Builder#Running_VPB
Specifically what are these 2 different sources?
There is now[1] a VPB docker
This is a link to a Wikipedia article container for Linux: https://hub.docker.com/r/flightgear/ws30-vbp-generator and https://github.com/fdalvi/flightgear-ws30-vbp-generator
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby Fahim Dalvi » Wed Jan 25, 2023 6:33 am

Hi wlbragg,

wlbragg wrote in Wed Jan 25, 2023 1:16 am:...Why do we need to make yet another account to build scenery?...


I believe this should not be necessary, but just helps by increasing the daily download rate limit for your machine; I'll test this in the coming days and mark it as optional if this is true.

wlbragg wrote in Wed Jan 25, 2023 1:46 am:Also Docker appears to require yet another application called Hyper-V in order to run on my version of Windows. Is all this really necessary?


Unfortunately I'm not well versed with docker on windows, no so idea about this.

wlbragg wrote in Wed Jan 25, 2023 1:46 am:Are there any instructions for setting this up manually, piece by piece?


Yes, I think you should be able to build VPB by following its compile instructions in its source, just like OSG, FG etc found here: https://github.com/openscenegraph/VirtualPlanetBuilder; the docker image is just a convenience with some extra stuff installed like gdal that helps manage the scenery files.

wlbragg wrote in Wed Jan 25, 2023 3:09 am:Where are these data folders, I recognize the structure from ws2.0 but not how they apply to the docker image nor if the data in them are similar


These are the folders created in the top most subsection "Getting the base data". You will have a directory `data` (which will have the dem and the tiff if I remember correctly) and a directory `output` (which will be empty)

wlbragg wrote in Wed Jan 25, 2023 3:09 am:Nope, the run command failed.


It seems like the `pwd` is not expanding to the current directory (as it does on linux etc. Can you try replacing the mounts with

Code: Select all
--mount "type=bind,source=<full-path-to-data-directory-created-above>,target=/home/flightgear/data,readonly"
--mount "type=bind,source=<full-path-to-output-directory-created-above>,target=/home/flightgear/output"


Hopefully you should be able to launch the container then!

wlbragg wrote in Wed Jan 25, 2023 3:09 am:I don't mean for this to sound negative in any way, I truly am totally stumped trying to follow this and I have successfully built 2.0 scenery.


Its great, hopefully you get it running and we can improve the documentation for future scenery devs! This whole pipeline is quite new and a handful of people have built scenery using this, so docs definitely need some ironing to work across the board.

Best,
User avatar
Fahim Dalvi
 
Posts: 1339
Joined: Sun Oct 14, 2007 1:08 pm
Location: Qatar
Callsign: Fahim
Version: 2020.3.11
OS: macOS

Re: WS3.0 and Docker requirement

Postby merspieler » Wed Jan 25, 2023 10:48 am

Yes, no account needed... also, I never ran into download limits...
Be the change you wish to see in the world, be an ally to all! Love is the answer.

If everything is going against you, keep in mind that airplanes take off against the wind, not with it.

Join the official matrix space
merspieler
 
Posts: 1771
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: NixOS

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 5:05 pm

Thank you Fahim,
These are the folders created in the top most subsection "Getting the base data". You will have a directory `data` (which will have the dem and the tiff if I remember correctly) and a directory `output` (which will be empty)


I totally missed this...
Put these files into a data directory and create an output directory next to it. This is where the scenery will be generated..

Now it makes sense. I may not have to change the paths to the data and output as suggested once I actually make the folders.


Are the source data tiff and dem files in separate subfolders?
How well does this new ws3.0 handle overlapping scenery, say one tile exists in two different scenery builds. Is that just handled in FlightGear as to which chunk is rendered?
Has that annoying boundary offset been eliminated?

Is there a limit as to how small a section you can build, could you build one tile at a time?
READ THE INSTRUCTIONS FARTHER DOWN

EDIT additional question...
Is the NLCD handled the same as it was in ws2.0 (shapefiles that fit the same area as the dem data)?
In other words, can we use the same shapefiles we used for ws2.0?
Last edited by wlbragg on Wed Jan 25, 2023 6:24 pm, edited 1 time in total.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 6:19 pm

Still didn't run

invalid argument "type=bind,source=D:/Scenery/ws3.0/Kansas/output,target=docker run --rm --mount type=bind,source=D:/Scenery/ws3.0/Kansas/data,target=/home/flightgear/data,readonly --mount type=bind,source=D:/Scenery/ws3.0/Kansas/output,target=/home/flightgear/output -it flightgear/ws30-vbp-generator:latest /bin/bashoutput" for "--mount" flag: unexpected key 'readonly --mount type' in 'readonly --mount type=bind'


I removed the readonly args and it worked, it left me in a linux type environment shell, I assume this is where I want ot be?
flightgear@ac4bec3ce423:~$ ls
data 'output -it flightgear' scripts

Also, I take it this Docker image is not local but being executed through Docker Desktop on a remote? Or how is that working, I don't see any new local files other than data structure I created?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 6:51 pm

Do we know if this build process is going to remain consistent in the near future?
If so I need to process the NLCD per the instructions and will consider documenting the steps (with pictures) in the wiki if I am successful in getting this working.

Learning ws2.0 was being left behind was the final nail in the scenery coffin for me.

It's very exciting to be able to start back up in scenery development using something that should be around for awhile. Scenery development was my first desire in FG. WS3.0 looks to be, if not right now at least in the near future, fairly robust and there are some much needed blending and lod features.

After using the newest MSFS for awhile now, I am seeing where we need to make some new effort and really hope we can catch up and at least mach in small chunks the resolution of scenery they seem to produce on such a mass scale.

I found with ws2.0 we could get close to matching that resolution. With ws3.0, OSM efforts and the addition of photo scenery, I feel we can match and maybe even exceed their scenery product resolution.

Thanks to everyone involved in all these disciplines. It may be slow, but it continues to make a difference.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby merspieler » Wed Jan 25, 2023 6:58 pm

on which remote machine do you think this would run? it's all on your own.
the --mounts maps your local structure into the container and the container it self.... is somewhere maybe %appdata% or something like that. but you don't need to care about that, docker handles that for you.

so once you're in the container, you can run ls on your data dir to check if you've got the right input data
you can also run a simple command like "touch <path/to/file>" into the output dir to verify the mapping works by checking it has also been created on the host
then you basically can start running the commands from the wiki

Details may change (such like stuart saying that one should now use png instead of the dds format) but that's just a matter of adjusting the command line arguments, the general process should remain roughly the same
Be the change you wish to see in the world, be an ally to all! Love is the answer.

If everything is going against you, keep in mind that airplanes take off against the wind, not with it.

Join the official matrix space
merspieler
 
Posts: 1771
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: NixOS

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 7:51 pm

on which remote machine do you think this would run?

No clue, that's why I asked. Maybe out of the original image on DockerHub. I didn't see it anywhere in my local files. I had no clue how Docker works. appdata makes sense, although I never cared for applications storing everything in appdata, including FG. I would prefer an application like Docker to at least allow you to store your containers and images where you want to store them. Maybe it does if your willing to spend a day reading through the documentation. The issue with applications defaulting to root appdata is suddenly you find your system files drive full of data files, instead of system files. In Windows OS the average user has no clue the User or Window directory is growing massively nor how to navigate it.

I surmised that Docker was allowing me to run a Linux type environment in windows. This whole Docker and Docker Hub setup appears to be a combination of a pseudo virtual machine and version control system.

Just another application to spend hours learning how to get the most out of it. I guess at my age, I have done this so many times with so many applications, it's just getting old.

Anyway, neither here nor there, doesn't matter. Thanks for info.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 8:23 pm

Next...

Re-classify the raster data using "Reclassify by table

My version of QGIS does not appear to have this process in the menu under "Raster"?
Does this apply anymore to ws3.0?

or

In other words, can we use the same shapefiles we used for ws2.0?

Is the landcover data generated the same way as it was for ws2.0?


I guess we were using vector data, not raster. We actually converted the raster to vector.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 8:54 pm

Re-classify the raster data using "Reclassify by table

It's in Processing/T oolbox
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: WS3.0 and Docker requirement

Postby wlbragg » Wed Jan 25, 2023 9:20 pm

Hum,

What I get out of the existing documentation is that we are opening the .tiff file and them applying a CRS of EPSG:4326.
Then we need to...
Re-classify the raster data using "Reclassify by table"

But there is missing information we need to know, how to use this tool. It is calling for a "Reclassification table", what is that and how do we supply it?
The image is a pallet, are we trying to tell it to separate it by color, if so how?
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
wlbragg
 
Posts: 6766
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Next

Return to Scenery

Who is online

Users browsing this forum: No registered users and 1 guest