WS3.0 and Docker requirement

[wlbragg]

I was getting started with setting up a ws3.0 scenery development environment and step 2 on the wiki requires a dockerhub account. Why do we need to make yet another account to build scenery?
I assumed the docker portion was to simplify setting up the scenery build environment. Is having this account a necessity if I want to generate ws3.0 scenery?

[wlbragg]

Also Docker appears to require yet another application called Hyper-V in order to run on my version of Windows. Is all this really necessary?

Are there any instructions for setting this up manually, piece by piece? Docker would be fine if it didn't require as much setup as the container I want to get with it.

The deal with Hyper-V was Docker failed to install and said I needed Server to be running in services. Well it is. So then another search said I also need this Hyper-V?

[wlbragg]

Well sorry for some of the noise, it turns out I needed some services running that were not, somehow they got shutdown. All better and was able to install Docker. Maybe now this will go a little smoother.

For the next person down the road that has this issue it was a cascading services failure starting at SMB 1.0 File Sharing Service.

[wlbragg]

If the instructions in the wiki https://wiki.flightgear.org/Howto:Create_WS3.0_terrain are meant for the casual scenery developer, they are missing details we need.
I am stuck at...

To generate terrain you need to run the Virtual Planet Builder tool within the container.

Firstly, get the container running from the directory containing your data and output directories:

docker run --rm --mount "type=bind,source=`pwd`/data,target=/home/flightgear/data,readonly" --mount "type=bind,source=`pwd`/output,target=/home/flightgear/output" -it flightgear/ws30-vbp-generator:latest /bin/bash
You should now find yourself in a bash shell within your container. You should see data and output directories with are linked to the directories you created earlier:

Firstly, get the container running from the directory containing your data and output directories:

Where are these data folders, I recognize the structure from ws2.0 but not how they apply to the docker image nor if the data in them are similar. I did notice it talks about the data farther down the wiki, but shouldn't that come first, not last? Am I supposed to surmise that I need to skip around the documentation in order to fill in any gaps at the start?

You should now find yourself in a bash shell within your container.

Nope, the run command failed.

Code: Select all

docker run --rm --mount "type=bind,source=`pwd`/data,target=/home/flightgear/data,readonly" --mount "type=bind,source=`pwd`/output,target=/home/flightgear/output" -it flightgear/ws30-vbp-generator:latest /bin/bash
docker: Error response from daemon: invalid mount config for type "bind": invalid mount path: '`pwd`/data' mount path must be absolute.
See 'docker run --help'.

You should see data and output directories with are linked to the directories you created earlier

What directories did I create earlier?

I think there is a big piece missing at this step. Maybe we are assuming the user has a certain level of knowledge at this point that we don't. I am not proficient in Linux, but I do recognize the data structure were attempting to use here. What I have absolutely no knowledge about is how to get around in the docker image, ie: make directories, run commands?
What and where are source=`pwd`/data and target=/home/flightgear/data supposed to be, how do we make or access them in docker?

I don't mean for this to sound negative in any way, I truly am totally stumped trying to follow this and I have successfully built 2.0 scenery. I feel like I am missing some very important information and/or steps. It could be I am just dense, I would be the first to admit this.

[wlbragg]

Is this even supposed to work in Windows?
After reading this I am not certain at all...
https://wiki.flightgear.org/Virtual_Planet_Builder#Running_VPB
Specifically what are these 2 different sources?
There is now[1] a VPB docker
This is a link to a Wikipedia article container for Linux: https://hub.docker.com/r/flightgear/ws30-vbp-generator and https://github.com/fdalvi/flightgear-ws30-vbp-generator

[Fahim Dalvi]

Hi wlbragg,

...Why do we need to make yet another account to build scenery?...

I believe this should not be necessary, but just helps by increasing the daily download rate limit for your machine; I'll test this in the coming days and mark it as optional if this is true.

Also Docker appears to require yet another application called Hyper-V in order to run on my version of Windows. Is all this really necessary?

Unfortunately I'm not well versed with docker on windows, no so idea about this.

Are there any instructions for setting this up manually, piece by piece?

Yes, I think you should be able to build VPB by following its compile instructions in its source, just like OSG, FG etc found here: https://github.com/openscenegraph/VirtualPlanetBuilder; the docker image is just a convenience with some extra stuff installed like gdal that helps manage the scenery files.

Where are these data folders, I recognize the structure from ws2.0 but not how they apply to the docker image nor if the data in them are similar

These are the folders created in the top most subsection "Getting the base data". You will have a directory `data` (which will have the dem and the tiff if I remember correctly) and a directory `output` (which will be empty)

Nope, the run command failed.

It seems like the `pwd` is not expanding to the current directory (as it does on linux etc. Can you try replacing the mounts with

Code: Select all

--mount "type=bind,source=<full-path-to-data-directory-created-above>,target=/home/flightgear/data,readonly"
--mount "type=bind,source=<full-path-to-output-directory-created-above>,target=/home/flightgear/output"

Hopefully you should be able to launch the container then!

I don't mean for this to sound negative in any way, I truly am totally stumped trying to follow this and I have successfully built 2.0 scenery.

Its great, hopefully you get it running and we can improve the documentation for future scenery devs! This whole pipeline is quite new and a handful of people have built scenery using this, so docs definitely need some ironing to work across the board.

Best,

[merspieler]

Yes, no account needed... also, I never ran into download limits...

[wlbragg]

Thank you Fahim,

These are the folders created in the top most subsection "Getting the base data". You will have a directory `data` (which will have the dem and the tiff if I remember correctly) and a directory `output` (which will be empty)

I totally missed this...

Put these files into a data directory and create an output directory next to it. This is where the scenery will be generated..

Now it makes sense. I may not have to change the paths to the data and output as suggested once I actually make the folders.


Are the source data tiff and dem files in separate subfolders?
How well does this new ws3.0 handle overlapping scenery, say one tile exists in two different scenery builds. Is that just handled in FlightGear as to which chunk is rendered?
Has that annoying boundary offset been eliminated?

Is there a limit as to how small a section you can build, could you build one tile at a time?
READ THE INSTRUCTIONS FARTHER DOWN

EDIT additional question...
Is the NLCD handled the same as it was in ws2.0 (shapefiles that fit the same area as the dem data)?
In other words, can we use the same shapefiles we used for ws2.0?

[wlbragg]

Still didn't run

invalid argument "type=bind,source=D:/Scenery/ws3.0/Kansas/output,target=docker run --rm --mount type=bind,source=D:/Scenery/ws3.0/Kansas/data,target=/home/flightgear/data,readonly --mount type=bind,source=D:/Scenery/ws3.0/Kansas/output,target=/home/flightgear/output -it flightgear/ws30-vbp-generator:latest /bin/bashoutput" for "--mount" flag: unexpected key 'readonly --mount type' in 'readonly --mount type=bind'

I removed the readonly args and it worked, it left me in a linux type environment shell, I assume this is where I want ot be?
flightgear@ac4bec3ce423:~$ ls
data 'output -it flightgear' scripts

Also, I take it this Docker image is not local but being executed through Docker Desktop on a remote? Or how is that working, I don't see any new local files other than data structure I created?

[wlbragg]

Do we know if this build process is going to remain consistent in the near future?
If so I need to process the NLCD per the instructions and will consider documenting the steps (with pictures) in the wiki if I am successful in getting this working.

Learning ws2.0 was being left behind was the final nail in the scenery coffin for me.

It's very exciting to be able to start back up in scenery development using something that should be around for awhile. Scenery development was my first desire in FG. WS3.0 looks to be, if not right now at least in the near future, fairly robust and there are some much needed blending and lod features.

After using the newest MSFS for awhile now, I am seeing where we need to make some new effort and really hope we can catch up and at least mach in small chunks the resolution of scenery they seem to produce on such a mass scale.

I found with ws2.0 we could get close to matching that resolution. With ws3.0, OSM efforts and the addition of photo scenery, I feel we can match and maybe even exceed their scenery product resolution.

Thanks to everyone involved in all these disciplines. It may be slow, but it continues to make a difference.

[merspieler]

on which remote machine do you think this would run? it's all on your own.
the --mounts maps your local structure into the container and the container it self.... is somewhere maybe %appdata% or something like that. but you don't need to care about that, docker handles that for you.

so once you're in the container, you can run ls on your data dir to check if you've got the right input data
you can also run a simple command like "touch <path/to/file>" into the output dir to verify the mapping works by checking it has also been created on the host
then you basically can start running the commands from the wiki

Details may change (such like stuart saying that one should now use png instead of the dds format) but that's just a matter of adjusting the command line arguments, the general process should remain roughly the same

[wlbragg]

on which remote machine do you think this would run?

No clue, that's why I asked. Maybe out of the original image on DockerHub. I didn't see it anywhere in my local files. I had no clue how Docker works. appdata makes sense, although I never cared for applications storing everything in appdata, including FG. I would prefer an application like Docker to at least allow you to store your containers and images where you want to store them. Maybe it does if your willing to spend a day reading through the documentation. The issue with applications defaulting to root appdata is suddenly you find your system files drive full of data files, instead of system files. In Windows OS the average user has no clue the User or Window directory is growing massively nor how to navigate it.

I surmised that Docker was allowing me to run a Linux type environment in windows. This whole Docker and Docker Hub setup appears to be a combination of a pseudo virtual machine and version control system.

Just another application to spend hours learning how to get the most out of it. I guess at my age, I have done this so many times with so many applications, it's just getting old.

Anyway, neither here nor there, doesn't matter. Thanks for info.

[wlbragg]

Next...

Re-classify the raster data using "Reclassify by table

My version of QGIS does not appear to have this process in the menu under "Raster"?
Does this apply anymore to ws3.0?

or

In other words, can we use the same shapefiles we used for ws2.0?

Is the landcover data generated the same way as it was for ws2.0?

I guess we were using vector data, not raster. We actually converted the raster to vector.

[wlbragg]

Re-classify the raster data using "Reclassify by table

It's in Processing/T oolbox

[wlbragg]

Hum,

What I get out of the existing documentation is that we are opening the .tiff file and them applying a CRS of EPSG:4326.
Then we need to...

Re-classify the raster data using "Reclassify by table"

But there is missing information we need to know, how to use this tool. It is calling for a "Reclassification table", what is that and how do we supply it?
The image is a pallet, are we trying to tell it to separate it by color, if so how?