Board index FlightGear Development Scenery

Strategy for updating Australian regions/materials in FGData

Questions and discussion about enhancing and populating the FlightGear world.

Strategy for updating Australian regions/materials in FGData

Postby frtps » Sat Jun 12, 2021 3:36 am

I would like to start merging those parts of the Australian scenery project that are ready into FGData. I had been waiting until I had a complete set of textures that I was happy with, but that is going to be several more years at my current pace. I'd like to get feedback on my proposed strategy before peppering FGData with merge requests. What is ready now:

  1. Detailed region boundaries based on climate classifications and study of aerial photos
  2. Some textures that are ready to go for suburban, town, summer/winter cropland, some types of Forest ("bush") etc.
  3. A few alternative shaders with overlay mixing parameters adapted to 1024x1024m texture size to remove tiling
  4. Trees - a bit too dark for my liking but perhaps better than the default

I'd like to present these as a series of merge requests to FGData in the following order. The idea is that the most acceptable changes are not held up by discussion over the relative merits of various textures. I'd also be happy to submit textures that are better than default but not yet ideal into FGData but I think churn of textures is a great way to burn up space in git repositories, especially ones that don't use Large File Storage, so I'm trying to avoid that.

  1. Updated materials.xml + regional xml files referring *only* to already-existing textures but including the new region boundaries.
  2. New textures and corresponding updates to the regional xml files
  3. New shaders and corresponding updates to the regional xml files

Note that all new textures are derived from CC-BY-4.0 orthophotos distributed by the Australian NSW government so there are no use issues (these are default ones shown by openstreetmap).
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby Thorsten » Sat Jun 12, 2021 6:56 am

We've so far had a policy of making it easy to create regional packages for a region so far only available as default - only if your work overrides what someone else has done, the standards have been a bit higher.

So if your changes affect only Australia, and fulfill rather general criteria (GPL compatible, needs to work acceptably also at low shader quality, should be free of tiling from high altitude at higher quality,...) I think there's no need to hold up a merge request - if you can't get someone else to do it, I'll be happy to push the button.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Sat Jun 12, 2021 10:10 am

Sounds good, I'll get started with creating the stack of merge requests.
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby vnts » Sat Jun 12, 2021 1:46 pm

frtps wrote in Sat Jun 12, 2021 3:36 am:A few alternative shaders with overlay mixing parameters adapted to 1024x1024m texture size to remove tiling

Not sure what you mean specifically - but it might be simpler to avoid creating a new shader, and expose more controls as parameters with a default value set to the current value? e.g. if the shader in question doesn't have overlay scale exposed as a control (e.g. <overlay_magnification> parameter), or you are changing an unexposed attribute of the noise function used in mixing. It's also possible to add a new feature gated by a uniform flag that is off by default, if it's not too big of a code change - e.g. if you are porting (?) the detiling code from the agriculture shader.

frtps wrote in Sat Jun 12, 2021 3:36 am:1. Detailed region boundaries based on climate classifications and study of aerial photos

You probably implemented these with lots of boxes? In the long-term a way to handle this type of boundary would be extending the box definitions to polygons an arbitrary number of lat/lon pairs defined clockwise.

In a lot of cases, natural boundaries are generally hard to do with boxes. For example I was looking at a valley floor texture for alps after it was pointed out in the mailing list that there were some landlcass mapping issues in certain countries. I had a look at geology too.

Geology of alps - a rock climbing site that came up in search - link . Geology affects appearance of vegetation, bare mountain look, mountain meadows, and even the colour of (glacial) rivers/lakes from things like suspended paricles in limestone areas:
Image

So regional definition structure for the alps covering multiple countries might be: 3 xml files for natural terrain with differring mountain & lake appearance - and terrain affected by human activity like alpine valleys, agriculture, towns in different countries can get a separate common alps definition or per-country regional definition.

frtps wrote in Sat Jun 12, 2021 3:36 am: I had been waiting until I had a complete set of textures that I was happy with, but that is going to be several more years at my current pace.

Release early and release often is best from an opensource point of view :mrgreen:. Unfortunately the amount of exposure drops exponentially if even 1 extra step is needed to get scenery - the FGUK photoscenery had only 70 downloads when I looked despite being the only such package for several months, while there are many thousands of downloads of FG per week. And as Thorsten said the standard for inclusion is that it's an improvement with no technical issues like tiling - so the bar is pretty low in regions with placeholder definitions.

frtps wrote in Sat Jun 12, 2021 3:36 am:I'd also be happy to submit textures that are better than default but not yet ideal into FGData but I think churn of textures is a great way to burn up space in git repositories, especially ones that don't use Large File Storage, so I'm trying to avoid that.

Is the binary file churn that serious if there's one or two iterations on 1-2mb textures? Might be something to ask the list..
People are expected to iterate on textures and binary files in the normal course of development - e.g. AI aircraft textures may be replaced by JPGs in future, and there's a steady stream of commits to AI craft in FGData. There appears to be ways of cleaning binary files from git history according to google if it becomes troublesome in future (?) (e.g. link), but this seems to need fresh clones and would need some careful backing up. Incidentally there's a FGAssets repo being set up for binary files of source data for OSM2City building improvements - that could also be used for regional texturing sources etc. or a similar one created for FGData if there's an unavoidable need (?) to download the whole repo when cloning.

Kind regards
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am

Re: Strategy for updating Australian regions/materials in FG

Postby Thorsten » Sat Jun 12, 2021 6:31 pm

Not sure what you mean specifically - but it might be simpler to avoid creating a new shader, and expose more controls as parameters with a default value set to the current value?


Actually the suggestion to solve some issues with custom shaders rather than try to adapt the defaults to something else is mine. It seemed to me at that time that the requirements were different enough to not try to use the same technology.

@frtps: With regard to shaders - I assume they're adapted for the compositor now - as this is the state of the art on current FGAddon?
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Sun Jun 13, 2021 1:36 am

Thorsten wrote in Sat Jun 12, 2021 6:31 pm:
@frtps: With regard to shaders - I assume they're adapted for the compositor now - as this is the state of the art on current FGAddon?


So all I did was change the noise term coefficients and scales when overlay/base/detail are mixed in terrain-ALS-ultra.frag. I'll base the merge off whatever is most recent in next. I'll have to read up on whether or not the compositor affects this particular part of the shader. I think when you were suggesting a different shader I was thinking of a whole new noise function. So I tried a fancier noise function, it was far too resource intensive. The current FG noise function has the disadvantage that there is some "striping" in the areas I was texturing at the scales of the textures I wanted to use, so in the end I went for the simple solution of changing the noise scales to match the textures. Thus @vnts' solution might be the better one now.

vnts wrote in Sat Jun 12, 2021 1:46 pm:
frtps wrote in Sat Jun 12, 2021 3:36 am:A few alternative shaders with overlay mixing parameters adapted to 1024x1024m texture size to remove tiling

Not sure what you mean specifically - but it might be simpler to avoid creating a new shader, and expose more controls as parameters with a default value set to the current value? e.g. if the shader in question doesn't have overlay scale exposed as a control (e.g. <overlay_magnification> parameter), or you are changing an unexposed attribute of the noise function used in mixing. It's also possible to add a new feature gated by a uniform flag that is off by default, if it's not too big of a code change - e.g. if you are porting (?) the detiling code from the agriculture shader.



Yes it seems very wasteful to create a whole new shader which is just a copy of the current one with some numbers changed just because I want some particular area not to be tiled. You are right that the length scale of the texture could be used as input to the fragment shader and the noise scales for input into mixing derived from that.
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Sun Jun 13, 2021 1:47 am

vnts wrote in Sat Jun 12, 2021 1:46 pm:
frtps wrote in Sat Jun 12, 2021 3:36 am:1. Detailed region boundaries based on climate classifications and study of aerial photos

You probably implemented these with lots of boxes? In the long-term a way to handle this type of boundary would be extending the box definitions to polygons an arbitrary number of lat/lon pairs defined clockwise.

In a lot of cases, natural boundaries are generally hard to do with boxes.


Ain't that the truth. But I don't think we have any other option unless some kind soul has implemented vector-based definitions of areas in the materials xml files and more importantly in the Simgear code. I did see that this came up a while back.

vnts wrote in Sat Jun 12, 2021 1:46 pm:
frtps wrote in Sat Jun 12, 2021 3:36 am: I had been waiting until I had a complete set of textures that I was happy with, but that is going to be several more years at my current pace.

Release early and release often is best from an opensource point of view.


Absolutely. I tried to put everything the Australian scenery repository at least but that is as you say a lot less exposure than having it in Flightgear.

vnts wrote in Sat Jun 12, 2021 1:46 pm:Is the binary file churn that serious if there's one or two iterations on 1-2mb textures? Might be something to ask the list..


Well, FGData is already a monster so maybe you're right that a few more 2Mb textures won't break the bank.
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby vnts » Sun Jun 13, 2021 12:22 pm

frtps wrote in Sun Jun 13, 2021 1:36 am:So all I did was change the noise term coefficients and scales when overlay/base/detail are mixed in terrain-ALS-ultra.frag. I'll base the merge off whatever is most recent in next.

The shaders are in pretty much exactly the same shape as before - adaptation to the compositor is fairly trivial with the way it's done - so you could just expose a few new parameters with defaults set to existing tuning and there shouldn't really be any merge conflicts.

(The current state of Compositor shaders is: in the fragment shader there's a call to getShadowing() before inserting the shadow value alongside cloud shadows. getClusteredLightsContribution() is called to get light contributions, before they are inserted along with the ALS environmental lights which are still in place. In the vertex shader there's calls to functions to setup lights/shadowing, but it's otherwise the same. The shadow/light functions are just in included shader files, and do the required lookups of 2d/3d tables/textures.)

Kind regards
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am

Re: Strategy for updating Australian regions/materials in FG

Postby Thorsten » Sun Jun 13, 2021 6:27 pm

I think when you were suggesting a different shader I was thinking of a whole new noise function.


Ah, yes, okay - I didn't know that never materialized. If it's just about scale parameters, it seems better to expose them and use different defaults, I agree.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Thu Jul 15, 2021 10:49 am

So I've just submitted a merge request for texture-size-dependent noise, it just uses <xsize> from the material description, it turned out very easy as the parameter was already in place. If someone could check that over that would be great!
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby vnts » Fri Jul 16, 2021 5:47 am

The commits for merge request from the FG website: link , link

- It seems there's a stray materials.xml file from the Australia regional definitions that refers to regional definition files that aren't yet merged (may cause console errors)

- I get a " Could not locate shaderShaders/deferred-gbuffer.vert" console error after downloading and copying the files from soruceforge (the latest widnows nightly is stuck on loading screen because of this I think). It seems terrain-default.eff in your branch is old - rembrandt has been removed (so geometry buffers aren't used) and there has been various adjustments and bugfixes to compositor effects. You could try rebasing the small change on top of the current next (I think that is the correct git approach?) - it shouldn't give any complicated merge conflicts or anything. There are changes to terrain shaders since then, and maybe materials.xml too. You could probably rebase all the Australia branches on top of next to be sure.

----

frtps wrote in Thu Jul 15, 2021 10:49 am:[..]texture-size-dependent noise, it just uses <xsize> from the material description


Looking at the commits on sourceforge (no test, as FG was stuck loading):

Not exactly sure what the goal is here. <xsize> is the distance over which the texture is stretched. <xsize> varies quite a bit try a text search for possible values "<xsize>4000" etc e.g. up-to 4000m (see brazil*.xml) or 10000m (africa-mediterranean). A lot of the larger sizes are the forest.eff, which is just a proxy for terrain-default. The xsize=256 values for grass seem to be handled by the airfield shader at highest quality, so don't need to worry about that.

So scaling the noise scale lengths will make scale lengths reduce/expand - e.g. a 25m noise for 2000m texture becomes ~10m noise for a 1000 texture, 50m noise for a 4000 texture, and 125m noise for a 10000m. A possible issue is that noise is faded out in some cases (mainly snow), to implement a form of antialising/mipmapping by the detail_fade() function. The noise that's faded out includes 50m-10m scales in snow which uses noise scales and snow noise derived from noise. Some of the larger noise functions may shrink to smaller scale categories, or expand to larger categories - causing the fading at various scales to be inaccurate? The fix is probably to scale the values in the detail_fade functions by swatch_size too.

(I'm not exactly sure what the problem being solved is - that some of the textures are low resolution and need more structure - so detail can be added by taking advantage of the fact that smaller texture are stretched smaller most of the time and reducing the scales of the noise layers to force detail? or something else.. Otherwise the noise detail scales are independent of how much the texture is stretched. )

- terrain-als.frag [1] swatch_size: Maybe the comment could tell where the value it comes from so it's less confusing to someone trying to figure out what's going on in future (there's a y size too but terrain textures are square) - e.g. 'Distance the base texture is stretched in m (c.f. xsize material definitions)'

----

Also I did bring up FGData repo size on the mailing list and James plan going forward until sourceforge supports binary files better, is to move things out of FGData as needed and install-on-demand or at install time from outside the repo - as is being planned for AI aircraft (which should reduce a lot of repo churn and free up space in the installer) : https://sourceforge.net/p/flightgear/ma ... /37304275/ and https://sourceforge.net/p/flightgear/ma ... /37304277/
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Fri Jul 16, 2021 10:40 am

Thanks for taking a look, I based off 2020.3.8, didn't realise there was quite so much going on in FGData. Will do the git invocations and try again.

(I'm not exactly sure what the problem being solved is - that some of the textures are low resolution and need more structure - so detail can be added by taking advantage of the fact that smaller texture are stretched smaller most of the time and reducing the scales of the noise layers to force detail? or something else.. Otherwise the noise detail scales are independent of how much the texture is stretched. )


The problem is that described in this thread, that is, the dominant scale in the overlay noise in the current distribution has patches that are too large for smaller (<4kmx4km) textures, and so show tiling from repetition of a single texture with no overlay to break it up. By adapting the noise scale to the texture scale this tiling can be more effectively removed. Mind you, nobody seemed to see the problem, maybe because I didn't include a few examples using real textures.
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby vnts » Sat Jul 17, 2021 4:47 am

frtps wrote in Fri Jul 16, 2021 10:40 am:I based off 2020.3.8, didn't realise there was quite so much going on in FGData.

Oh next has diverged a huge amount compared to the release branches, as far as effects (and to a lesser extent, shaders) are concerned - changes from next are carefully cherry picked to the LTS these days for point releases, so it's better to use next for developing to make it easier to test/merge.
frtps wrote in Fri Jul 16, 2021 10:40 am:The problem is that described in this thread, that is, the dominant scale in the overlay noise in the current distribution has patches that are too large for smaller (<4kmx4km) textures, and so show tiling from repetition of a single texture with no overlay to break it up. By adapting the noise scale to the texture scale this tiling can be more effectively removed. Mind you, nobody seemed to see the problem, maybe because I didn't include a few examples using real textures.

I see - so the gaps are too large. The shader computes noise on various scales and uses a different noise scales for various things where needed - so there isn't a need to change the scales of the noise as much as just add a combination of smaller scale functions to the part where the base and overlay textures are mixed. But as was mentioned in the other thread it's tricky to hide tiling (I've used very low res / smaller scale overlays that tile a lot in airport keep effects like in California with the overlay bias tuned so the overlays only shows here and there).

With regards to performance, if the useage of certain noise scales are in conditional block(s), and the noise scales aren't needed outside the blocks, there won't be a performance hit.

It's possible to auto-select different combinations of noise scales for different texture stretch sizes, if you don't want to have the smaller scale noise on by default to save computation when it's not needed (although it might improve regional definitions around the world that have tiling while not degrading any effects so it could be worth enabling auto-scaling everywhere...). Condition block(s) if swatch_size uniform is within different ranges should work fine for auto-selection. To allow manual selection of different combinations of noise scales, it's also possible to use an integer uniform with different mixing modes (maybe define some extra transition_model values) - although this requires additional documentation in the wiki compared to automatic selection.

Another option / solution is to port the random domain based mixing code from the agriculture shader - for both the base and overlay textures, or maybe only for the usually smaller scale overlay texture. But this won't work with tree/object overlays - but it works great for bare/grassy terrain, even though the voronoi noise functions used are on the slightly expensive side I think.

If it helps, it's also possible to expose controls for the width of the transition from base to overlay textures (e.g. a value to add/subtract from the 1st 2 parameters of smoothstep function).

Kind regards
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am

Re: Strategy for updating Australian regions/materials in FG

Postby frtps » Mon Jul 26, 2021 10:30 am

So I've now redone that merge request: it's a very simple translation of the intent of the original, which looks to be for 4kmx4km texture blocks, to arbitrary sized texture blocks. At the moment only for terrain-ALS-ultra, so you can easily compare before and after by switching between the "high" and "ultra" rendering levels.
frtps
 
Posts: 145
Joined: Sun Aug 05, 2018 10:58 am

Re: Strategy for updating Australian regions/materials in FG

Postby vnts » Tue Jul 27, 2021 11:45 am

I tested it out on QLD just now on FGData next - no errors etc. (I seem to have QLD and a few other test sceneries from last year. TerraSync doesn't appear to have the updated terrain yet.)

Most of the landclasses rendered by the terrain shader I found were Evergreen/Shrub/Savanna type classes that didn't have big tiling problems, but the change in noise functions was visible showing the code worked. Huge amounts of the terrain seems to be classed as drycrop/drycop pasture and is handled by the agriculture shader. so But it seems to still produce quite large blobs of classes right now (the affected noise scale has gone from 500->2000 to 500->4000). I think it helps if the scale of the blobs is smaller than the identifiable repeating pattern of the texture, so the pattern becomes unidentifiable - so breaking it into scales smaller than 500m might be useful for some textures with tiling. Maybe the scales it's broken up into could be configurable based mixing mode uniform or something.

Album: https://imgur.com/a/NLU5Rs3

----
This is sort of what I mean by the snow and detail fading for snow (procedural anti-aliasing / mipmapping) being affected as a side effect - the detail fade functions are also affected as they expect certain noise scales - but if the noise scales if the noise scales change based on texture stretching they don't get faded in/out correctly.

Thick snow covering up the entire texture (a lowish sun and hilly terrain makes it easy to see snow detail):

Before:
Image

After, the two aren't identical:
Image

Kind regards
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am

Next

Return to Scenery

Who is online

Users browsing this forum: No registered users and 4 guests