Board index FlightGear Development Scenery

Coverage of the current material regions in FGDATA

Questions and discussion about enhancing and populating the FlightGear world.

Re: Coverage of the current material regions in FGDATA

Postby wlbragg » Fri Apr 03, 2020 12:16 am

I'm sure you're right, that would be irrigated crop. That's the beauty of dictating into your phone.
I think I got yes as an answer in your reply.
Specifically extreme northern Ireland, is this showing irrigated crop defined in three separate files for the same location, 1)North Atlantic islands, 2)United kingdom, 3)Ireland (extreme northern Ireland)?
I'm not making any point or judgments at all, I'm just trying to verify I am understanding the data.

I'm not sure I agree with using a vector area, looks too complicated to me. A box works fine, it's getting the correct size box and also getting the definition in the correct layer. As long as you have a map showing what's been defined and in which files, adding to it should be no problem. It's still quite flexible.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
User avatar
wlbragg
 
Posts: 5607
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: Coverage of the current material regions in FGDATA

Postby www2 » Fri Apr 03, 2020 12:40 am

@wlbragg
I think that vector area is not to complex to do from an user standpoint i don't know about the code side.
For the example I use google earth to make an raw sketch of Africa and past the coordinate to my early post this can be done in a tool.
www2
 
Posts: 267
Joined: Thu Apr 16, 2009 1:58 pm
OS: Ubuntu

Re: Coverage of the current material regions in FGDATA

Postby Thorsten » Fri Apr 03, 2020 5:08 am

Specifically extreme northern Ireland, is this showing irrigated crop defined in three separate files for the same location, 1)North Atlantic islands, 2)United kingdom, 3)Ireland (extreme northern Ireland)?


Nested definitions are not crazy, if you

* define in North Atlantic general characteristics of the region
* specify in UK what is UK specific and
* specify in Ireland what is specifically different in Ireland

aka they make sense as long as the latter only re-define a subset of the former, but they do not make sense otherwise.

Whether this is a bug or a feature really depends on how it is used.
Thorsten
 
Posts: 11648
Joined: Mon Nov 02, 2009 8:33 am

Re: Coverage of the current material regions in FGDATA

Postby wlbragg » Fri Apr 03, 2020 3:16 pm

aka they make sense as long as the latter only re-define a subset of the former, but they do not make sense otherwise.

Whether this is a bug or a feature really depends on how it is used.


I agree, I think it is quite powerful. The nesting of three deep, identical definitions, shows the importance of a good visual map, especially as the amount of file definitions grow.

I doubt the system is taxed much during the processing of the xml as it is still doing what it was designed for, but ideally I would want to eliminate duplicate material type overlapping definitions.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
User avatar
wlbragg
 
Posts: 5607
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: Coverage of the current material regions in FGDATA

Postby ludomotico » Fri Apr 03, 2020 3:50 pm

IMHO, the regional materials in the world are more or less under control. There are some regions that probably need to be lower in materials.xml ( africa_mediterranean.xml, for example), but these are little issues. Also, california.xml is a great region with very good custom materials, and it is not loaded in materials.xml

On the other hand, South America, and specially the many overlapping regions on the Amazonia, is probably "abusing" the system and that part of the word may need a rethinking.

I don't think FlightGear needs changing how materials are managed, but a protocol to create a custom region. For example: "define all available materials for your region". Or "define only differences over the region you are stacking on"

Finally, custom region materials maybe could be distributed as a custom scenery, loading "Materials" from other directories other than FGROOT. In fact, a custom scenery needs to be able to include also custom materials. I don't know how difficult is to code this.
User avatar
ludomotico
 
Posts: 1110
Joined: Tue Apr 24, 2012 1:01 pm
Version: git
OS: Debian GNU/Linux

Re: Coverage of the current material regions in FGDATA

Postby Johan G » Sat Apr 04, 2020 5:41 am

ludomotico wrote in Fri Apr 03, 2020 3:50 pm:[...] custom region materials maybe could be distributed as a custom scenery, loading "Materials" from other directories other than FGROOT. In fact, a custom scenery needs to be able to include also custom materials. [...]

Good point.
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5791
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: Coverage of the current material regions in FGDATA

Postby Thorsten » Sat Apr 04, 2020 5:42 am

also, california.xml is a great region with very good custom materials, and it is not loaded in materials.xml


I've disabled it because it is a region exclusively designed for low-level flight - it has very strong and pronounced tiling from airliner altitude and this needs fixing.

I agree that the textures are great and that the low-level visuals are very compelling, but things really need to work for more use cases.

Or "define only differences over the region you are stacking on"


That's been my general principle when working on a region.

Finally, custom region materials maybe could be distributed as a custom scenery, loading "Materials" from other directories other than FGROOT. In fact, a custom scenery needs to be able to include also custom materials. I don't know how difficult is to code this.


Many of these are rather FG version-specific because they interface with the effect system, so it's not quite obvious whether this really is a good idea in the longer run.
Thorsten
 
Posts: 11648
Joined: Mon Nov 02, 2009 8:33 am

Re: Coverage of the current material regions in FGDATA

Postby Hooray » Sat Apr 04, 2020 8:44 am

ludomotico wrote in Fri Apr 03, 2020 3:50 pm:custom region materials maybe could be distributed as a custom scenery, loading "Materials" from other directories other than FGROOT. In fact, a custom scenery needs to be able to include also custom materials. I don't know how difficult is to code this.


This is actually a general trend in FG, i.e. all sorts of subsystems are incrementally extended to support the notion of "overlays" that take priority over default/fgdata level search paths/directories and files.
This has been happening for $FG_SCENERY, $FG_HOME, addons, graphics card profiles, and it's also what's currently taking place behind the scenes with the compositor.

What's really needed is to formalize and generalize the concept and introduce a dedicated SimGear level helper for these sorts of "PropertyList/XML Overlays".

For the time being, these have been implemented separately, i.e. one by one - whereas the underlying idea remains the same, you only need to look at the SimGear::ResourceProvider abstraction to see that the project is basically half-way there already. It's just that people recognize the need for these "overlays" at different times.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11836
Joined: Tue Mar 25, 2008 8:40 am

Re: Coverage of the current material regions in FGDATA

Postby vnts » Mon Apr 06, 2020 12:19 pm

www2 wrote in Fri Apr 03, 2020 12:08 am:what the news system looks like:
<vector_area>
-10.54509670418442,31.56079284897752,0 -17.03064667719349,24.05248434111258,0 -20.19204639655966,14.20751320035382,0
...
wlbragg wrote in Fri Apr 03, 2020 12:16 am:I'm not sure I agree with using a vector area, looks too complicated to me. A box works fine, it's getting the correct size box and also getting the definition in the correct layer. As long as you have a map showing what's been defined and in which files, adding to it should be no problem. It's still quite flexible.

Not sure exactly what www2 meant (coords=lat,lon, elevation?).

A definition of area need not be more complicated(?) to use. The tricky part might be C++ (?) code doing an intersection to check with a user defined polygon instead of a box. A map visualisation tool is needed in both cases.

e.g. A list of N points defining a polygon moving clockwise to connect dots:
<polygon_point_list>
lat1, lon1
lat2, lon2
lat3, lon3
lat4, lon4
...
latN, lonN
</polygon point list>

Existing box definitions would be 4 points. Tricky bits: visualising overlap, a way to check points were defined correctly e.g. not a straight line.
AIUI the benefit is defining complex boundaries materials and vegetation affected by geology/terrain.
I think this is the same effect as what www2 meant ..?
Hooray wrote in Sat Apr 04, 2020 8:44 am:This is actually a general trend in FG, i.e. all sorts of subsystems are incrementally extended to support the notion of "overlays" that take priority over default/fgdata level search paths/directories and files.

What's really needed is to formalize and generalize the concept and introduce a dedicated SimGear level helper for these sorts of "PropertyList/XML Overlays".

This reminds me. The same principle could apply to every type of FGdata file.
Something like this may be a solution to multiple issues that I had been considering mentioning for a long time. But AFAICS there is no dedicated forum for interfaces / interaction (UI and UX).. so no obvious place to mention it, or check if it had been brought up, for people still pretty new to or with intermittent time with the project.

The idea:

To make modifications & experimentation easier, maybe FG could search through a Changes (Mods) folder for changed&new files. To replace & supplement FGdata files.
Requests to read/write from an FGdata file would use files from an experiment instead. Example: data/Shaders/someshader.vert -> Changed/experiment01/data/Shaders/someshader.vert. A 2nd experiment building on the 1st by another person: Changed/experiment01/data/Changed/2ndexperiment01/Shaders/someshader.vert. Multiple sets of unrelated changes would work provided they didn't touch the same file.

If multiple experiments changed a file maybe priority system can help. Example: like setting scenery folders priority in the GUI, or maybe sorting experiments by name in the changed folder.

Benefits/problems addressed:
- Maintaining a list of local experiments/changes when moving FG versions or changing nightlies. Switching between parallel installs. It's just not experiments: there could be .xml settings/input changes. Or lists of things like pylon models that were moved from FGdata to terrasync(?), which old sceneries still request that hasn't been fixed yet.
- Making it easier to get into tinkering with FG. Avoids needing to make backup copies. Creates the (correct) perception there's no risk to tinker - small example about perception from a previous discussion about tweaking a shader (link).
- Simpler to distribute WiP things to a wide range of people to test. No overwriting needed. No text modifications to xml files. The Australian demo scenery is an example with it's .xml modifications and installing textures. If needed to distribute to a wider range of people, it's possible to even have an install mechanism similar to scenery with check boxes like in the nightly build. Being able to turn off changes by a check box is also convenient for testing.
- Potentially changes to FGdata in nightlies could be delivered by this system via terrasync without big downloads/bandwidth usage/installation times/parallel install problems. Similar to aircraft downloads functionality, but with changed files relative to last major FG version at different commits instead of aircraft files. Potentially this might allow a wide range of users test things, and switch between history of nightlies. Maybe binary files could be included this way as well. This might(?) need a changed folder for binary files & the ability to start and send a commandline to an updated fgfs.exe. Having a few test branches for big changes may have helped things like the compositor development. Switching between nightlies for specific commits might make it easy to pinpoint when problems start without needing to compile from source. Maybe unstable versions could be released this way as the total size of changed files will often be small.

Just an approach that occurred

Kind regards
vnts
 
Posts: 174
Joined: Thu Apr 02, 2015 12:29 am

Re: Coverage of the current material regions in FGDATA

Postby Hooray » Mon Apr 06, 2020 5:02 pm

the idea sounds interesting, we've previously had discussion about "modding" - more recently, the addon system has been added.

http://wiki.flightgear.org/Addons
http://wiki.flightgear.org/Howto:Creati ... _framework

Features like Erik's GPU Profiles, are basically about the same thing: http://wiki.flightgear.org/Graphics_card_profiles

There is already support for "partial overlays" via FG_SCENERY and FG_HOME
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11836
Joined: Tue Mar 25, 2008 8:40 am

Re: Coverage of the current material regions in FGDATA

Postby stuart » Wed May 20, 2020 10:24 pm

HI Folks,

Returning to this thread somewhat late, for which I apologize. To make up for it, here's a consolidated set of answers/opinions :D.

So, for maximum amusement I hit _exactly_ the problem mentioned higher up in this thread just this evening trying to debug a Compositor issue:
- north-atlantic-islands.xml defines Landmass with a very dark texture and shrubs. It's southernmost extent is lat 55, which overlaps most of Scotland.
- united_kingdom.xml is higher in priority, so should over-ride it. Except it doesn't defined Landmass, so the dark Landmass is applied.

@Thorsten - do you recall why north-atlantic-islands.xml is defined with such a low lat? Sheltand is above lat 60, so I think it's minimum lat should be set to that. Also, the terrain-overlay Effect is broken on Compositor, rendering almost pitch black at ultra-high settings. Any suggestions where to look first to debug it?

Addressing a couple of more general points raised in the thread:

* The algorithm for working out the materials definition looks up the individual material through the material definition set based on lat/lon. It doesn't attempt to find the region for a given lat/lon. The distinction is important as it allows inheritance to work (for better or worse as noted above!)

* Changing from a lat/lon box to an arbitrary boundary would certainly be do-able code-wise, but I'm not convinced would gain us much and would be more complex for material developers who I suspect would just define a rectangle anyway.

* I don't think distributing custom materials with scenery is a good idea. Materials are very tightly coupled to bother the Effects system and the textures. Both of these are defined in fgdata. By distributing them with scenery you would be unintentionally tying the scenery to a specific FG version. Also, from a high level design perspective, it breaks the current separation we have between the data (scenery), and how it is rendered (OSG/Effects/Shaders/Textures/materials.xml). I think that separation is a good one, and allows us to improve rendering for all scenery by changing Effects/Shaders/materials.xml.

* Re: overlays. What vnts describes is already supported by using branches on simgear/flightgear/fgdata. That's going to be a far more robust solution than trying to create some sort of Flightgear specific CSM. As an aside, I don't recall if the resource loaders will first check under FGHOME before FGDATA for every resource type, which would allow changing textures etc. without having to make changes to FGDATA. Might be something worth considering if it doesn't already do so.

-Stuart
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1559
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

Re: Coverage of the current material regions in FGDATA

Postby vnts » Thu May 21, 2020 2:14 pm

@stuart Re:the terrain overlay effect being broken on compositor: It might just be a quality settings & selection thing in the effect file. I looked at the shaders after the last Compositor update a while ago and didn't find anything obvious with them. The grass overlay works, so that might be a reference. Grass uses a different set of shaders/effect file, and additionally has the shadow map applied at the cost of reducing the number of layers slightly. But otherwise both use a similar principle.

Somethings are merely not ported yet, somethings like water lack shadows (not sure if intentional for performance reasons), and some craft like Alouette give errors about creating a technique because of hardcoded? searching for shaders that are named in the non-compositor format and being redirected to the compositor folder. Terrain overlay problems maybe just a WiP thing.

stuart wrote in Wed May 20, 2020 10:24 pm:As an aside, I don't recall if the resource loaders will first check under FGHOME before FGDATA for every resource type, which would allow changing textures etc. without having to make changes to FGDATA. Might be something worth considering if it doesn't already do so.

Hadn't heard of FGHOME taking priority for all files, just something about using it for preferences (wiki FGHOME page seems outdated). I'm just keeping parallel installs around with various experiments right now to avoid losing things when trying new versions. If the code support is already present for letting FGHOME take priority for every type of file in FGDATA, maybe it could be made more explicit by creating a FGHOME/mods folder with a your.modifications.readme.txt file saying it's the preferred way to experiment with changing FGDATA. Being able to run multiple different/conflicting experiments using different folders inside mods like mods/experiment3 would be useful (then that could be toggled like scenery files to use/test the unmodifed FG).

The idea was to require the absolute minimum entry barrier to start tinkering while making it seem as low risk as possible (knowledge of creating folders is as basic as it gets, and there's not much time spent learning). The thought was partially in the context of threads like this [1], apparently showing non-technical users (I'm not a programmer professionally either..) can be put off if tinkering is awkward at first - this person was seemingly very non-technical but had a long history of modding art in XP IIRC and tried his best even buying a flash drive.

stuart wrote in Wed May 20, 2020 10:24 pm:What vnts describes is already supported by using branches on simgear/flightgear/fgdata. That's going to be a far more robust solution than trying to create some sort of Flightgear specific CSM.

Firstly, I meant this more as a philosophical point to raise that might become relevant if there's a refactor in the way things are done at some point in future. Something like this might require a lot of work, and the short term benefit may not be worth the effort.

If I understand correctly about branches&SCM, that's talking about pulling from git and compiling? I meant more a way of letting (mainly windows?) people who aren't compiling from source access to a ~1 year history of binaries which are already produced and matching FGDATA at different points so they can test easily without being put off by parallel nightly installs, so they can be aware of what's changing as commits are displayed in launcher, and so they can switch between points rapidly to compare, as well as maybe trace the commits when problems started. Manually searching different points in commit history requires recompiling simgear+flightgear each time, which would be time consuming for even people who usually compile? My limited impression is a lot of windows people heavily involved working on aircraft or other areas who can pick up on problems don't compile from source, and some may not use nightlies as parallel builds? Making commits switchable through the launcher effectively means less-stable releases and nighties are accessible to everyone using LTS without needing to mess around with parallel installs. The thought came up in the context that currently the 1 stable release per year format came about as there wasn't sufficient testing time available for 2-4 stable releases (?), and maybe some of the bug hunting time that delayed the 2019.2 release could have been avoided with a quicker way to compare points in commit history, or sidestepped with easier access to nightlies and quicker bug reports.

Just some thoughts

Kind regards
vnts
 
Posts: 174
Joined: Thu Apr 02, 2015 12:29 am

Re: Coverage of the current material regions in FGDATA

Postby stuart » Fri May 22, 2020 7:38 am

Thanks for the hints on the terrain-overlay. I'll do a comparison with the grass overlay. It's slighly odd, as the textures referenced in the materials file are all very dark, and at first glance it looks like the Effect may be doing exactly what it is being configured to do.

I take your point about making it easier for users to move back-and-forth between versions and making things more accessible for windows users. I'll have a chat with Richard and James.

The new approach with a Long Term Maintenance release (2018.3) and a more stable "preview" (2020.1) might help this as aircraft developers shouldn't have to follow the nightlies as much.

-Stuart
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1559
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

Re: Coverage of the current material regions in FGDATA

Postby vnts » Fri May 22, 2020 3:56 pm

stuart wrote in Fri May 22, 2020 7:38 am:Thanks for the hints on the terrain-overlay. I'll do a comparison with the grass overlay. It's slighly odd, as the textures referenced in the materials file are all very dark, and at first glance it looks like the Effect may be doing exactly what it is being configured to do.

The overlay technique does the normal pass over terrain with full lighting/haze, and the overlays are done in a separate pass with geometry shaders adding detail on top in layers. Since there's nothing that stands out with the compositor overlay shaders, what seems to be happening causing the problems and darkness is likely in the effects file (maybe just selection logic). I'm not sure what the system is with the quality settings/other renderers, so I didn't look further after looking at the shaders back then. The normal terrain pass seems to have problems when overlays are turned on, while the overlays themselves render ok. Sometimes the normal terrain is pitch black, other times there is some thing low quality(?) showing underneath with lighting that changes with view angle, and at other times the normal terrain doesn't seem to render. The blackness doesn't seem to be due to the underlying terrain materials..e.g. on light ground with overlays defined near ENBR, the ground is light yellow/brown until overlays are switched on.

ENBR, non-compositor & compositor: https://imgur.com/a/pzsv8qa
stuart wrote in Fri May 22, 2020 7:38 am:..as aircraft developers shouldn't have to follow the nightlies as much

If it's something to be discussed in future, another point I forgot to add is it can give full benefits of 'release early and release often' for aircraft testing. There was some frustration/confusion IIRC over not being able to get releases out to LTS users as craft needed newer core functionality present in less stable releases. This probably affects craft most when they are being developed to unprecedented levels of detail compared to craft of that type in FG, like with the airliners. But taking advantage of this would maybe need a way to allow users to select between craft compatible with LTS, less-stable release, and unstable nightly versions (for WiP craft, maybe encouraging FGAddon to be used for development versions).

Having said that, since the benefit from any of these depend purely on how people respond, and I can't really comment if benefits are worth the cost especially short term as I don't have that much time/familiarity with FG due to using intermittently over the last few years.

Kind regards
vnts
 
Posts: 174
Joined: Thu Apr 02, 2015 12:29 am

Re: Coverage of the current material regions in FGDATA

Postby Thorsten » Fri May 22, 2020 5:55 pm

the terrain overlay effect being broken on compositor:


It's broken, it's easily fixed at the expense of inserting I think two lines, I have a fix on the harddisk of my windows box, there's no mystery to be solved here - I just need to somehow find the time to get a new repo clone, copy the result over and push a fix (if no one beats me to it).
Thorsten
 
Posts: 11648
Joined: Mon Nov 02, 2009 8:33 am

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: YandexBot [Bot] and 3 guests