Board index FlightGear Development Effects and shaders

Landmass problem

An exciting "new" option in FlightGear, that includes reflections, lightmaps, the particle system etc.. A lot is yet to be discovered/implemented!

Re: Landmass problem

Postby Thorsten » Mon Aug 14, 2017 5:12 am

You want to delve into Shaders/agriculture* which is what is running where you see the weird colors.
(...)
Maybe it is simply buggy, or maybe the FG-side GLSL uses a shady expression that's processed fine by almost any other driver - in any case it can't be fixed without identifying the line in the code that makes it fail.
Thorsten
 
Posts: 9993
Joined: Mon Nov 02, 2009 8:33 am

Re: Landmass problem

Postby enrogue » Mon Aug 14, 2017 8:15 am

abassign - Thanks for the video - it looked like from that it's something that is affected by distance

Thorsten - thanks for the pointer - by pure luck I may have found the line - at least for me, commenting the following line out removes the artifact:

agriculture-ALS.frag, line 328

texel.rgb = mix(texel.rgb, grain_texel.rgb, grain_strength * grain_texel.a * (1.0 - mix_factor) * (1.0-smoothstep(2000.0,5000.0, dist)));

I obviously know nothing about GLSL, but I'm guessing the next step would be to break down which part of the line is causing the issue

abassign - can you verify commenting out the line removes the issue for you?
User avatar
enrogue
 
Posts: 109
Joined: Mon May 19, 2014 6:40 pm
Location: London (UK)
Callsign: enrogue
OS: OS X, Ubuntu

Re: Landmass problem

Postby Thorsten » Mon Aug 14, 2017 9:03 am

Code: Select all
texel.rgb = mix(texel.rgb, grain_texel.rgb, grain_strength * grain_texel.a * (1.0 - mix_factor) * (1.0-smoothstep(2000.0,5000.0, dist)));


Fancy that... the same line occurs in the normal terrain shader where it runs just fine according to the screenies. So it seems unlikely that something in the line as such causes issues...
Thorsten
 
Posts: 9993
Joined: Mon Nov 02, 2009 8:33 am

Re: Landmass problem

Postby enrogue » Mon Aug 14, 2017 10:10 am

Hi Thorsten

As I mentioned - I really know nothing about the shaders or GLSL, but I can see it's pretty odd that it's running fine in one shader but not the other

I can verify that when forcing software rendering with LIBGL_ALWAYS_SOFTWARE=1 (so using the llvmpipe renderer) the issue isn't there, and we know that it appears to be an intel only issue - Mint is based on Ubuntu (or was) so I would expect them to have the same of similar kernel/mesa versions

I agree with your previous post that it's most likely a buggy driver, I was just wondering about how to nail it down so as to file a bug with the Intel driver guys
User avatar
enrogue
 
Posts: 109
Joined: Mon May 19, 2014 6:40 pm
Location: London (UK)
Callsign: enrogue
OS: OS X, Ubuntu

Re: Landmass problem

Postby abassign » Mon Aug 14, 2017 1:21 pm

Meanwhile thank you all for the interest of this strange bug!

In the code:
Code: Select all
texel.rgb = mix(texel.rgb, grain_texel.rgb, grain_strength * grain_texel.a * (1.0 - mix_factor) * (1.0-smoothstep(2000.0,5000.0, dist)));

if I modify:
Code: Select all
(1.0-smoothstep(2000.0,5000.0, dist))
with
Code: Select all
(1.0-smoothstep(0.0,0.0, dist))
the problem disappears

This is the example:

With the line commented:
Image

With the line active, but:
Code: Select all
(1.0-smoothstep(0.0,2000.0, dist))

Image

With the line active, but:
Code: Select all
(1.0-smoothstep(0.0,0.0, dist))

Image

Being a very well-known airport in LOWI, I would be curious that someone would do a test to verify the visual difference.
Image
Image

Is the smoothstep function wrong?

I tried to replace the code: ... with a scaling value from 1.0 to 0.0 I never had any problem. Then I begin to raise the value, up to 10 seems to be fine, 100 is the defect but with different colors. If you look at the code as the "1.0 - smoothstep (x, y, z)" function, then I thought smoothstep would give a distance value and not a value between 0 and 1.0. I did the test with negative values and I checked that if I insert -10.0 and smaller values you have the init of the defect.

To figure out if the smoothstep function is the problem, I wrote the second function of Wikipedia (https://en.wikipedia.org/wiki/Smoothstep) Surprise! I got the same result, smoothstep function is not the problem.

Code: Select all
float clamp_A(float x, float lowerlimit, float upperlimit)
{
    if (x < lowerlimit) x = lowerlimit;
    if (x > upperlimit) x = upperlimit;
    return x;
}

float smoothstep_A(float edge0, float edge1, float x)
{
    // Scale, bias and saturate x to 0..1 range
    x = clamp_A((x - edge0)/(edge1 - edge0), 0.0, 1.0);
    // Evaluate polynomial
    return x*x*(3 - 2*x);
}

void main()
...
...
texel.rgb = mix(texel.rgb, grain_texel.rgb, grain_strength * grain_texel.a * (1.0 - mix_factor) * (1.0-smoothstep_A(2000.0,5000.0, dist)));
...
{


This is the result:

Image

Then I replaced "dist" with the "length (relPos)" value from which it was derived:

Code: Select all
texel.rgb = mix(texel.rgb, grain_texel.rgb, grain_strength * grain_texel.a * (1.0 - mix_factor) * (1.0-smoothstep_A(2000.0,5000.0, length(relPos))));


The result was the same, the system displays an incorrect image. At this point, the problem is likely to be in the "dist" values that seem to have problems.
abassign
 
Posts: 737
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2018.3
OS: Linux Mint 19. x

Re: Landmass problem

Postby enrogue » Mon Aug 14, 2017 3:43 pm

Well, if you look at the rest of the shader code, and even just in agriculture-ALS.frag, smoothstep is used everywhere.

I found a reference to the function here (it appears to be a built-in GLSL maths function)

http://www.shaderific.com/glsl-functions/

and some explanation of the renderman version which appears to be identical here:

http://www.fundza.com/rman_shaders/smoothstep/

So I'm at a loss as to how the function is busted - it it's possible the combination of functions are triggering a bug in the driver though (is there such a thing as colour overflow? - I'm think of each component of colour being operated on?)
User avatar
enrogue
 
Posts: 109
Joined: Mon May 19, 2014 6:40 pm
Location: London (UK)
Callsign: enrogue
OS: OS X, Ubuntu

Re: Landmass problem

Postby abassign » Mon Aug 14, 2017 3:56 pm

When the soil is wet, you have the same problems, in fact in the section: "get distribution of water when the terrain is wet" has the same problem, in fact the function is also used: "smoothstep (2000.0, 5000.0, dist)":

Code: Select all
// get distribution of water when terrain is wet

float combined_wetness = min(1.0, wetness + intrinsic_wetness);
float water_threshold1;
float water_threshold2;
float water_factor =0.0;

if ((dist < 5000.0)&& (quality_level > 3) && (combined_wetness>0.0))
      {
      water_threshold1 = 1.0-0.5* combined_wetness;
      water_threshold2 = 1.0 - 0.3 * combined_wetness;
      // water_factor = smoothstep(water_threshold1, water_threshold2 ,   (0.3 * (2.0 * (1.0-noise_10m) + (1.0 -noise_5m)) *   (1.0 - smoothstep(2000.0, 5000.0, dist))) - 5.0 * (1.0 -steepness));
   }


If you take off the part that seems to be defective here it all works well:

Code: Select all
water_factor = smoothstep(water_threshold1, water_threshold2 ,   (0.3 * (2.0 * (1.0-noise_10m) + (1.0 -noise_5m))) - (1.0 -steepness));


It therefore seems that this function, for some reason unknown, but not dependent on how the function itself was written, creates a problem.
The fact that removing "smoothstep (2000.0, 5000.0, dist)" does not produce any visual defect, it may be advisable to remove this feature and then reduce the problems when using different video cards. But the problem might also be the indication that there are "hidden" errors that maybe other drives "cover", but not the Intel drive I use.
abassign
 
Posts: 737
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2018.3
OS: Linux Mint 19. x

Re: Landmass problem

Postby abassign » Mon Aug 14, 2017 4:39 pm

enrogue wrote in Mon Aug 14, 2017 3:43 pm:So I'm at a loss as to how the function is busted - it it's possible the combination of functions are triggering a bug in the driver though (is there such a thing as colour overflow? - I'm think of each component of colour being operated on?)


Hi, I honestly would not know, when I replaced the function with the code, according to Wikipedia, the defect came in the same way, quidni does not seem to be the problem derived from the function. I tried to experiment the algorithm in a spreadsheet and it always gave me consistent results.

Image

Sincerely, I do not want it to be a problem, which is born from the length function, which maybe releases some value out of scale.

I had the same identical problems on my old NVIDIA (800 series with 6 GB of RAM) two years ago when there was snow on farmland. That time I signaled the problem and a few days later it was resolved, I do not know how.
abassign
 
Posts: 737
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2018.3
OS: Linux Mint 19. x

Re: Landmass problem

Postby Thorsten » Mon Aug 14, 2017 5:04 pm

- it it's possible the combination of functions are triggering a bug in the driver though (is there such a thing as colour overflow? - I'm think of each component of colour being operated on?)


I suspect something like that is what's happening. If even software rendering gets it, the code is very likely to be algorithmcally sound.

I had the impressions drivers don't actually execute the code as written but run agressive optimization at shader compile time - and that sometimes does weird things.

I've spent once three days chasing an issue where I could render two channels, both individually fine and as expected, with no dependence on view axis. Yet when I summed them (and clamped the result to [0:1] to be sure to always get a valid color - I saw a dependence on view axis and the result did not look like the sum at all. Mathematically this is impossible - the video driver managed just fine.

Sometimes you can manage to not trigger these issues by re-writing the code in a slightly different way. Sometimes not.

I have no good advice how to isolate this further to give meaningful feedback to driver developers. The practical solution seems easy enough - don't use the agriculture effect while the driver exhibits the problem.


it may be advisable to remove this feature and then reduce the problems when using different video cards


smoothstep is a perfectly fine GLSL function - which is why it runs with basically any other video driver and even in the same form in another shader on the Intel driver you're using. There's no line in the code that's not there for a reason and can simply be removed without side effects, and I strongly suggest that you don't consider 'random deletion of lines' a valid strategy to create rendering code in the future.

We have quality settings for this very reason - if your setup can't handle something, you can dial quality a notch down.
Thorsten
 
Posts: 9993
Joined: Mon Nov 02, 2009 8:33 am

Re: Landmass problem

Postby Hooray » Mon Aug 14, 2017 5:26 pm

From a troubleshooting standpoint, it would be interesting to know if the same shaders show the same problem when executed outside the fgfs environment - e.g. using a GLSL debugger/development environment.

In addition, anything involving fgviewer/osgviewer (without fgfs) would help nail down what is going on, or least determine if the issue is fgfs specific or not.

Meanwhile, a fair rnumber of legacy features are known to interfere badly with some of the more recent OpenGL/GLSL code

For example, some of the screenshots show PUI widgets while those artifacts are visible - from a troubleshooting standpoint it would be interesting to know if hiding/disabling PUI entirely has any effect or not (likewise with other features using legacy GL code, e.g. HUDs).

There is a Nasal snippet that allows you to easily disable PUI at runtime:
http://wiki.flightgear.org/Howto:Reset/ ... leshooting
Image
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11186
Joined: Tue Mar 25, 2008 8:40 am

Re: Landmass problem

Postby Thorsten » Mon Aug 14, 2017 5:38 pm

From a troubleshooting standpoint, it would be interesting to know if the same shaders show the same problem when executed outside the fgfs environment - e.g. using a GLSL debugger/development environment.


Out of curiosity - assuming they would run fine in a debug environment (which seems likely given that the software renderer processes them fine) - what would you actually do with that information?
Thorsten
 
Posts: 9993
Joined: Mon Nov 02, 2009 8:33 am

Re: Landmass problem

Postby Hooray » Mon Aug 14, 2017 5:48 pm

Note that my responses wasn't intended to imply that I disagree with anything you said, it's just that most GLSL drivers these days are based on the same compiler back-end (LLVM), so there are various ways to look at the generated code, and even to tell the optimizer to be less aggressive - furthermore, a GLSL-enabled debug environment will provide much more GLSL related information than we'd typically see in fgfs or osgviewer, even with all debugging flags enabled.

You can basically do OpenGL/GLSL tracing line by line in such an environment, which makes it much easier to draw informed conclusions.


http://glsl-debugger.github.io/
Image

Image

https://www.khronos.org/opengl/wiki/Debugging_Tools

Note that this is not to say that your shaders would work "out of the box" in such an environment, they would certainly require changes - but it's just another tool, and one that has nothing to do with fgfs, and one that can be used to look at what is going on behind the scenes (imagine conducting the same tests using different GPUs/drivers). Even if the conclusion should ultimately be that the issue is indeed fgfs specific, that would be helpful to know.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11186
Joined: Tue Mar 25, 2008 8:40 am

Re: Landmass problem

Postby abassign » Mon Aug 14, 2017 6:57 pm

I have updated the driver version using the PPA: ppa:paulo-miguel-dias/mesa (in this link: https://www.epicgames.com/unrealtournament/forums/unreal-tournament-discussion/ut-development-bug-reports-qa/linux-troubleshooting/14891-how-to-update-open-source-graphic-driver-in-ubuntu)
Now the latest in this version, I do not think there is much:

Code: Select all
abassign@abassign-P15SM-A-SM1-A ~ $ glxinfo | grep 'version'
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
    Max core profile version: 4.5
    Max compat profile version: 3.0
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.1
OpenGL core profile version string: 4.5 (Core Profile) Mesa 17.3.0-devel - padoka PPA
OpenGL core profile shading language version string: 4.50
OpenGL version string: 3.0 Mesa 17.3.0-devel - padoka PPA
OpenGL shading language version string: 1.30
OpenGL ES profile version string: OpenGL ES 3.1 Mesa 17.3.0-devel - padoka PPA
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10


I always get the same problem that I solve, as usual, by editing the line of the "agriculture-ALS.frag" code.

I repeat, I sincerely would like anyone who has a drive that can not handle the problem can send my own image in LOWI. I want to see if there are any substantial differences in the image shown. LOWI I think it is great for this test as it is a very well known airport in FGFS and has many agricultural areas.

If see the same things, then there are good reasons to think that the problem is actually common, only in my case, is displayed more clearly.
abassign
 
Posts: 737
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2018.3
OS: Linux Mint 19. x

Re: Landmass problem

Postby erik » Mon Aug 14, 2017 7:44 pm

Personally I suspect there is a NaN generated somewhere, or something similar.
Could you test what happens if you replace dist with a fixed value?

Erik
erik
 
Posts: 1504
Joined: Thu Nov 01, 2007 1:41 pm

Re: Landmass problem

Postby abassign » Mon Aug 14, 2017 7:53 pm

I have first described various tests, amongst which the one that replaces the value matrix with a constant all works regularly if the value ranges is from 0 to 1. If the value is much greater than 10 the defect colour is near a dark brown, if the constant value is less than zero, the defect appears with a cyan-colored image.
Last edited by bugman on Tue Aug 15, 2017 6:40 pm, edited 1 time in total.
Reason: Please do not quote the entire previous post.
abassign
 
Posts: 737
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2018.3
OS: Linux Mint 19. x

PreviousNext

Return to Effects and shaders

Who is online

Users browsing this forum: No registered users and 1 guest