Board index FlightGear Development Effects and shaders

HUD shader for parallax correction

An exciting "new" option in FlightGear, that includes reflections, lightmaps, the particle system etc.. A lot is yet to be discovered/implemented!

HUD shader for parallax correction

Postby colin_geniet » Fri May 15, 2020 10:58 pm

To avoid ambiguity, I am only talking about Canvas HUDs here, it has nothing to do with the Flightgear built-in HUD.

Currently, most aircrafts with a HUD have a parallax issue: when moving the viewpoint to the side, the HUD image just stays in place, whereas it should stay straight in front of the viewpoint. Because of this, as soon as one moves the viewpoint, indications such as flight path vector, pitch scale, radar returns positions, become wrong. For instance on the F-16: https://i.postimg.cc/9M66JgZL/f16-hud-deform.png

I am aware of two aircrafts which have solved this issue: the Saab 37 Viggen, and to some extent the Space Shuttle [1]. Both have the same solution: transformations are applied to the Canvas based on the view offset to compensate for parallax. Example with the Viggen: https://i.postimg.cc/2yNQHn6Q/viggen-hud-parallax.png The view was translated to the left, the HUD follows and the image remains straight in front of the view point (a part of the HUD image is cut because it is outside of the HUD mirror).

However this solution is not flawless: because the view movements are compensated by computing a transformation in Nasal and applying it to the canvas, the correction will always be at least 1 frame late. When using an IR head tracker (which allows to move the point of view a lot), this delay causes a very noticeable stuttering.

The solution I propose is to do all the required transformations in the HUD shader. I have implemented a modified version of the shader on the Viggen, available here:
https://github.com/NikolaiVChr/flightgear-saab-ja-37-viggen/tree/hud
Diff with the current FGData shader:
https://pastebin.com/r98siWen

It boils down to a couple of lines in the fragment shader:
Code: Select all
vec2 relAngle = vec2(
    atan(optical_relpos.y, -optical_relpos.x),
    atan(optical_relpos.z, -optical_relpos.x)
);
TexCoord = (relAngle / vec2(radians(hud_width), radians(hud_height))) + 0.5;

Here, 'optical_relpos' is the vector from the viewpoint to the fragment position. The angle between this vector and the HUD forward axis determines the position sampled in the texture (TexCoord) after scaling/centering. Thus the HUD image is always displayed in front of the viewpoint, and has a constant angular size.

To get in more details:
  • The optical axis (i.e. the direction to which the center of the hud image is mapped) is by default the '-x' axis in the HUD object model space.
    In general, this corresponds to the aircraft forward axis. However, any rotate animation applied to the HUD object will be taken into account.
    Additional rotations can be applied to the optical axis using effect parameters.
  • A common issue with canvas HUD beside parallax that the canvas is mapped to a slanted surface, which causes deformation. This is visible for instance on the F-16: https://i.postimg.cc/9M66JgZL/f16-hud-deform.png The red lines aligned with speed/altitude scales should appear parallel, but are not because the top of the HUD surface is closer to the viewpoint than the bottom.
    This shader would also avoid this issue: in fact the Canvas image would be seen with no deformation no matter how weird the shape used as HUD object.
  • I believe the shader would make it easier to write the Canvas part of the HUD. Indeed, HUD elements position are typically given as pitch/yaw angles. With this shader, there is a linear relation between pitch/yaw angles and Canvas coordinates.
    Of course porting existing aircrafts is another question...
Since the main improvement is the removed stuttering, I think it is best demonstrated in a video. It compares:
  • An old Viggen version without any parallax correction
  • The current Viggen, with parallax correction in Canvas
  • Parallax correction done in the shader
https://peertube.co.uk/videos/watch/684aa804-199c-423a-8acc-9815e7e627c1


[1] The Shuttle correctly compensates lateral and vertical movements with translations, but not longitudinal movements, which should be compensated by scaling the canvas.
colin_geniet
 
Posts: 21
Joined: Thu May 14, 2020 11:07 am
Location: France
Callsign: condor2,F-ENSC
IRC name: colin
Version: next
OS: various linux

Re: HUD shader for parallax correction

Postby Thorsten » Sat May 16, 2020 8:23 am

[1] The Shuttle correctly compensates lateral and vertical movements with translations, but not longitudinal movements, which should be compensated by scaling the canvas.


Actually the COAS recticle correctly does that (the HUD kinda was the first canvas HUD to work out how to correctly implement a canvas HUD, so it's an oversight).

However this solution is not flawless: because the view movements are compensated by computing a transformation in Nasal and applying it to the canvas, the correction will always be at least 1 frame late.


Technically putting quickly changing properties to a shader would also have a delay - I've seen this in a bunch of cases - Nasal or not - and I'm kinda surprised you'd not see that. As far as I know both the canvas replacement and the property to effect delay are bugs which 'should not happen'...

I believe the shader would make it easier to write the Canvas part of the HUD. Indeed, HUD elements position are typically given as pitch/yaw angles.


Sorry, I've never seen that convention. I usually see them printed out where one can measure pixels. So I don't get that argument.

***

Anyway - I think it's a neat bit of technology, but I am rather concerned about requiring a shader for things which are crucial to the usefulness of the plane...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: HUD shader for parallax correction

Postby Hooray » Sat May 16, 2020 8:50 am

Thorsten wrote in Sat May 16, 2020 8:23 am:Anyway - I think it's a neat bit of technology, but I am rather concerned about requiring a shader for things which are crucial to the usefulness of the plane...


I strongly agree with that; In fact, Thorsten previously pointed out how the shuttle's ADI ball [1] was quite a handful when it was implemented via Nasal and Canvas - at the time, I suggested to introduce a dedicated canvas element to render and transform 3D models arbitrarily [2] for performance reasons, at that point Thorsten said that something like the ADI ball would ideally be done in shader space instead - but that'd obviously leave people without shaders without an ADI ball, too - which is why he ended up using the Canvas based approach, which is inferior in performance terms.

Anyway, I guess that providing an option to make this configurable via some property/UI checkbox, might still be a nice compromise ?


[1] http://wiki.flightgear.org/Shuttle_ADI_ball
Image

[2] http://wiki.flightgear.org/Howto:Extend ... _3D_models
Image
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: HUD shader for parallax correction

Postby colin_geniet » Sat May 16, 2020 9:56 am

Technically putting quickly changing properties to a shader would also have a delay [...] and I'm kinda surprised you'd not see that.

I don't have quickly changing properties passed through the effect, in fact the only additional uniforms are (supposed to be) constant.
The vertex shader computes the 'viewpoint to vertex' vector in the model space (in fact this was already done for different reasons), and this is the only new input needed by the fragment shader, beside scaling constants.

As far as I know both the canvas replacement and the property to effect delay are bugs which 'should not happen'...

I see. Can we reasonably expect the update order between user inputs, nasal, canvas, and rendering to be the correct one for this not to occur?

I am rather concerned about requiring a shader for things which are crucial to the usefulness of the plane...

So was I... I guess it's the end of that.

However, if canvas transformations are the correct way to solve this, do you have any idea where to look in order to solve this delay? I assume it would have to do with the order of execution of the systems in the main loop, which would be fairly risky to change, or am I completely off?
colin_geniet
 
Posts: 21
Joined: Thu May 14, 2020 11:07 am
Location: France
Callsign: condor2,F-ENSC
IRC name: colin
Version: next
OS: various linux

Re: HUD shader for parallax correction

Postby colin_geniet » Sat May 16, 2020 10:03 am

which is why he ended up using the Canvas based approach, which is inferior in performance terms.

This is not directly related to the topic, however you mentioning this made me wonder: does this mean that the Shuttle takes a performance impact due to the ADI ball in order to support graphics cards without shaders support? Would such cards really be able to handle the Shuttle with usable performance?
colin_geniet
 
Posts: 21
Joined: Thu May 14, 2020 11:07 am
Location: France
Callsign: condor2,F-ENSC
IRC name: colin
Version: next
OS: various linux

Re: HUD shader for parallax correction

Postby Hooray » Sat May 16, 2020 10:29 am

Feel free to run a forum search to look up Thorsten's comments in the context of the "ADI ball canvas nasal shader": search.php?st=0&sk=t&sd=d&sr=posts&keywords=adi+ball+canvas+nasal+shader

Canvas ADI ball (shuttle) / circular clipping
Thorsten wrote:The actual bottleneck is the ADI ball canvas code. As I've tested it all needs to be done in the same frame, and the ADI should be updating reasonably fast. Also, getting the properties to display is not the issue, setting the points for canvas is - data provider ain't helping a bit.

(ultimately as I said, the ADI is probably best drawn by a GLSL shader which is actually designed for this kind of crap, then it happens at near zero performance footprint - for the time being, we have what we have).



Shuttle 2D drawing helpers / G1000 effort ?
Thorsten wrote:The ADI ball is just another procedural element being produced in there - the most complicated one by some margin, granted, but no different from a pitch ladder in concept (if you need a 2d ADI ball, you can just call the routine once and use translations).


Space Shuttle
Thorsten wrote:Clock it up then - it's configurable runtime via slider. And move ADI ball quality down to get more framerate - it's another slider.
(Graphics card is unimportant for that one, it's property IO which is the bottleneck. Canvas isn't designed for this, I suppose eventually I'll move the ADI to GLSL shader code...).



Canvas ADI ball (shuttle) / circular clipping
Thorsten wrote:The center portion of the ADI ball

Image

consists of the line grid which is generated by projecting a pattern of meridians and coordinate circles on a sphere into 2d space and then clipping the central portion out of it.

That pattern is generated in the following way:

* start from the mathematical expression for meridians and circles on a unit sphere
* find the rotation matrix corresponding to the current attitude and apply it
* evaluate the expression for a finite set of points along circles and meridians (at this stage, a resolution is picked)
* project the resulting set into 2d space
* clip to the visible portion of the circle

-> At the output stage of this, there's a long Nasal array filled with point coordinates and a flag that tells whether the plotter should draw a line between two points or just move to the next point.

I can run all these operations at high resolution without being able to detect any framerate hit.

* clear all points of the canvas path element from the last update, then write the whole set of points as determined above via .moveTo and .lineTo (at which point canvas must execute setprop() )

(Now, note that we can't assume much about the points having moved, or even being the same number - dependent on what part of the original structure survives projection and clipping, numbers and what is connected in the array is quite different - some meridians may not be visible at all,... - which makes moving points rather than drawing from scratch a very difficult exercise and which is why real time 3d rendering strategies never bother really bother to re-use parts of the scene)

Now, I can disable the update after drawing once, so I get the pure cost of rendering. Which is low.

Since doing the projection and the math is not the holdup, and since the actual rendering is not the holdup, it stands to reason that it's the property IO.

The resolution chosen for the evaluation grid is the single decisive parameter for the framerate cost. Now, why do we need a high resolution - look at the screenshot, and you'll see that clipping is not good - the circle and meridian lines sometimes do not reach the boundary circle, sometimes go beyond it. So the resolution needs to be good to get decent edge detection and this drives the numbers.

Now, we don't need the resolution in the center to represent curvature - so I've introduced another step that after doing 2d projection and clipping, it culls points in the center of the display.

The result is that while Nasal computation overhead goes up (the additional step needs to be processed), property I/O goes down (because the array requested to be written is smaller), and the net result is a decent improvement in framerate.

You can feel entirely free to use any profiling tools you like, but to me the math of performance consumption is very well established by these findings. The optimization goal is to minimize property throughput by culling as agressively early on as we can possible get away with.

(See - we agree on 'too many properties' the question is just to reduce their need - if we had circular clipping on the fragment shader level for this, we'd get by with a lot less resolution and it'd be a simple task for the GPU which, as I said elsewhere, is designed to do this kind of crap.

Anyway, the ADI ball is created as part of Nasal/PFD/p_pfd.nas using the lower-level routines of Nasal/canvas_draw.nas where the whole math of projection and culling is done. I think that calls the Tait-Bryon rotation equations which are elsewhere (probably in Nasal/rel_orbital.nas)


(emphasis added)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: HUD shader for parallax correction

Postby Thorsten » Sat May 16, 2020 11:11 am

does this mean that the Shuttle takes a performance impact due to the ADI ball in order to support graphics cards without shaders support? Would such cards really be able to handle the Shuttle with usable performance?


The Shuttle is primarily CPU limited (there's a ton of JSBSim systems running underneath, there's several hundred properties being displayed on the avionics, there's Nasal systems for slowly evolving things, there's fault detection running, there's the trajectory estimation in a separate Nasal thread,...)

Whereas rendering usually is 'light' - in Space the only task is the cockpit - which is heavy compared with other cockpits, but light compared to the usual scenery mesh with trees on it.

The vertex shader computes the 'viewpoint to vertex' vector in the model space (in fact this was already done for different reasons), and this is the only new input needed by the fragment shader, beside scaling constants.


Ah, get it - yeah, that'd avoid the problem... neat!
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: HUD shader for parallax correction

Postby icecode » Sat May 16, 2020 1:49 pm

Saying that using shaders for critical systems is a bad idea is like saying that using 3D graphics to represent the aircraft is too risky and we should instead present the user with an altitude and airspeed plot.

If you are able to run FlightGear at all, your computer will support shaders. The programmable pipeline was implemented in most graphics drivers 20 years ago, and they are a 1st class feature no different from rasterizing triangles or setting the viewport size.
icecode
 
Posts: 709
Joined: Thu Aug 12, 2010 1:17 pm
Location: Spain
Version: next
OS: Fedora

Re: HUD shader for parallax correction

Postby Hooray » Sat May 16, 2020 2:19 pm

As you know, there is a ton of legacy features/code kept around that will need to be phased out sooner or later (OpenGL core profile/ES support) - and once there is better integration for shaders/effects with the canvas system and the compositor, it's increasingly likely that more and more novel features will be implemented that way - at that point, avionics are certainly also going to use shaders, not just for "effects", but for actual functionality - the way, the canvas system is being extended to support an agradar/wxradar or other TAWS by manipulating individual pixels, makes it obvious that there are better options causing less runtime overhead - just like a tail-cam/synthetic terrain view is best implemented by providing new hooks, rather than using, and misusing, existing infrastructure that was never designed with such use-cases in mind.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: HUD shader for parallax correction

Postby colin_geniet » Sat May 16, 2020 2:49 pm

Hmm, I'm not sure what to think about requiring shaders now.

Currently, the shader will not run if ALS is disabled, which would result in the HUD being messed up. This is obviously unacceptable.
I can create a lightweight version of the HUD shader for non-ALS rendering, removing all the fancy graphical stuff, and keeping only the parallax correction which is required to have a usable HUD.
(I could also do some simplification to the latter: for instance the angle computations could be moved to the vertex shader, with the texture interpolated between the vertices. Mathematically, this is incorrect and will result in some deformation, but it seems like a reasonable tradeoff for performance.)

Now of course anyone with a graphics card which does not support shaders at all will still get a non-working HUD. But as Icecode says, the requirement to be able to run shaders is so low that this situation seems fairly unlikely (or to be more precise, not having a working HUD might be the least problem of someone running FG with a card without shader support).

My issue is rather with users who set shaders settings to the minimum. My understanding is that no shaders whatsoever are supposed to be ran in that case. So what to do?
  • Disable the HUD shader, resulting in a broken HUD? It's the obvious option, but problematic for such users.
  • Disregard their request and run the shader anyway, on the basis that it is required for functionality? Seems like a bad idea.
  • Maybe the reasonable option (but more complicated), on the long term, and assuming this gets integrated with flightgear, would be to have yet another graphic setting for 'shaders required for functionality'. But I'm probably thinking too far ahead here.
colin_geniet
 
Posts: 21
Joined: Thu May 14, 2020 11:07 am
Location: France
Callsign: condor2,F-ENSC
IRC name: colin
Version: next
OS: various linux

Re: HUD shader for parallax correction

Postby icecode » Sat May 16, 2020 3:05 pm

The new ALS pipeline under the Compositor will require shaders, i.e. there is no option to disable shaders entirely. If you prefer (and wanna be future-proof) you can make your shader ALS-only and leave the default hud for low spec pipeline users, who can switch off shaders.

And don't worry about leaving users behind. The ALS pipeline will have several quality settings, so the hardware range it can run on is large enough. The low spec pipeline targets very extreme situations, e.g. 20+ year old hardware, software rendering, etc.
icecode
 
Posts: 709
Joined: Thu Aug 12, 2010 1:17 pm
Location: Spain
Version: next
OS: Fedora

Re: HUD shader for parallax correction

Postby colin_geniet » Sat May 16, 2020 3:23 pm

Icecode GL wrote in Sat May 16, 2020 3:05 pm:You can make your shader ALS-only and leave the default hud for low spec pipeline users, who can switch off shaders.

This is the current situation. The problem here is to ensure that the HUD is still displayed at the correct position/scale when the shader is turned off.
Canvas HUDs typically have a fair amount of computations to determine the appropriate canvas transformations to place the HUD image correctly. I was hoping to get rid of them with this shader, but I guess I can keep them to have the HUD working when the shader is turned off. This may be the best solution.

I think I will still make a non-ALS version of the shader as well.
colin_geniet
 
Posts: 21
Joined: Thu May 14, 2020 11:07 am
Location: France
Callsign: condor2,F-ENSC
IRC name: colin
Version: next
OS: various linux

Re: HUD shader for parallax correction

Postby Thorsten » Sat May 16, 2020 3:40 pm

Saying that using shaders for critical systems is a bad idea is like saying that using 3D graphics to represent the aircraft is too risky and we should instead present the user with an altitude and airspeed plot.


Well, fancy that - FG supports just that in case you don't want to have 3d graphics. :D So yes, we're providing fallback options there as well. There's also people who use them...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: HUD shader for parallax correction

Postby Thorsten » Sat May 16, 2020 4:29 pm

My issue is rather with users who set shaders settings to the minimum.


First, worry about leaving users behind - it happens and they get angry. :D

My suggestion would be - do the shader solution for the ALS pipeline, use the canvas translation/scaling solution in case the ALS pipeline is not on (you have it anyway, and it's just a listener to a property to start/ stop the computation runtime).
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: HUD shader for parallax correction

Postby icecode » Sat May 16, 2020 5:35 pm

Well, fancy that - FG supports just that in case you don't want to have 3d graphics. :D So yes, we're providing fallback options there as well. There's also people who use them...


Woah, nice! I'm sure those users will appreciate a cool looking parallax corrected HUD next to their 2D plot!
icecode
 
Posts: 709
Joined: Thu Aug 12, 2010 1:17 pm
Location: Spain
Version: next
OS: Fedora

Next

Return to Effects and shaders

Who is online

Users browsing this forum: No registered users and 7 guests