Trying to get shadows...

[fredb]

osgShadow works on the scenegraph, not on the Gbuffer. That's why I borrowed the code but didn't use it directly. Standard shadow map is an interim solution. Final goal is CSM (which is not implemented in OSG) including a variant that prevent shadow border to 'dance'.

[Zan]

Shadow volumes have some pro sides when compared to shadow maps, and most important is the resolution. Shadow maps always have limited resolution, but with volumes you can look near and far and still have sharp border (buffer/float precision limits this, though). And they can be used with deferred shading too.

But I think it might be in vain if Icecode continues his approach, as it seems to have some problems for which there is no solution yet. At least that's how I see it. But maybe we could use some of his work for light volumes in the deferred shader? And what about systems not supporting deferred shading? I think we might want some kind of shadow system for those too?

If you fred get the basic system to work, and maybe clean enough so that it can be commited to GIT, we can also experiment with fourier shadow maps and other methods, which I think is the most important aspect.

Zan

[Thorsten]

How well does this generalize to cloud shadows which can't be shadows of the 'real' object (as this is just a stack of textures)?

[fredb]

I imagine that drawing Cb cloudlets/sprites facing the sun during the rendering of the shadow map will give cloud shadows. I am not there yet but worth experimenting.
I don't know for 2d layers because transparency is not an option. We would have to set an alpha threshold to render them in the shadow map

[fredb]

But I think it might be in vain if Icecode continues his approach, as it seems to have some problems for which there is no solution yet. At least that's how I see it. But maybe we could use some of his work for light volumes in the deferred shader? And what about systems not supporting deferred shading? I think we might want some kind of shadow system for those too?

The only value I see to shadow volumes is to build light shaft (god rays), in complement to the scattering shader

[icecode]

The only value I see to shadow volumes is to build light shaft (god rays), in complement to the scattering shader

No, god rays are better as a post-processing effect. It would be really, really CPU intensive with a geometry approach. (God rays is in my TODO list too BTW).

But I think that none will mind if we have two shadow algorithms to choose from, right? When Fred finishes this, I can try with shadow volumes again, but with new deferred rendering.

[icecode]

So a week ago I decided to have a look at this project and see if 5 years made any difference. Well, it turns out they did.
I've been following Thorsten's progress with ALS and it amazes me how much it has changed. It started with a skydome shader and currently it's almost a standalone renderer. Since all efforts are focused on improving ALS and Rembrandt took a background place, trying to get shadows in the classical/forward is relevant again. I'm aware of 3D fuselage shadows and interior "shadow maps", but as Thorsten himself says, they are quite limited: they need to be precomputed for every aircraft and 3D fuselage shadows don't produce self-shadowing or realistic ambient colors.

Why shadow volumes?

Although shadow volumes are deprecated in today's computer graphics, they still seem more suitable than other approaches for FlightGear. The main advantage of this technique is that shadows are pixel accurate, but soft shadows aren't available. This isn't a problem since the only light in FlightGear is the Sun, which produces hard shadows anyway. The quality of shadow maps, even with more advanced techniques like cascaded shadow mapping, can only be improved with more texture memory. This is an issue in really large worlds.

How to implement them in the classical renderer?

Two options: directly via OSG or via the Effects framework.

• Via OSG. My knowledge of the OSG internals in FlightGear is limited, but I think it's enough to know this approach wouldn't be very suitable considering the maturity of the FlightGear code base. As seen in the osgShadow implementation, all branches that participate in the shadowing process (aircraft, scenery etc.) would be added as children to a new root group. So either the entire globals system in FG gets replaced or we have duplicated information (two scene graphs, the original and the shadows one).
• Via the Effects framework. In my opinion this is the more benerable approach. The only problem is that we are stuck with shaders - we can't use any CPU precomputations. This is a real problem when it comes to silhouette determination (more on that later).

So using the Effects framework we have shaders, depth/stencil buffer operations and all the scenery information at our disposal. Due to the nature of shadow volumes, we need to create geometry on the fly. Almost every modern GPU is capable of performing geometry shaders, which provide a natural solution to geometry generation on the GPU. Another (older) option, as seen on many articles, is to generate the geometry on the CPU and then extrude it using a vertex shader. This isn't a possibility for us since we don't have access to CPU calculations.

The most in-depth article on the subject is this one by NVIDIA: Efficient and Robust Shadow Volumes Using Hierarchical Occlusion Culling and Geometry Shaders. The only problem is that the silhouette detection algorithm requires adjacency information for every primitive given as input to the geometry shader. osg::Drawable geometry doesn't contain any adjacency information by default, so this becomes a real issue. As seen on this thesis around page 15, a solution would be to create a new derived class from osg::Geode that contains adjacency information and use it instead of the default osg::Geode for leaf nodes of the shadowed scene graph (the one I mentioned earlier). Again, this isn't an option since we don't want to replace the scene graph.

A brute force approach to extrude the shadow volumes is to extrude every triangle individually, instead of the object silhouette. At first glance this might look really expensive, but to my surprise performance isn't really affected. It's not the most elegant and definitely not the most efficient solution either, but it works. Even so, any feedback or ideas on the matter would be greatly appreciated.

Now that we are able to generate shadow geometry, let's setup the render pipeline. Since we need to generate the shadow volumes for every model on the scene, it makes sense to use model-default.eff as the Effect so we don't need to worry about specifying every object invidually as a shadow caster.
The Effect consists of three passes:

1. Render every object as if it was in shadow, that is only with the ambient term of the Phong shading model. This pass also accumulates the depth info of every object in the scene and clears the stencil buffer. This is accomplished by using a different renderbin than the rest of the passes, so every object runs this pass before the others. (Thank you Hooray for recopiling all of this information in the wiki, it saved my life: http://wiki.flightgear.org/The_FlightGear_Rendering_Pipeline).
2. Do stencil operations following the z-fail algorithm. This algorithm is actually patented in the US by Creative, so I have no idea of the legal implications this brings. The only thing I know is that it doesn't affect other countries, but I don't know which country FlightGear "belongs" to.
3. Add the diffuse and specular terms to the fragments that aren't in shadow (stencil=0).

This Effect generates shadows only for aircrafts because we aren't modifying anything of the terrain Effect. This can be solved by rendering the scenery in two passes (one for shadowed fragments and another for lighted fragments). Although this isn't as simple as it might seem because of how FlightGear manages the near/far camera for depth precision, it should be possible but I haven't looked at it yet.

Another much simpler option is to skip the 3rd pass in the previous pipeline and render a fullscreen quad colored in black for the fragments whose stencil is not equal to cero. This does the trick for terrain shadows without modifying terrain-default.eff but shadows look artificial (plain transparent black) instead of ambient colored. Since this is a simulator and physics are important, I decided to discard this option.

Stop talking and give me some results. You talked a lot 5 years ago as well but didn't deriver much!




So what's left to do?

Quite a lot:

• Probably the next big thing would be to allow shadows to be casted on terrain. Terrain is much more complex from what I know, specially because of the near/far cameras I mentioned earlier. Still, it should be possible.
• Fix culling issues. As described in the FlightGear Rendering Pipeline wiki article, OSG culls all geometry that isn't in view. This generates some artifacts when the geometry that is supposed to be giving shadow gets culled away. What I mean can be seen more clearly on this GIF.
• Integrate with ALS. This shouldn't be too hard, but I don't know how having multiple passes can affect ALS. Some shaders may need to be modified to allow single term lighting (ambient, diffuse and specular). Maybe some ALS guru or Thorsten himself can help with that.
• Organize render bins: render bins are a mess right now. Giving them some fixed order will help in the long run as well as fix some artifacts that may appear when the terrain supports shadows (terrain needs to be rendered first).

I apologize for revisiting such an old thread, but I thought it was best for continuity purposes. If the mods think otherwise, don't think twice when moving it elsewhere. Also this discussion might be better suited for the mailing list, but I don't plan on doing a merge request any time soon. That's why I'd like to hear any feedback anyone on the community might have.

[Thorsten]

Sweet... I guess you'll be making some friends here.

A brute force approach to extrude the shadow volumes is to extrude every triangle individually, instead of the object silhouette. At first glance this might look really expensive, but to my surprise performance isn't really affected. It's not the most elegant and definitely not the most efficient solution either, but it works. Even so, any feedback or ideas on the matter would be greatly appreciated.

I guess it's fair to say that our objects in the scene fall in two classes - possibly highly detailed airplane models and very coarse everything else (in fact, the scenery object repository rejects multi-MB sized models). So you can target your strategy accordingly.

Actually, several aircraft have a low-poly 'shadowy' volume mesh for the 3d fake shadow variant, so I would suggest investigating whether this can be utilized. Running a geometry shader on a 60 MB mesh for a high-end airplane and then extruding every triangle might be a problem, but 3d modelers can easily supply a low poly geometry for you offline.

it makes sense to use model-default.eff as the Effect so we don't need to worry about specifying every object invidually as a shadow caster.

I wouldn't vouch that everything that uses model-default.eff casts a shadow - that's really a very general effect used to render lots of things (I suspect even with some transparency, though I'm not sure).

Integrate with ALS. This shouldn't be too hard, but I don't know how having multiple passes can affect ALS.

Fundamentally not, but the way lighting information is computed and used is fairly monolithic and streamlined to strip redundant operations, so actually separating this into shaders which compute first only ambient light and then diffuse and specular light might not be so trivial.

But yeah, I can help with that.

[radi]

Sweet... I guess you'll be making some friends here.

+1

Icecode GL, post screenie #2 with the rendering dialog open and I'm sure this will be the screenshot of the month! Nice work!

[icecode]

Actually, several aircraft have a low-poly 'shadowy' volume mesh for the 3d fake shadow variant, so I would suggest investigating whether this can be utilized.

The problem with using a low-poly version of a model is that it doesn't exactly match the original mesh (as one would expect). This generates shadows where there shouldn't be any and vice versa. It's hard to introduce precomputations to "true" shadow volumes because they are designed to work for dynamic geometry and lighting. One of my main goals is that shadows blend seamlessly with FG, without any extra work from aircraft/scenery developers. Of course you can't always get what you want, but I always try to keep that in mind.

I wouldn't vouch that everything that uses model-default.eff casts a shadow - that's really a very general effect used to render lots of things

Following the premise I just mentioned, the only effect that gets applied automatically to every mesh in the scene is model-default.eff. Maybe we should have an aircraft-default.eff, building-default.eff etc. that inherit from model-default.eff? That might open some customization options for the user, who can choose the objects that cast shadows and those that don't (apart from other applications different to shadows). I have no idea if this would be detrimental to performance or even useful in the long run though.

(I suspect even with some transparency, though I'm not sure).

On the topic of transparency, right now transparent objects are rendered as if they were opaque (see the second screenshot). Transparency shouldn't be too hard to take into account via shaders, but if that doesn't work perhaps an effect similar to the noshadow animation from FG 1.0 can be used.

Icecode GL, post screenie #2 with the rendering dialog open and I'm sure this will be the screenshot of the month! Nice work!

Are these okay?

[radi]

Are these okay?

I'd say they stand a pretty good chance even without shadows!

[Thorsten]

Okay, I guess it's time to talk philosophy.

One of my main goals is that shadows blend seamlessly with FG, without any extra work from aircraft/scenery developers.

One of my main goals with ALS is that it works fast - and that means pre-computing whatever can be reasonably pre-computed and letting aircraft developers do the work before once rather than the GPU do it every frame.

Rewind a little: The reason Rembrandt hasn't really been taking off is, I think, largely performance. After running it on a gaming laptop and getting 15 fps once shadows were at the level where they weren't flickering any more, I cancelled all my plans to make the atmosphere rendering work for it. The thing is that ALS is a lot heavier on the GPU than classic or Rembrandt - for comparison, the ALS high quality terrain shader has about 15x more lines and easily 30-50x more instructions than the default terrain shader (close to 600 LOC + included files vs. 60) - so resource management is getting crucial.

Theories abound as to what the reason for the Rembrandt performance is, here's my suspicions:

* our scenegraph is very likely not particularly optimized, so we end up having to do many draws in the scene, often even with a texture sheet change (the scenery DB requires each building to be independent, so each static object will cause its own draw with a texture sheet change). Most contributors have simply no idea how to optimize 3d meshes for rendering, and others (such as myself) have some notions but lack the details. So likely this won't change a lot.

The upshot is that there's (also dependent on graphics drivers, NVIDIA seems to be much better than OpenSource or AMD/Radeon in that department) plenty of CPU overhead created when rendering FG scenes - which often gets to be the choke-point.

That overhead needs to be multiplied by the number of passes over the scene, so Rembrandt with its multiple camera passes is much more affected than ALS rendering everything in a single pass.

* in addition, lower to mid-range GPUs seem to be fairly weak in the fragment pipeline, creating a choke-point just where a deferred renderer does most of the work. Squeezing all the ALS-specific operations in there will be devastating for performance.

So, reading through your remarks my concern is simply that we'll repeat the story - different shadow technique, but still multiple passes, i.e. still the performance hit from doing more than one pass. No special choke point in the fragment shader - but a geometry shader. I've tinkered a bit with geometry shaders, and the performance impact wasn't fun back then. Maybe on high-end GPUs it's not an issue these days, but on many it will be - if they run at all (which also wasn't always a given).

Assume you implement the technique everywhere, but it turns out it is no faster than Rembrandt once you run it on everything, and there's hence no performance left to squeeze the ALS-specific operations in - what would you argue we have gained? Or, to turn it around - is there reason to believe this is much faster than a shadow map?

In the end it's a resource allocation question - you tend to need lots of infrastructure in terms of additional passes and buffers to support real shadow-casting. How important are automatic shadows for a flightsim in comparison with other visual features, and how much performance are they worth (and isn't it at some point more efficient to fake them?) And note that just like a shadow map, shadow volumes are anyway no solution for 'special objects' like trees, clouds, parachute fabric or glass.

Summarizing: Any shadow effects would have to be kept as separate techniques, to be switched on by an extra parameter in an actual implementation. We can't assume that people running a renderer will have a working geometry shader. The effect framework is designed to support this, so it shouldn't be an issue.




Some more specific remarks below:

The problem with using a low-poly version of a model is that it doesn't exactly match the original mesh (as one would expect). This generates shadows where there shouldn't be any and vice versa

Here's the secret - hardly anyone will notice whether a shadow is exact or not. Most of the time you can't even see shadow-casting object and shadow on the same screen, and if you can you can't see sun position - so you'd really need to sit down and do measurements to figure out whether a shadow is correct.

Usually the deal is that you can make it plausible with an approximation for a fraction of the cost of computing the actual shadow.

Please use e.g. Vostok-1 for your performance testing - this will give you a high-end 3d mesh with tons of vertices - I suspect this may be more problematic (for comparison, Cub.ac is 1.7 MB, the full Vostok-1 carrier exterior model is ~ 56 MB).

That might open some customization options for the user, who can choose the objects that cast shadows and those that don't (apart from other applications different to shadows).

I suspect you will need a per-object way to disable shadow-casting as well as a global way (see above), but both are no problem to code in the predicate of the technique.

On the topic of transparency, right now transparent objects are rendered as if they were opaque (see the second screenshot).

My understanding was that the volume shadow technique can't handle transparent surfaces - it'll always assume for shadow computation purposes that they're opaque (just like the shadow map in fact). Which is one of the (many) advantages of the opacity map shadows for the interior effect, because that can do semi and tinted shadows.

But please correct me if that is wrong.

[icecode]

I've read your thoughts on rendering in FG several times before in the wiki. Evidently performance is the most important factor when implementing new stuff in real-time applications. In fact, modern real-time computer graphics mostly consist of "trickery" to make polygons look real at the lowest performance cost. But as it has been discussed many times before in the mailing list, FG targets a really wide audience: from hardcore PC users with NVIDIA Titans to children passionate about aviation who can’t afford a gaming PC. That doesn’t mean people with better hardware can’t enjoy better eye-candy. The problem with Rembrandt is that it doesn’t target anyone - everyone gets low fps.

So, reading through your remarks my concern is simply that we'll repeat the story - different shadow technique, but still multiple passes, i.e. still the performance hit from doing more than one pass. No special choke point in the fragment shader - but a geometry shader. I've tinkered a bit with geometry shaders, and the performance impact wasn't fun back then. Maybe on high-end GPUs it's not an issue these days, but on many it will be - if they run at all (which also wasn't always a given).

I can confirm that the bottleneck in the current implementation is the geometry shader. Such a brute force approach has its consequences and unless that changes, performance is going to be a huge factor. On the Cub I get solid 60 fps. Once I change to a much more complex aircraft, fps drops to around 15. These are bad news, but they are expected bad news. Extruding every polygon that is in shadow is expensive and I expect this current technique to be a placeholder.

My only objective when bringing back this old topic was to prove that shadow volumes in the Effects framework could be done, a proof of concept. It didn’t take months of my time, not even weeks. I’m also a student, so any time spent investigating this topic isn’t wasted time. The topic name is “Trying to get shadows”, that is I just want to get shadows. I don’t mind how or when, I just want to explore every possibility and hopefully get the best one pushed upstream.

Or, to turn it around - is there reason to believe this is much faster than a shadow map?

Absolutely not. Shadow volumes are much slower than shadow maps. There is a reason why they are almost never used today. I think Fred said that adding shadow mapping directly to the forward renderer isn’t possible, although I have no idea of the reasoning behind that. He knows more about the subject than me so I just assumed shadow mapping wasn’t a possibility and turned my attention to implementing shadow volumes through the Effects framework.

Still, rendering a camera view positioned at the light source to a texture and somehow passing the result to the Effects framework doesn’t look trivial but not impossible either. I’m sure you know Zan tried adding several capabilities to cameras, notably render to texture, but I don’t know why they didn’t get pushed (I haven’t been following FG development in a long time).

Anyway I’ll probably try adding shadow mapping before continuing with my current implementation, just for learning purposes and “trying new stuff”.

In the end it's a resource allocation question - you tend to need lots of infrastructure in terms of additional passes and buffers to support real shadow-casting. How important are automatic shadows for a flightsim in comparison with other visual features, and how much performance are they worth (and isn't it at some point more efficient to fake them?)

Everything is dispensable if you really want them to be. In terms of rendering FG has no reason to envy modern AAA games, but as it has been said many times, we don’t have an art department. What’s a simple feature that can help make FG look slightly better and more “realistic” that doesn’t require any artistic skills? My answer was shadows. Take the ephemeris as an example. The original FG developers could have pasted a texture on the sky and called it a day, and yet they didn’t. Is it worth the performance hit? That's the user choice. Of course as developers our "obligation" is to give as much eye-candy as we can to as many people as we can.

And note that just like a shadow map, shadow volumes are anyway no solution for 'special objects' like trees, clouds, parachute fabric or glass.

From what I know ALS already does tree and cloud shadows. Glass is just assumed to be completely transparent so it doesn’t cast a shadow. Sacrifices have to be made.

Summarizing: Any shadow effects would have to be kept as separate techniques, to be switched on by an extra parameter in an actual implementation. We can't assume that people running a renderer will have a working geometry shader. The effect framework is designed to support this, so it shouldn't be an issue.

This is already done. See the rendering dialog in the first screenshot of my previous post.



So did I reach a point where “hey it can be done!” no longer suffices and I need to advance a little further if I want to get this merged? Yes, and I guarantee I’m not going to do a merge request if I get 15 fps with Vostok-1. If it’s technically impossible to do that (which shouldn’t be), well, I’ll just leave all the remarks being discussed here and the repository URL in some random wiki page so anyone who wants to further investigate this topic has a base to work on.

[Thorsten]

I think Fred said that adding shadow mapping directly to the forward renderer isn’t possible, although I have no idea of the reasoning behind that.

Shadow maps render a pass of the scene seen from the light source - all pixels that survive the z-buffer by definition are lit. But to carry that information into the pass seen from the camera requires a geometry buffer - hence the forward renderer can't do it - e.g. a stencil buffer is only useful if you don't change camera position between passes.

That doesn’t mean people with better hardware can’t enjoy better eye-candy.

That's missing my point I guess. I proud owner of a new GeForce 1080 (second only to the TitanX in performance) - but implementing all effects I would like to have (before shadows), e.g. a full terrain heightmap, a second vegetation layer, rendering fluffy clouds further out,... still saturates it.

Even if we all had three times the performance we enjoy now, I'd still think that there's more important things to do with it than have every object cast a real shadow. I think shadows are pretty unimportant in a flightsim, We'd get much more bang for the buck by improving fog, clouds or terrain further.

But that's of course my opinion only.

Then again, this matters as far as ALS is concerned. The criterion to include anything there is that it's harmless (i.e. possible to opt-out if it takes lots of performance) and (and that's more tricky) doesn't substantially affect my maintenance load of the rest of the framework.

My only objective when bringing back this old topic was to prove that shadow volumes in the Effects framework could be done, a proof of concept.

Since you've done that, I started to talk philosophy. Naturally I'm interested in where you want to go with it. Naturally I'm interested in knowing whether we have enough overlap in goals to create something together we can both be happy with, or whether you better create your own sandbox because our development philosophy is fundamentally different.

Personally, I'd be more than happy to see this as part of the ALS shadow trickery suite for self-shading the aircraft - for aircraft maintainers and users who opt into that. Ideally based on a reduced polygon shadow volume. In that role, it'd basically reside in one effect, both performance impact and maintenance load would be low.

Maybe also to cast a shadow onto the airport keep and runway - that'd affect a second shader then.

But at this point, I'm not seeing a solution I'd be happy with in which all models cast shadows on all terrain. You'd have to duplicate lots of shaders with associated maintenance issues being created.

So, that's the idea here - not to discourage you, but just to put out visions and see whether it makes sense to work together or not.

[icecode]

Naturally I'm interested in knowing whether we have enough overlap in goals to create something together we can both be happy with, or whether you better create your own sandbox because our development philosophy is fundamentally different.

Why not both? Our goals are completely different. I want to bring shadows to FG as if they were never gone, while you are content with having a shadow under the user's aircraft. Where we overlap is at the interest of bringing shadows back. The thing is your goal is more realistic (apart from less time consuming, easier to maintain and easier to implement in general).

I don't mind doing things "your way" first, that is getting "true" shadows to work for the user's aircraft with any precomputations needed to allow an acceptable performance. The truth is that a minimal percentage of aircrafts in FG have active developers, and an even lesser amount of those would bother about providing these precomputations, that's why I'm not in favor of "your way".

But anyway I don't have commit access to the official repo so I can experiment as much as I want on my sandbox.

Personally, I'd be more than happy to see this as part of the ALS shadow trickery suite for self-shading the aircraft - for aircraft maintainers and users who opt into that. Ideally based on a reduced polygon shadow volume. In that role, it'd basically reside in one effect, both performance impact and maintenance load would be low.

So now that I gave in, let's talk about what could be precomputed to help with performance issues. As I said before, precomputing and shadow volumes don't get along really well. There are several things that are performance intensive but can't be changed due to the nature of shadow volumes:

• Geometry shaders. I'm sorry for those who don't have GPUs that don't support them, but there is no other way to generate geometry on the GPU.
• Multiple passes. All shadow volumes algorithms need multiple passes to do stencil operations.
• Using the stencil buffer. You can use the alpha buffer, but I don't think that's needed in today's computers.
• Separating the phong shading terms. Either via a single shader and a uniform that says which term needs to be outputed or one shader per term, but they need to be separated.

So what can be changed to squeeze out those frames? The reduced polygon shadow volume won't work, and even if it did, I would prefer to opt for another solution because results would be kind of horrible. Since the bottleneck in shadow volumes happens at the geometry shader (and consequently increased fill time), it makes sense to optimize it as much as possible with as many precomputations as needed. I've read several papers on the matter, but they employ too many low-level instructions that simply aren't available to us in the Effects framework. I think that precomputing adjacencies to allow for a proper silhouette detection algorithm is the way to go. This could be done through a texture, where with some clever index work one could access the three adjacent normals of a given triangle. I'll see what I can come up with.