I need about ten seconds to come up with this example why automatic testing is more complicated than it seems (there's probably thousands more):
If you don't see a light, it's because a pixel is set to a certain value. The value will still have [rgba] in a perfectly reasonable output range [0:1] - so formal testing for a valid output range gets you nowhere.
You can test whether the point sprites as such exist - but they may not since we didn't generate them intentionally (there are, after all, options to prevent this). Formal test for sprite creation or not doesn't get you anywhere either.
In fact, whether you should see a light or not depends for many lights on whether the scene is dark or not. To do a computation of whether the sun below the horizon at an arbitrary part of the world is complicated enough that you can't probably do it with pen and paper yourself - but that's not the same as scene brightness, as you'll quickly discover when you go to sun just a bit below horizon and switch ALS on and of or change from clear to overcast skies.
So what you actually would like to test is whether the light is on when the scene is dark enough, but in fact FG does not use a global definition of scene brightness in all rendering frameworks, it doesn't even use one independent of aircaft position in Rembrandt and default and in ALS there is no cross-talk between pixels - the terrain never communicates its scene brightness to the light, they're computed independently and set up to agree if the input parameters are the same. You can't formally test by computing intermediate values at runtime because it's all parallel and without any option of cross-talk, all you can do is compare the final pixel output.
That's before taking the possibility into account that the light may be fogged...
Good luck in scripting an automatic check for just this single case - it's probably going to take you at least a week to come up with something that really covers all eventualities. Not to mention that we may want to add more parameters in the future, at which point the test unit needs to be revised.
the idea is to do a quick checklist in order to be sure that very basic features for a flight simulator are Ok, no need to test all the sophisticated features
Yes, but in my book the very basic features are
* doesn't crash immediately
* I see meaningful terrain, it is solid
* renders with plausible framerate
* I have an aircraft, it starts and flies and responds to control input
* I see a scene without visual artifacts
... and that's basically it.
Neither sound nor shaders nor surface lights feature in my list of very basic features.
Hence my question:
can you even write an exhaustive list of features that is so basic that it should be tested for after every modification and agree on this list with other users such that the end result is not 'Test everything!'?