Sometime, with Yasim... frame-rate is slow

[abassign]

Trying to understand the reason of bad performance of EFA (Eurofighter) I discovered that the problem is intimately connected to the correct Yasim configuration. I wanted to investigate the matter and discovered that many planes present on Flightgear have the same problem! Too much often the Flighgear low performance are due to planes that have a bad FDM configuration. In my experience simulation systems used for engineering has always been evident the need to put the simulation parameters in the correct way. The simulation system is not a magic box, but a complex set of programs that be driven in order to get the most rapid solution to the problem. If, for example, is the case of EFA (Eurofighter), the plane has sophisticated stability systems active, it will be difficult to properly configure Yasim in order to get the same true aircraft performance, without embarrass the system. But olso this is true for the quietest aircraft, such as heavy-metal. The 757 is an example, its low performance are due to excessive FDM load (display from Parameter "flight" of the performance monitor) due to an evident incorrect configuration.
The manual Yasim tell that each cycle of calculation analysis is about 500-1200 iterations, this is no bad... because this calculation cycles, in one seconds, are about 100-120! But if it can not find the correct solution, in slang is told that "the system converges slowly," then the FDM can be up to 10000 cycles before stopping automatically and use the last solution found, whatever it is. At this point there are two consequences: The plane is not exactly what it should do in reality and the time to do 10000 cycles instead of 500-1000 will obviously be much longer!

I quote here a few examples:

EFA (Eurofighter), is a disaster!



757: It is one of the planes heavy-metal I love the most, but it has always had speed problems, I have now discovered that the load of FDM is too high, evidently Yasim is misconfigured for that type of aircraft.



F20: The performance is no bad, but the load dell'FDM is still likely to be high, I think we can still improve a bit 'is possible to have so an improvement of at least 25-30%!



777-200: It is a magnificent heavy-metal for which the FDM, notwithstanding the complexity, it is really well done and it shows!



F14: It is a wonderful example of how a great programmer expert simulation systems can get! I saw the tables used by JSBSim, they are derived directly from the data NASA. In this case JSBSim proves its quality and its excellent realism. I think that all fighter aircraft, equipped with active control of flight, should be made using JSBSim for maximum fidelity and excellent performance.



In conclusion:
My advice is to revise profoundly the way we develop with YASim, and absolutely not to complain if the speed of Flighgear is not always the best! Often the problem is another ... Canvas often have nothing to do on the low performance of the aircraft! The problem is often the FDM misconfigured!

[Hooray]

This is interesting, but also unexpected - both FDM engines are implemented in C++ space, and XML constructs are usually directly mapped to C++ data structures and algorithms, so should normally be pretty fast, even "naive" coding constructs are likely to be performance-wise superior. So I guess there's something else going on here. I am not sufficiently familiar with YASim, but there's certainly some kind of debugging mode where you could sample/query the time required to update individual components - that should give you much more fine-grained information in terms of which XML constructs are mapped to C++ code that ends up having a massive impact on performance. In coding terms, it is certainly possible to implement inefficient algorithms - and if people don't understand how XML directives are mapped to C++ code, it is also possible to end up with slowly-performing native code.
But to see if that's really the case, I'd suggest to review those YASim FDMs, and ideally run them in standalone mode first to see if the problem persists there.
Otherwise, it is perfectly possible for FG-space code (think other subsystems like Nasal, Canvas etc) having an impact on FDM performance, too.
So first of all, you should check what YASim standalone performance is like or if the problem seems specific to YASim running inside FG.

If you're able to build from source, one simple way to get run-time profiling information is using the built-in profiler: http://wiki.flightgear.org/Built-in_Profiler
By default, this would give you a comprehensive view of the whole process. Once you have confirmed that the problem seems to be in FDM space, I would suggest to run the whole thing just via the FDMShell ctor/dtors. If you can confirm the problem persisting there, I'd suggest to directly add the profiling statements to the YASim ctor/dtor.


Your theory could be spot-on, but I'd like to see some evidence first - for instance, the "events" subsystem is also showing up there, and "events" is typically just an alias for "Nasal timers" - equally, the GUI itself is showing up due to the performance monitor - I'd suggest to enable the system monitor but inspect those results using the property browser, or even better, using the web based GUI (or telnet) which would help remove GUI-related performance issues.

In summary, YASim FDMs tend to use lots of custom Nasal code to extend the FDM (unlike JSBSim), so it isn't unlikely that Nasal-space listeners/timers will affect the FDM, too - especially if the corresponding Nasal code is timers/listeners to get updated at FDM rate. Equally, it is possible that there are leaking callbacks running, registered via timers/listeners that don't properly manage already running callbacks.

If you're able to build from source, I can provide more specific advice to patch FG to see if those issues are due to rogue Nasal scripts, and/or due to certain FDM constructs.

[Buckaroo]

The manual Yasim tell that each cycle of calculation analysis is about 500-1200 iterations, this is no bad... because this calculation cycles, in one seconds, are about 100-120! But if it can not find the correct solution, in slang is told that "the system converges slowly," then the FDM can be up to 10000 cycles before stopping automatically and use the last solution found, whatever it is. At this point there are two consequences: The plane is not exactly what it should do in reality and the time to do 10000 cycles instead of 500-1000 will obviously be much longer!

I suspect you misunderstand YASim solutions. YASim calculates a "solution" once, when the FDM is initially loaded. This produces static values that are used for the entire Flightgear session. They are not re-calculated during the session. YASim either succeeds in providing reasonable values or fails, in which case it reports "Solution failed to converge...". There's no middle ground. If YASim fails to find a solution after 10000 iterations, then it doesn't use the last set of values, it simply dies-- your Flightgear session will never begin.

I haven't studied your findings-- frankly I don't know enough about what those report values actually represent to comment. I would go with what Hooray says and make sure that tests are actually representing YASim performance and not a combination of other factors.

It's true that YASim is generally more computationally heavy than JSBsim, though results will vary with the aircraft. Also many YASim FDM's are configured with elements that don't contribute significantly to fidelity but take up valuable processing resources. But like Hooray suggests, other things are more likely to be bigger resource consumers.

-Buck

[Thorsten]

I don't see how you judge FDM performance. The for the 757 you write load of FDM is too high and it has a mean value of 0.60 ms per iteration shown. For the F-14b, you comment on its excellent performance and we find the flight subsystem comes to a mean value of 0.59 ms - so it is 0.01 milliseconds faster and that's the difference between 'too high' and 'excellent performance' ? The F20 has a mean value of 0.34 ms, i.e. much better than the F-14b - yet you see room for improvement?

What numbers are you looking at?

There's vastly differing scenery in the background - the observed framerate is most likely driven by the different graphical environments and has nothing to do with the FDM. Computers from 20 years ago could solve FDMs in real time, in computational terms an FDM is perhaps 0.01% of the computing power rendering at high settings consumes.

[Gijs]

The "flight" subsystem does not only monitor "pure" fdm work, so it isn't necessarily a good indicator of the fdm's performance. See this post for an example where some nasal code slowed it down considerably viewtopic.php?p=214714#p214714

[abassign]

Very interesting! The problem may be related to listener, which are used NASAL layer, which interfaces the FDM. Indeed NASAL operates normally with 20-40 fps while the FDM operates at 120-130 fps (equivalent). If the listner is tied with FDM can then trigger an interrupt and "wake up" the code NASAL with a frequency on average 5 times faster, at this point I wonder where is this code NASAL? You can find it? Another question: the parameter "Flight" becomes high because it stops waiting for a response from the code NASAL and then it would explain why he has to increase in this way? If true this is obviously not a problem of convergence dell'FDM to the solution, but code NASAL true?

Imagined instead a different mechanism for the FDM that has to interface with the system. That they executed the cycle regardless of code NASAL, who questioned him when he woke up 20-40 times per second. If not, it means that the problem is perhaps more easily solved and could be written "guidelines" to avoid these problems.

[Hooray]

like I said previously, it would make sense to check the situation by running the corresponding FDMs in standalone mode, to exclude any FG-level subsystems/dynamics.

EDIT: Yes, you can patch the C++ code to maintain a list of callbacks invoked via timers/listeners and log everything to the property tree/console by printing out the file/line number where each callback was registered - I previously posted a patch doing this, so you could run a forum search - it would provide you with a list of file names and line numbers, including SGTimeStamp based timing for each running callback. There's another patch identifying repeatedly registered callbacks, too - by checking the address of each Nasal callback against a vector of already known callbacks.

In general, Nasal has to do garbage collection, which is why you should not link Nasal code into the FDM loop. I do think that this is very much a problem that needs solving at the guidelines/best practices level, because most people don't understand the dynamics involved here - but it would be possible to provide a "Callback monitor" that shows a log of all running Nasal callbacks invoked via timers/listeners, sampling their dt, duration and frequency, as well as file name and line number. I think Philosopher has code in one of his SG branches that could help with this. Given the known performance issues with the PUI GUI, it would make sense to log things via the property tree and use a Canvas GUI dialog to visualize all Nasal callbacks.

[abassign]

An example, I removed any reference in the EFA (Eurofighter) code NASAL and I renamed the directory that contains the procedures NASAL. In this way the system starts, but they do not work some parts of the plane. When I open the Performance Monitor and I observe the "Flight parameter" with value 121 total/ms 42 fps (the scenery is easy and the size little (800x600)). I run the Yasim with this file and I fount this: "Solution results: Iterations: 351"

I change the <approach speed="130" aoa="9"> to <approach speed="30" aoa="9"> and restart the test.

In the system monitor the flight param is now 21 total/ms (123 iterations) ! Is a very good value, and if I run Yasim the "Solution results: Iterations: 351" is identical!

So where is the problem? NASAL code does not seem to be the cause, because it was all off, a simple change of a parameter in Yasim configuration file has made the process faster than 5 times! And the fps is increased, at this point I can not understand!

[Hooray]

what you did there works, but only for aircraft-specific Nasal code residing in that folder - I am not familiar with the aircraft and how it is structured.
But to check Thorsten's suggestion, you can use draw-masks to disable rendering:

http://wiki.flightgear.org/Troubleshoot ... up_profile

Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the Property browser:
--prop:/sim/sceneryloaded-override=1
--prop:/sim/rendering/draw-mask/terrain=0
--prop:/sim/rendering/draw-mask/aircraft=0
--prop:/sim/rendering/draw-mask/models=0
--prop:/sim/rendering/draw-mask/clouds=0

To check your own theory, please consider running YASim in standalone mode with just the FDM. That is the only reliable method to see if there are FDM-level issues involved or not.

[Thorsten]

So where is the problem? NASAL code does not seem to be the cause, because it was all off, a simple change of a parameter in Yasim configuration file has made the process faster than 5 times! And the fps is increased, at this point I can not understand!

Look at your own data before you post anything. Do a few numbers.

Let's take the Eurofighter screenshot. You have 25 fps on average in that one. This is the reason everything that tracks the graphical frames undergoes 26 iterations, everything that is FDM relevant is kept at ~125 fps no matter what instead.

So, the mean submodule time, multiplies with the framerate, should give you the total consumed time of the subsystem each second. That checks out. Now - sum them up - you will find about 390 ms. So, each second, your CPU takes just 0.39 seconds computing through the subsystems. Why is your framerate then not 60 fps - we'd use 0.94 seconds of CPU time every second (less in fact, since we don't scale up the submodules running at FDM rate)?

The reason is that your graphics card is also busy crunching to fill the framebuffer, and the graphics card happens to need about 0.04 second per frame, or in other words the GPU computes the full second every second. That's the bottleneck on this data sheet - not systems, it's the rendering. Your data shows this very clearly.

So, if the flight subsystem would take another 400 ms total time per second, i.e. triple its needs, you would still be limited by the rendering. If the flight and event subsystem for some magical reason would use no time at all, or if you would have eight CPU cores and perfect parallel computing, you'd still not get more than 25 fps. Only of the flight subsystem would grow above a total of 850 ms, it would be able to slow down the framerate below 25.

In other words, whether flight or events consume more in one plane than in another plane is irrelevant as long as the total is faster than your graphics card does its job. Whether you have multiple CPU cores or not is irrelevant as long as the GPU is the bottleneck. That's the story your numbers tell.

[Hooray]

to see if that's the case, use the draw-masks mentioned above, and view the osg on screen stats - ideally also posting screen shots of those showing fps/ms: http://wiki.flightgear.org/Howto:Use_the_system_monitor

[onox]

What numbers are you looking at?

@abassign: could you answer Thorsten's question please?

If you want better fps: disable those 3D clouds, then disable the 2D clouds using the draw mask, disable the trees, disable ALS (sorry Thorsten ), disable the fog and reduce the visibility to at most 1 tile. Then I'm sure you will get a better fps.

[abassign]

@Onox

@Thorsten is a great developer of OpenGL code and is doing a wonderful job with the ALS! But I believe that many problems that complain of FGFS are due to problems that can be solved. Meanwhile I want to anticipate that the developer EFA (Eurofighter), reading this post has analyzed more carefully the code NASAL and found that the problem was related to an error in the NASAL code.
I await further details from him in order to write something on the Wiki. I think it's important for everyone and for the quality of FGFS this work methodology. I think often we complain about the performance of the program, but then it turns out that the problem is maybe in the code of the aircraft with which we are flying. The examples that I put just need to try and understand where must go look for!

[KL-666]

Not knowing much about fg coding in general, just generally speaking, a scripting language is not very efficient. So i wonder why fg leans so heavily on a scripting language like nasal.

Kind regards, Vincent

[Hooray]

Not knowing much about fg coding in general, just generally speaking, a scripting language is not very efficient. So i wonder why fg leans so heavily on a scripting language like nasal.

1) So far, we're lacking hard evidence to prove that this is related/unrelated to Nasal - however, that wouldn't be necessary had the OP followed the instructions we posted.
2) Scripting languages don't generally need to be inefficient (c.f. V8)- however, Nasal definitely is a "niche" language that is not particularly well-maintained. Attempts by others to contribute patches/improvements have been generally treated unfavorably by core developers.
3) FlightGear in general doesn't rely heavily on scripting at all - however, there's a bottleneck when it comes to development, especially core development. You only need to look at core development vs. fgdata development over the course of last couple of years to see that there is a problem - or even just the number of core related features that never made it into fgfs (e.g. look at osgEarth support or the radio propagation code to name just two)
4) Nasal is extremely accessible, people don't need to be proficient coders to write working code - equally, getting Nasal code committed is usually fairly easy, too
5) in general, people writing inefficient/slow Nasal code would not necessarily be in a good position to write much better C++ code - however, C++ just happens to be much faster obviously.
6) with Nasal, there is basically no deployment challenge
7) the way Nasal is integrated with FlightGear, it is also unable to provide its full potential due to architectural FlightGear issues - those would apply just as well using another scripting language
8) equally, even skilled Nasal coders need to re-invent the wheel frequently, because very few core developers are actively seeking to expose existing C++ functionality to the scripting system - which is the reason for having so much redundant code when it comes to scripted FDMs, autopilot/route manager code and other code for which we have exceptionally-clever C++ code in SG/FG, but which is inaccessible to fgdata developers

I actually once put a branch together where the Nasal engine would be completely disabled for performance reasons - having such a mode could still be useful for troubleshooting, and those wanting to use other languages - but in general, the problem is not so much Nasal itself, but the lack of maintenance and the lack of willingness among core developers to nominate other contributors to help maintain Nasal, in conjunction with a crazy rate of adoption by aircraft developers and other base package contributors not wanting/able to write "native" code due to a plethora of reasons.

For a summary on the C++ vs. Nasal situation in FlightGear, refer to the wiki: Shouldn't we favor C++ over Nasal?

PS: Assuming that you are on Linux and/or using FireFox or Chrome, you are using highly-performant platform with tons of scripted code (think XUL/JavaScript). Application level scripting is extremely common and useful and may sometimes simply not be as obvious. The main issue FlightGear is facing is the fact that Nasal is a niche solution and not widely used. However, even supporting Python, Ruby or Lua would bring its own challenges - not just in terms of enormous flexibility/momentum (community), but also due to the sheer number of optional dependencies - i.e. all of a sudden, aircraft might need their own shared libs. Equally, all the architectural issues at the core level would not be magically solved by replacing Nasal with a better-maintained solution like V8 or Lua - i.e. main loop synchronization and threading would still be a huge undertaking (think extension functions). Basically, some kind of IPC mechanism (like HLA) is the best option for having scripting running in a separate thread/process with explicit synchronization. Ripping out/replacing Nasal would be possible - in fact, having a startup mode that boots a subset of FG without any Nasal would be a very good thing to have for regression testing purposes - but otherwise, FG without any scripting would not be a good thing - just look at the degree of fgdata coding vs. core development, so FG would be severely crippled. Replacing Nasal with a more mainstream language like Python or Lua would be a cool thing from a community standpoint, but it would make all the architectural/core development related issues even more prominent, and that probably very soon. Whenever there's very little feedback/interest or action from core developers to implement a certain features/ideas directly, scripting is the most natural approach - which is why we have features like the Bombable add-on and the local weather system, optional stuff that core developers didn't need to get involved in.

Generally, there are very few core developers writing non-aircraft specific Nasal code regularly - those who do occasionally do so because their code doesn't necessarily belong into the fgfs core. But otherwise, most Nasal developers are not core developers.