FG 2018.2.2 memory management

[V12]

Many times I monitored memory consumption and found strange behaviour, some like this :
- I spawned on PHNL, memory consumption for fgfs process was 2.0GB physical RAM and 3.5GB virtual
- I took off toward KSFO, memory consumption did not changed
- I flew 200 nm from PHNL, physical memory consumption has increased to 2.2GB physical and virtual to 3.7 GB - more than 400 MB more than on PHNL, but around me was only deep blue sea without any clouds. I expected rapid memory usage decrease...
Does FG unloads unnecessary scenery chunks, or keep it in memory ? Is OK that memory consumption increment ?

My system :
Ubuntu 18.04, i5 2550, 16 GB RAM, 500 GB HDD, 2GB Nvidia GTX750

[Thorsten]

Does FG unloads unnecessary scenery chunks, or keep it in memory ? Is OK that memory consumption increment ?

Generally FG always oscillates between 'we need more efficient memory management' and ' we need better performance' - dependent on what the current HW can deliver and what features are in development.

By caching stuff in memory one can usually improve performance, by clearing memory immediately performance tends to deteriorate and memory usage tends to improve. What is done I believe is complicated in detail, but definitely memory is not automatically freed once a scenery chunk is unloaded.

And then there's also leakage, which is memory not being freed by mistake and memory usage gradually increasing over long times. Though given that people have done orbital flights for a couple of days with FG by now, that also seems to be reasonably under control at the moment.

[V12]

Thank You for explanation.

[wkitty42]

something else to remember is that if, for instance, you have your LOD and visibility settings set so you can actually see the horizon 370km away when you are at 35000ft altitude, that distance is loaded all around you... not just in front of you...

i saw some code, recently, where if you are something like LOD + 10km, then the tiles behind you will unload... i think that is used for all tiles in the 740km diameter circle around you...

note also that at the time i measured this stuff, FG would allocate at least 9.5Gig of RAM when using the above visibility settings...

in testing is a newer LOD scheme where you set you distance for the detailed LOD and the other two are deltas from that (or from each other, i forget which)... this makes it easier to set the LOD properly... especially in the GUI... i'm not yet sure what properties are used to be able to set them from the command line...

[V12]

I was 200 nm (370 km) away from nearest ground at FL600. When I was 300 nm from KSFO, fgfs used around 5 GB of physical RAM and 4 GB virtual RAM. When landed at Frisco, physical ram was 7 GB. I have Bare LOD range 270 km. After some tests with 500 km bare LOD for my FL600 flights without visual benefits, I returned to 270 km.

[Hooray]

Unfortunately, it is next to impossible to provide a single "correct" answer, let alone a comprehensive one.
FlightGear is a "moving target", and resource utilization will vary depending on your startup/runtime settings and obviously your hardware.

The majority of performance issues that I have seen are unrelated to the scenery/terrain engine and are not conventional memory leaks at the C/C++ level, but rather leaking Nasal callbacks (timers and listeners), and more often than not, these are not "leaking" in the conventional sense, but rather end up being called unnecessarily (as in, too frequently) -

We don't have any suitable tooling to watch Nasal's use of timers/listeners, but the majority of code in question is not written by professional coders, and it is using fairly low-level APIs that are not suitable to be used properly by non-coders.

Unfortunately, it is impossible for non-coders to easily tell what's going on, because the syntax and semantics of the code in question is usually correct, it's just that the code is "doing the right thing too often", and tiny coding errors accumulate over time.

It's hard to bring across the whole point to non-coders here, I once tried using this analogy:

777 freezes and FPS loss

There is a lot of Nasal/Nasal GC bashing going on, but in most cases we've seen so far, it's really improper use of lower-level APIs like timers and listeners that are not properly managed by aircraft developers - which is unlikely to show up on powerful systems, unless you keep FG running for a long time - because the only thing that is really happening is that callbacks (=code) is triggered more often than necessary.

Imagine it like configuring your mobile phone to check your eMail inbox once per hour - in FlightGear, there is no notion of a "scheduler" to do this correctly, instead "timers" and "listeners" are used to associate callbacks (=code routines) with certain events, such as a timer expiring (e.g. after a few seconds) or a certain property changing (or any combination of these, i.e. by triggering listeners to be registered when a timer expires and vice versa).

Thus, we really cannot provide a correct answer that will be correct for most people - that is one of the reasons why I have been promoting to expose more internals at the property tree level so that we can gather different metrics, to enable end-users to make meaningful bug reports, and so that power users/developers can draw more informed conclusions based on the information that is provided.

So to "answer" questions like these has much more to do with sharing and documenting the underlying thought/troubleshooting process, rather than providing a specific answer that may quickly get out of date.

In other words, what is really needed are more hooks to gather internals statistics analogous to the osg stats, performance monitor, fps, frame-spacing (ms), number of Nasal callbacks (timers/listeners), Nasal GC stats, ram utilization, vram use, swapping etc

Combined, these metrics (and a handful others) can enable even non-coders to tell if something went haywire - for instance, the infamous effect listener cache was not specific to Nasal at all, but could have been detected much earlier at the mere expense of watching/dumping the total number of listeners to the console (or even just a property) every once in a while.

Likewise, Nasal GC issues related to the way Canvas ghosts are used can be tracked by dumping Nasal GC stats to the console.

Besides, Nasal is easy to exclude from the equation at the mere cost of providing an option to disable it entirely (i.e. for troubleshooting purposes): http://wiki.flightgear.org/Howto:Disable_Nasal_entirely

For all of these, we do have patches all over the place, it just seems that these tools are only being used by people who know how to patch/rebuild fgfs, i.e. they are not integrated at all - and even people intimately familiar with Nasal/fgfs internals are sometimes surprised at how some performance issues end up being caused:

Choppy Framerate on Latest Version

I've just discovered a fault in 2018.3.1 (and next) where setting fuel freeze will cause lots of timers to be created (aar.nas) which resulted in the framerate getting progressively worse (after a long test I had 0.3fps).

This will be fixed in 2018.3.2 but for now don't use fuel freeze.

Canvas ADI ball (shuttle) / circular clipping

and in fact, the problem is that it all goes through props.nas largely,

props.nas is pure evil for high throughput. I'm surprised it's being used at all.

http://wiki.flightgear.org/Resource_Tra ... FlightGear


And then, people need to understand how to completely disable features so that they can come up with a baseline benchmark, i.e. using no complex aircraft/scenery and incrementally adding bits while comparing these stats as things are added (monitoring how RAM/VRAM/CPU and listener/time utilization change over time depending on certain settings).

http://wiki.flightgear.org/Minimal_Startup_Profile


Numerous people have been wanting to create an actual FlightGear benchmark to compare fgfs performance across different systems and profiles: http://wiki.flightgear.org/FlightGear_Benchmark


This can be considered the prerequisite for any form of system-wide "feature scaling", as mentioned by James Turner: http://wiki.flightgear.org/Feature_Scaling


777 freezes and FPS loss

What is happening behind the scenes is a bit obscure, but that has nothing to do with Nasal or its garbage collection (GC) scheme, it has more to do with the low-level nature of the functions provided by core developers to "manage" such callbacks, i.e. the settimer() and setlistener() APIs are particularly tedious to manage properly - but the de facto practice is to use those as the main building blocks to write/integrate full subsystems into the FlightGear main loop - tiny coding errors may not have much of an effect, but under certain circumstances, those coding errors will add up (i.e. over time), so that the original "task" of checking your inbox once per hour, ends up being executed hundreds of times per second - the underlying code would still be correct though, it's just the event handling code that is not written correctly, which is often the case when using reset/re-init or when changing locations - because the simulator hasn't been designed, developed and maintained with these requirements in mind.

Furthermore, there's not introspection facility provided that would allow people to look behind the scenes of the simulator to understand just how often a certain timer/listener is registered and triggered, so that the whole thing is extremely obscure - but like I said, this is not a problem inherent to Nasal coding, it also happens/happened at the C++ level - e.g. look at Torsten's fix for the effects code, which was also leaking/re-running listeners like crazy (unnecessarily) - thus, it's redundant work that is getting re-scheduled to over and over again due to buggy code - the problem is not necessarily the buggy code though (writing broken code is part of the whole process), the real problem is that we don't provide any functionality that would allow people to look at where their resources (CPU, RAM, VRAM, GPU) are utilized, so that uninformed conclusions are drawn by some people (like "Nasal is bad ... and needs to be replaced with .... ").

Thus, what you can do is to overload the corresponding APIs (extension functions) and treat the id() of the callback as the key for a hash lookup to gather data on how many timers/listeners are registered, and sample those over time.

You will almost certainly end up seeing dozens of unnecessary callbacks being invoked by certain code - even though that may not necessarily be restricted to Nasal code, like I said.

IDG A32X Development

To really understand where this is going, you need to understand what a frame is, how many frames are typically created per second (30-60), and that code running at frame rate may thus also have an impact in terms of I/O at 30-60 hz - unnecessarily, because most properties don't share that often (or by that much), and then there is also the fact that you can only detect certain changes, i.e. updating a property at 60+ hz will usually do no good, because the human eye (or computer screens for that matter), don't "update" at that rate.

Besides, looking at your own example of 3 property accesses - keep in mind the cumulated overhead of even such trivial examples, i.e. 3 property reads per frame, times 40 frames per second = 120 getprop() calls X number of screens/instruments (say 3): 360. In other words, understanding what is going on here can reduce your I/O overhead down to only doing 1/360th of the work (at best).

Understanding timers and listeners is only the very first part of the equation here, because there is so much more going on and there is a plethora of ways to screw up in your code, even though it's technically doing the right thing - it may still be doing things in a rather dumb fashion.

For instance, imagine your mobile phone was checking your inbox at 60 hz (60 times per second) - would you consider that a good use of your computing/bandwidth resources ?

Probably not, but that's unfortunately the kind of kind of coding we're seeing all over the place when it comes to people implementing Nasal/Canvas based avionics, because they usually don't understand dataflow/update requirements. So it's really a good to do some reading/research first, and maybe talk to folks like Richard, Thorsten or Stuart who created done well-performing Canvas MFDs, and compare their approaches to less-performing implementations (for instance, the Avidyne Entegra R9000 on the extra500)

[wkitty42]

After some tests with 500 km bare LOD for my FL600 flights without visual benefits, I returned to 270 km.

you'd have to be flying at 65000ft altitude to see 500km... you won't see much benefit at that distance... at best, you wouldn't see as much of the zigzag of the square tiles' edges...

i used this site as a quick look-see to figure out how high up you would have to be for the horizon to be 500km distant...
https://www.vcalc.com/wiki/vCalc/Distance+to+Horizon+at+Altitude
set the top field to 35000 and the units to feet.
set the bottom field to 0 and the units to feet.
a result box will appear with the answer... set its units to kilometers.

if you do the above with 65000 in the top field, the result is that the horizon is 503.88971662km distant...

[Thorsten]

you'd have to be flying at 65000ft altitude to see 500km...

Well, he's usually in the Concorde, the plane goes (nearly?) that high. Concorde, U-2 and SR-71 are kind of on the verge where the standard FG engine starts being rather inefficient and produces not so compelling visuals, but where Earthview even at hires isn't able to deliver compelling non-pixelated visuals yet.

it's a rather peculiar rendering regime.

[V12]

On wiki (https://en.m.wikipedia.org/wiki/Visibility) I found that maximum theoretical visibility range in ideal 100% clean atmosphere in ideal meteo condition is 296 km. At normal condition this limit is reduced to approximately 160 km in clean air at high mountains. I will make some tests with this settings.

[Thorsten]

Yes, but that value is for air at sea level - you have to do the line integral through the density profile to get the equivalent attenuation from 54.000 ft (aka, if you simply insert it, you'll get the wrong result).

(You do see thousands of kilometers from space...)

[wkitty42]

(You do see thousands of kilometers from space...)

trillions even, depending where/what/when you are looking

[V12]

Strange tile management in FG2018.3.1 Linux :



I had bare LOD at 30000 meters, but in mistake left max visibility limit on 250 kilometers. And observed this scenery. IMHO, system should not load tiles with distance more than 30 kilometers, but there are dispalyed tiles far away, probably more than 120 km. Is this behaviour normal ? Because when I set low bare LOD, I want low memory consumption. But when FG loads tiles in distance 120 km and probably more, memory consumption will be large.
How exactly works tile management ? For me, it is very strange if I take off from Paris, heading direct to KJFK, when I leave Europe, I have 8 GB used memory and this value slowly rise to 9 GB (I expect significant drop) when I 200 nm before Newfoundland with bare LOD at 270 km and max. visibility 250 km. Is possible turn off tile cache or some similar for better memory management ? It is strange, when with FSX I can make hop from Heathrow to JFK on 4 GB RAM limited 32 bit Windows with visibility limits set on 160 km and FL550 and FG needs 9 GB over deep blue sea...

[wkitty42]

scenery tile management is undergoing some significant changes in newer versions of FG... the whole LOD system has been reworked and from what i've seen, things are a better than they were... more testing is still needed, though...

[V12]

What is 'newer version' ?

[wkitty42]

currently that would be the "next" branch in the code repository...

i'm a little confused, though, because this topic is about 2018.2.2 but you mention 2018.3.1...