Board index FlightGear Development Nasal

the real cost of the Nasal Garbage Collector

Nasal is the scripting language of FlightGear.

Re: the real cost of the Nasal Garbage Collector

Postby Thorsten » Wed Nov 25, 2015 3:55 pm

I'm just saying... what is 'best practice' seems to depend on whom you ask.
Thorsten
 
Posts: 10640
Joined: Mon Nov 02, 2009 8:33 am

Re: the real cost of the Nasal Garbage Collector

Postby Hooray » Thu Nov 26, 2015 3:06 pm

To be fair, "best practices" may also become obsolete in the light of certain observations, i.e. new data and findings.
I think, we have seens tons of such incidents - e.g. Nasal constructs considered superior despite being very much inferior. I know you once proved my own props.nas related advice wrong.

It was easy to come up with the advice (and still is) that props.nas should be faster than its setprop/getprop equivalents, because the algorithm of caching/reusing a handle is more efficient than using a hash lookup and concatenating strings - but that does not take the Nasal overhead account.

Should, some day, cppbind be used to wrap props.nas, the opposite would likely be the case.

That would be particularly good for heavy property tree systems, like Canvas.

And Canvas itself is another good example of "best practices" that are just about to be established.

Many of these things don't matter in C++ space, because C++ is compiled and heavily optimized using the compiler, but in scripting space, such stuff may obviously add up quickly.

Just like most of us were never aware of the Nasal GC and its impact, because few people actually know how Nasal code is invoked via callbacks triggered by timers and listeners, and that the "nasal" subsystem itself is unlikely to show up in any profiles, whereas "events" and _any_ system supporting listeners can easily call tons of Nasal code, and thus, invoke the GC arbitrarily.

Adapting the GC to turn it into a generational one is not rocket science, supporting/adopting alternate schemes would also be fairly straightforward - it just seems that this isn't on anybody's todo list currently.

Thus, Stuart's HLA work is likely to provide the greatest bang for the buck, because any Nasal subsystem could immediately benefit from that
Last edited by Johan G on Sat Nov 28, 2015 11:08 pm, edited 1 time in total.
Reason: .bas --> .nas, as per PM with Hooray
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: the real cost of the Nasal Garbage Collector

Postby Hooray » Sun Feb 07, 2016 1:33 pm

AndersG wrote in Tue Nov 24, 2015 3:07 pm:Running the Nasal GC in a different thread could certainly be revisited, and if the GC updated to an incremental one it could be done without impacting the main loop much.


Now that gitorious repositories are available again, here's the latest set of patches/commits that I could find related to this:

https://gitorious.org/fg/philosophers-s ... 92f447d950
https://gitorious.org/fg/philosophers-f ... d840e453c4
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: the real cost of the Nasal Garbage Collector

Postby hamzaalloush » Sun Feb 07, 2016 2:13 pm

Wow what a great resource, considering it's Philosopher's , i'm really happy that we are considering finding the main culprit, to be honest weather or not Nasal stays or not, that woud not change my mind that it is very controlled and bisected enough that is optimized for even low level security patchs that Rebecca often post about.

I think the priority should be to fix the GC issue first, rather blowing FG to a thousand fragemen just the GC penalty, if we adopt a multi scripting approach, even thuoght i hate Nasal syntax addmitadly, i guess the one way for someone to practice with it, either through trail and errors, or the new innovate suggested way of running a seperate Nasal console with the minimum subsystem allowed.
hamzaalloush
 
Posts: 632
Joined: Sat Oct 26, 2013 9:31 am
OS: Windows 10

Re: the real cost of the Nasal Garbage Collector

Postby AndersG » Sun Feb 07, 2016 3:05 pm

Hooray wrote in Sun Feb 07, 2016 1:33 pm:
AndersG wrote in Tue Nov 24, 2015 3:07 pm:Running the Nasal GC in a different thread could certainly be revisited, and if the GC updated to an incremental one it could be done without impacting the main loop much.


Now that gitorious repositories are available again, here's the latest set of patches/commits that I could find related to this:

https://gitorious.org/fg/philosophers-s ... 92f447d950
https://gitorious.org/fg/philosophers-f ... d840e453c4


Yes, those seem identical to the patches I got from James.
Callsign: SE-AG
Aircraft (uhm...): Submarine Scout, Zeppelin NT, ZF Navy free balloon, Nordstern, Hindenburg, Short Empire flying-boat, ZNP-K, North Sea class, MTB T21 class, U.S.S. Monitor, MFI-9B, Type UB I submarine, Gokstad ship, Renault FT.
AndersG
 
Posts: 2433
Joined: Wed Nov 29, 2006 9:20 am
Location: Göteborg, Sweden
Callsign: SE-AG
OS: Debian GNU Linux

Re: the real cost of the Nasal Garbage Collector

Postby Hooray » Sun Feb 07, 2016 3:21 pm

I also have a set of patches that were intended to help experimenting with different GC schemes, e.g. to make the GC entirely self-contained, so that alternate GC implementations (open-source) could be implemented next to the existing scheme, without having to make any invasive changes - i.e. possibly by using a startup switch to select a certain GC implementation (which is how Java supports different GC implementations):

https://gitorious.org/fg/hoorays-simgea ... 6c2f6bdaaf

For instance, by moving the whole naPool stuff into the GC allocator, we can encapsulate all GC internals, and simply provide a different naGC_alloc() implementation:

https://gitorious.org/fg/hoorays-simgea ... 150d1cfcff

A generational collector could be implemented by adapting the existing scheme, which already uses naPool for different types, so that different generations could simply be represented in terms of having an array of naPools.

In other words, I think it would be a worthwhile goal to refactor gc.c accordingly, so that we can tinker with different GC implementations, without any chance "breaking" the default scheme, i.e. by adding other GC schemes as "options".

Equally, your threaded GC implementation could become a startup option that way, so that people can set a property via --prop:/sim/startup/nasal-gc-mode=threaded to turn it on, what do you think ?
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: the real cost of the Nasal Garbage Collector

Postby Hooray » Mon Feb 08, 2016 9:38 pm

another idea would be using OpenMP directives to partially introduce threading for certain parts of the GC phase - for instance, the reap() phase could in theory be multi-threaded without requiring very tight synchronization.

Another possibility would be running one marking thread for each generation/naPool - obviously, we would need to review the Globals struct to be sure that no global state is mutated, i.e or use locking otherwise.

Personally, I would be more likely to tinker with OpenMP instead of using low-level pthreads/synchronization primitives, because that would allow us to experiment with modifications to the GC scheme without changing it in any major way, i.e. the single-threaded mode would remain unmodified.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: the real cost of the Nasal Garbage Collector

Postby Hooray » Sun Feb 21, 2016 5:28 pm

Referring to: Subject: Nasal Callback tracking (timers & listeners)
Thorsten wrote:I suspect this is behind aircraft performance deteriorating steadily over a few hours.


Note that there's another issue that shows up when running the GC, without having much to do with the mark/sweep collector itself, and Andy is/was perfectly aware of it - it's that naContexts are leaking when using timers and listeners, and this can be easily observed by patching gc.c to dump the number of naContexts it is traversing in each run: https://sourceforge.net/p/flightgear/si ... l/gc.c#l35

Code: Select all
c = globals->allContexts;


Here, "c" is a pointer to a linked list of active contexts - so you can either add a counter to the while loop or count the number of non-NULL pointers in that list to see how large the number of contexts really is.

When I am using the minimal startup profile, I am typically seeing 3-5 Nasal contexts created during startup - however, once I start using Nasal timers and listeners, e.g. by opening/closing a bunch of PUI dialogs (or the Aircraft Center), you can typically watch the whole thing (number of active Nasal contexts) grow over time - it starts out at 10 contexts after a few minutes, and ends up at ~25+ after 10-15 minutes, but continues to grow - depending on the scripts that are active/running.

This is obviously adding to the GC overhead, because the GC needs to traverse/manage all these contexts.

Presumably, it will continue to grow, which will become noticeable for people using FlightGear for several hours (multiplayer/VA use).

The solution for this is much more straightforward than fixing the GC scheme itself, i.e. Andy suggested to get rid of the global context and only use temporary contexts: http://wiki.flightgear.org/How_the_Nasa ... s#Contexts

Andy Ross wrote:New nasal objects are added to a temporary bin when they are created, because further allocation might cause a garbage collection to happen before the code that created the object can store a reference to it where the garbage collector can find it. For performance and simplicity, this list is stored per-context. When the context next executes code, it clears this list. Here's the problem: we do a lot of our naNewXX() calls in FlightGear using the old "global context" object that is created at startup. But this context is no longer used to execute scripts* at runtime, so as Csaba discovered, it's temporaries are never flushed. That essentially causes a resource leak: those allocations (mostly listener nodes) will never be freed. And all the extra "reachable" Nasal data floating around causes the garbage collector to take longer and longer to do a full collection as time goes on, causing "stutter". And scripts that use listeners extensively (the cmdarg() they use was one of the affected allocations) will see the problem more seriously. (That's a feature, not a bug. Once listeners were added, scripts could be recursive: (script A sets property X which causes listener L to fire and cause script B to run) We need two or more contexts on the stack to handle that; a single global one won't work.) I didn't like the fix though. Exposing the temporary bin as part of the Nasal public API is ugly; it's an internal design feature, not something users should tune. Instead, I just hacked at the FlightGear code to reinitialize this context every frame, thus cleaning it up. A "proper" fix would be to remove the global context entirely, but that would touch a bunch of code.



Overall, this is probably one of the lowest hanging fruits when it comes to addressing Nasal performance issues, and the Nasal VM could probably be patched to use only temporary naContexts by using a coccinelle patch that looks up all uses of the global context...

For the time being, it would make sense to expose the number of active Nasal contexts in the form a counter that is added to the property tree - possibly in conjunction with the size of each memory pool (naPool).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: the real cost of the Nasal Garbage Collector

Postby Richard » Mon May 06, 2019 6:13 pm

I've been recently doing some work on the performance of the Nasal in the F-15 and as part of this I've noticed that every second or so there is a blip of about 10-20ms that doesn't appear to be in my code (after much investigation)

The real culprit for this is the Nasal garbage collection; and here are the figures

Total time taken in garbage collection taken over a period of 213 seconds after startup

  • average 24.65 msec
  • min 1.488 msec
  • max 45.34 msec

This breaks down as (average, max, min)
  • freedead time 2.14 5.14 0.12
  • mark 18.11 35.41 0.42
  • reap 3.82 4.48 0.62

As this only happens once a second or so the impact on the overall frame rate is negligible; however it is quite a severe impact to an individual frame. I know there have been efforts to improve Nasal GC previously - however the point of this post really is just to put some numbers.

All measured on my system I7-4790K
Richard
 
Posts: 693
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Previous

Return to Nasal

Who is online

Users browsing this forum: No registered users and 0 guests