Board index FlightGear Development Nasal

Running Nasal in a separate thread for better performance

Nasal is the scripting language of FlightGear.

Running Nasal in a separate thread for better performance

Postby V12 » Wed Nov 20, 2019 8:47 am

Split off from the topic A320-family development. Quotes added for context.


legoboyvdlp wrote in Tue Nov 19, 2019 9:08 pm:Hi,
Anyone have any evidence that the FIX / VOR / NDB layers are hidden by the respective ND being above 40 nautical miles? This makes sense to avoid long delays if you accidentally move the knob, but I can't find such a reference in the FCOM. Anyone got any information on that?

merspieler wrote in Tue Nov 19, 2019 10:30 pm:Note that it IS realistic, that it takes a bit of time. During that time RANGE CHANGE or MODE CHANGE (when changing mode) is display.

You see it in this video (best i could find on a quick search...)

legoboyvdlp wrote in Wed Nov 20, 2019 12:07 am:Yes but I don't think the real world freezes completely!

I can't find any reference to a 40nm limit so will probably remove it.

merspieler wrote in Wed Nov 20, 2019 2:14 am:Can't we get the loading out of the "real world" and in a background thread?

Not, FGFS is singlethreaded :( . All NASAL things should be executed in other thread than main loop. FDM should be in another thread.
Last edited by Johan G on Mon Nov 25, 2019 7:05 pm, edited 1 time in total.
Reason: Split off from the topic "A320-family development". Quotes added for context.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: A320-family development

Postby legoboyvdlp » Wed Nov 20, 2019 9:19 am

You can run Nasal functions in their own threads but I think the fix lookup etc being C++ probably still will freeze the sim; but I can try it. Unfortunately there's no documentation at all of this threading feature and it can and will crash FlightGear if used unwisely.
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: A320-family development

Postby V12 » Wed Nov 20, 2019 9:24 am

It will be nice to have separate NASAL space. It is interpreted language with all negatives resulting from it. Even better should be dedicated thread for FDM.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: A320-family development

Postby merspieler » Wed Nov 20, 2019 10:01 am

Isn't the FDM already an own thread? At least it is possible (eg. be running jsb independently.... not sure tho, what it does automaitcally)
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: A320-family development

Postby Thorsten » Wed Nov 20, 2019 10:14 am

You can thread out Nasal, but you can not interact with the property tree in a separate Nasal thread in any way. So this makes sense if you have a long computation that should not drag a frame in unneeded ways and is difficult to split across frames (orbital targeting is run in a separate thread) but it does not help speeding up e.g. a canvas display which needs to do property I/O (and usually has rendering of it as main bottleneck anyway).

(Otherwise, please disregard what V12 says - it's basically a couple of red herrings based on reading PR announcements - the main chunk FG computes is rendering, whether you split the few percent which are not rendering into separate threads is usually not that relevant (and given FGs general unplanned architecture might even slow things down at the sync points).)
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A320-family development

Postby Hooray » Sun Nov 24, 2019 3:23 pm

V12 wrote in Wed Nov 20, 2019 8:47 am:Not, FGFS is singlethreaded :( . All NASAL things should be executed in other thread than main loop. FDM should be in another thread.


Flightgear does uses multiple threads, Nasal scripting is not run in one of those however - for the reasons that Thorsten outlined.
It is trivial to run Nasal in another thread, and even to thread out algorithms using Nasal.
Nasal itself was designed with thread-safety in mind, by an enormously talented software engineer with a massive track record doing this kind of thing (background in embedded engineering at the time).
FlightGear however was never "designed" like Thorsten alluded to, rather its architecture "happened" by dozens of people over the course of almost 2 decades meanwhile.

The bottleneck when it comes to threading in Nasal is indeed FlightGear, the very instant you access any non-native Nasal APIs, i.e. anything that is FlightGear specific (property tree, extension functions, fgcommands, canvas) - the whole thing is no longer easy to make work correctly, without re-architecting the corresponding component (think Canvas).

In the case of Canvas, it would be relatively straight-forward to do just that, by introducing a new canvas mode, where each canvas (texture) gets its own private property tree node (SGPropertyNode) that is part of simgear::canvas, at that point, you can also add a dedicated FGNasalSys instance to each canvas texture (Nasal interpreter), and that could be threaded out using either Nasal's threading support or using simgear's SGThread API.

Obviously, there would remain synchronization points, where this "canvas process" (thread) would fetch data from FlightGear (properties) and also send back its output to FlightGear (aka the final texture).

Other than that, it really is surprisingly straightforward to come up with a thread-safe version of the Canvas system by making these two major changes - the FGNasalSys interpreter would then no longer have access to the global namespace or any of the standard extension functions, it could only manipulate its own canvas property tree - all I/O between the canvas texture thread (Nasal) and the main loop (thread) would have to take place using a well defined I/O mechanism, in its simplest form a simple network protocol (even telnet/props or Torsten's AJAX/mongoose layer would work "as is") - more likely, this would evolve into something like Richard's Emesary system.

Like Thorsten said already, you cannot "simply" thread out "all nasal" without either changing all existing Nasal code or without re-architecting FlightGear along the way.

Based on my own understanding of FlightGear, its main loop and the scripting layer, the most promising way forward would indeed be to tinker with a new addon-mode where scripts could be run inside such a sandboxed environment, using a background thread. This would be akin to firefox web extensions, that basically hit the same restrictions because of the proliferation of javascript in browsers - so, this kind of model has been demonstrated to work: one background thread for the work, and main loop scripts for the interaction with the rest of the environment.

This kind of thing can be worked on without breaking things, and it is largely facilitated by bugman's unit testing work, i.e. being able to start independent instances of the Nasal interpreter and test these outside the sim.

Once you are able to do just that, you can also easily take FGNasalSys and come up with a stripped-down version to remove all the stuff that makes such an instance thread-unsafe, and re-add what's useful later on. Probably, using some kind of RPC/IPC mechanism - socket I/O for starters should do.

The very moment you see bugman making reports about testing Nasal standalone in conjunction with certain FG APIs, all the building blocks will be in place.
A new addon mode/version could be added to support threaded addons, which is a no-brainer to do, because it cannot break anything, since we don't have any threaded addons yet.
And at that point, it would also be trivial to tinker with a new canvas mode, that has its own private property tree and its own private Nasal instance.

This is a really low-hanging fruit to be honest, and it's straightforward path to provide FlightGear with better threading support, so that anything involving new Nasal work, can be made to live inside separate threads, i.e. using such addons or canvas textures that are updated asynchronously, and which are only synchronized at certain time steps.

In addition, from a canvas standpoint this would provide for an excellent mechanism to bring unit testing to canvas-based avionics, because those can then trivially be executed outside the main loop, so that we could even run a batch job on the build server to create screen shots of avionics (say a PFD or ND) purely based on hooking them up to a pre-recorded flight or some other stored state vector containing all the properties/data.

Just running "all of Nasal" outside the main loop is going to be much more work, than being smart about it, and by preparing the hooks to thread out the interesting stuff, and provide an infrastructure to port/implement new features in the future.

Such a modified/modernized Canvas system would then contain its own private property tree for each instance and its own scripting interpreter (context), which would mean that it could even be compiled into a standalone executable, and even be executed in a headless fashion:

http://wiki.flightgear.org/Canvas_Devel ... FlightGear
http://wiki.flightgear.org/FlightGear_Headless
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A320-family development

Postby merspieler » Sun Nov 24, 2019 9:26 pm

Hooray wrote in Sun Nov 24, 2019 3:23 pm:This is a really low-hanging fruit to be honest


Then go ahead an grab it :P
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: A320-family development

Postby Hooray » Sun Nov 24, 2019 9:57 pm

The point was, threading out all of Nasal is not trivial at all - however, modifying a handful of subsystems to allow future features to run outside the main loop, would be relatively self-contained task.
If you have ever done any C++ programming for FlightGear, you realize that there is a thing called the global property tree, and that there is a single global scripting interpreter. The bottleneck when it comes to Nasal and Canvas is unnecessary, because the property tree merely serves as an encapsulation mechanism, i.e. strictly speaking, we're abusing the FlightGear property tree to use listeners that are mapped to events, which in turn are mapped to lower-level OSG/OpenGL calls - which is to say, this bottleneck would not exist, if a different property tree instance were used.

This, in turn, is easy to change - because during the creation of each canvas, the global property tree _root is set, which could also be a private tree instead.
Quite literally, this means changing 5 lines of C++ code to use an instance-specific SGPropertyNode_ptr instead of the global one.

At that point, you have a canvas that is inaccessible from the main thread (which sounds dumb, but once you think about it, that's exactly the point).
So, the next step is to provide this canvas instance with a way to access its property tree, which boils down to adding a FGNasalSys instance to each Canvas - that way, each canvas texture would get its own instance of SGPropertyNode + FGNasalSys

Anybody who's ever done any avionics coding will quickly realize that you still need a way to fetch properties from the main loop (think /fdm, /position, /orientation) but that's really easy to do using the existing infrastructure, you could really use any of the existing I/O protocols (think Torsten's ajax stuff), and you'd end up with Nasal/Canvas running outside the main loop.

The final step is obviously making the updated texture available to the main loop, but other than that, it's much easier to fix up the current infrastructure than fixing up all the legacy code ...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A320-family development

Postby merspieler » Sun Nov 24, 2019 10:01 pm

You seem to know what you're speaking of, so why don't you go ahead and change those "5 lines of C++"?
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Running Nasal in a separate thread for better performanc

Postby Hooray » Sat Nov 30, 2019 2:07 pm

like I said already, telling the canvas system to use another property tree (SGPropertyNode instance) is really straightforward - but at that point, it's no longer accessible to the rest of the sim.
You can easily try it for yourself, and just add a "text" element to that private canvas.
The interesting part is making that show up again (i.e. via placements).
Once you are able to tell a placement to use such a private property tree, you can use synchronize access by using a separate thread for each canvas texture (property tree).
But again, it would be a static property tree until you provide /some/ access to it - so that it can be modified at runtime, and given what we have already, hooking up FGNasalSys is the most convenient method. But all of the canvas bindings/APIs we have already would need to be reviewed to get rid of the hard-coded assumption that there is only a single canvas tree in use.

Like you said, changing fgfs to operate on a hidden/private property tree is the easy part, interacting with that property tree is the interesting part.

Also, it would be a very different way of coding, we would need to use some kind of dedicated scheduling mechanism, or such background threads might "busy wait" unnecessarily.

If you know how to build sg/fg from source (git) and how to apply patches, I can provide the corresponding pointers to get you started experimenting with such an adapted Canvas system, we experimented with it a couple of years ago, and there should still be patches somewhere on the forum or the wiki.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A320-family development

Postby Hooray » Sun Dec 01, 2019 10:07 pm

legoboyvdlp wrote in Wed Nov 20, 2019 9:19 am:You can run Nasal functions in their own threads but I think the fix lookup etc being C++ probably still will freeze the sim; but I can try it. Unfortunately there's no documentation at all of this threading feature and it can and will crash FlightGear if used unwisely.


threading in Nasal is generally not the solution to deal with performance issues, exploring the potential merits of multi-threading in Nasal requires a solid fg/sg background (or CS/SE in general).
It is far too easy to misuse threading in Nasal, due to FlightGear's architecture, which was never designed with threading in mind, in particular the scripting layer and the extension functions (property tree) are not designed with thread safety in mind.

For this particular problem (navdb look-up of fixes), it makes more sense to split-frame the loop, i.e. spread it across several frames (milliseconds).
In the case of the MapStructure/ND stuff we're using delta-queries to keep the runtime footprint low.
We also tinkered with a LOD scheme to query the nav DB based on the position/range and map mode selected, allocating all matching navaids into canvas groups that would at the same time be used for LOD purposes - that way, you can conveniently show/hide layers, and only have to deal with a really low number of symbols. Again, that's the sort of thing that the MapStructure code was designed for

So, threading will not automatically help you for the problem at hand ...
Anyway, you can find a number of discussions/examples on the forum/wiki by looking for "newthread": search.php?st=0&sk=t&sd=d&sr=posts&keywords=newthread

Property rule based road traffic
Hooray wrote:thread.newthread(CALLBACK) will start a new background thread (outside the main loop) running the function specified via callback.
In other words, you could implement an infinite loop in the compute function or use a timer to check if there is any work to be done in the queue (e.g. a vector with tasks).

The thing to keep in mind here is that you need a way to synchronize both threads - i.e. the code running in the main loop, and the one in the background/worker thread.
Imagine it like having two people collaborate - i.e. once creating (producing) a list of positions/orientations and the other one "consuming" that list.
If these two parts (threads) don't communicate properly, they may see invalid/incomplete state.

I am not really suggesting to use any of this, without having done some benchmarking first - otherwise, this could be a dead-end.
Apart from that, there are dedicated sychronization helpers available in the thread module - so called semaphores and mutexes.
For starters, it may be sufficient to use global variables - with one thread reading the variable only, and the other one only writing to it.
That way, you can implement a simple signalling scheme - e.g. the main thread would be polling (checking) the variable to determine the state of the worker thread, which would only update the variable once it is actually fiinished.

But again, don't spend any time playing with this unless you have confirmed that it's worth it - e.g. a simple benchmark would be creating a conventional Nasal function to do your current computations in the main loop (writing only to Nasal variables) - and using debug.benchmark() to see how long this takes, ideally writing everything to a Nasal vector.

Once that is working, the same function could be invoked via thread.newthread()

PS: An inifite loop would be something like this (this would never terminate:
Code: Select all
var compute = func() {
var condition = 1;
while(condition) {
# any code here would never terminate
}

} # compute()

Obviously, condition could also use an external/global variable, e.g. to implement a simple scheduling scheme that would check the task queue for any work, and if there isn't any, wait for 5-10 seconds to check again, and if there is some work to be done, do batches of that work in a background thread, spread across several frames (seconds), and the communicate to the main thread that the work queue is "full"
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU


Return to Nasal

Who is online

Users browsing this forum: No registered users and 1 guest