Board index FlightGear Development Nasal

Nasal must go

Nasal is the scripting language of FlightGear.

Re: Nasal must go

Postby Thorsten » Sun Oct 09, 2016 4:41 pm

I'm certainly not opposed to any such efforts - I'm just not sure I'd flag that as our most pressing problems right now.

For instance, my actual problem is property I/O - I can't read/write several hundreds of properties per frame without creating a bottleneck. So it's largely irrelevant how fast the Nasal code runs, whether it's parallel or whether it's Python-driven code running on the GPU - as long as property I/O speed doesn't change, performance will be stuck right there.

And I have a suspicion a badly implemented modularization will just make this worse.
Thorsten
 
Posts: 10953
Joined: Mon Nov 02, 2009 8:33 am

Re: Nasal must go

Postby Hooray » Sun Oct 09, 2016 4:51 pm

one of the ideas repeatedly tossed is having subsystem-specific property trees which are "private" by nature.
Which basically means that different threads can safely operate on their own property tree, without stepping on anyone's toes.
Another possibility would be reviewing the property tree implementation itself (there are alternate implementations available, e.g. via boost - which is already a fg/sg dependency).
And then there's also the option to look at the actual use-cases where you are hitting this limit, and see if/how this can be optimized - anything involving Nasal/Canvas can certainly be optimized, I think I repeatedly mentioned how the property I/O involving Nasal/Canvas MFDs can be signficantly reduced.

But again, everybody seems to be hitting different limits - i.e. the difference between being CPU vs. GPU bound is down your use-case and your hardware obviously, as well as your concrete startup/runtime settings.

I think we did cover in the shuttle thread that there is an actual startup flag to sample/log all property I/O to the console - and there also is a standalone benchmark in simgear somewhere.

Given your hardware/specs (assuming that gaming notebook is still working), I'd be surprised if reading/updating 100s of properties per frame (1/60 second) should be an actual limit for the standalone benchmark. I think it's more likely some "emergent" behavior (accidental mutual dependencies) or another bug in our C++ coed - e.g. remember the effects/listener leak

I guess it would be a good idea to move this a separate discussion (I am defintiely interested) - but we need to have a self-contained test, and it would be good to learn what the standalone benchmark tells you about your system's max performance.

Besides, if you are referring to updating Canvas related properties, those may actually trigger an update (and possibly FBO redraw), so that it does make sense to look at the "backtrace" to see what is actually happening when updating the corresponding properties - likewise, FDM properties (and other tied ones) are not trivial at all.

I don't think we have any good means to understand what is going on other than actually running gdb or patching the source code

And I have a suspicion a badly implemented modularization will just make this worse.

of course, the point being "badly implemented" ;-)
Apart from that, one property tree (SGPropertyNode or props.Node) object per thread can easily be updated much more often than anything running in the main loop, which is obviously framerate bound
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal must go

Postby bugman » Sun Oct 09, 2016 6:38 pm

My Python OO implementation of the property tree is quite fast :P Give an example of a set of property updates as a standalone Nasal script, and we can have our first Python vs. Nasal in FG benchmark! Though I have a feeling it is the type checking on the C++ side that is the killer.

Regards,
Edward

P.S. Hooray, you know how it goes - flag the first post you propose to split and suggest a subforum and subject.
bugman
Moderator
 
Posts: 1698
Joined: Thu Mar 19, 2015 9:01 am
Version: next

Re: Nasal must go

Postby Hooray » Sun Oct 09, 2016 6:47 pm

IIRC, somewhere in SimGear/props there should be some kind of standalone C++ benchmark that David M. originally came up with and that Andy later on used to test his assertion that he could directly use properties (instead of properties tied to raw C++ PODs) in YASim (and he ended up being correct).

My guess is that there's too much going on that we're not aware of when Thorsten refers to property updates showing up in any significant form, i.e. layers of indirection that we aren't aware of or something else along those lines.

Regarding a standalone Nasal vs. Python benchmark of "props", I think there still should be a patched version of Andy's standalone nasal-bin that has simgear/props bindings, but need to check the canvas-hackers clone if I can find it ... however, regardless of the outcome, I highly doubt that this is relevant in the context of Thorsten's observations.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal must go

Postby curt » Sun Oct 09, 2016 8:36 pm

I haven't done any formal performance studies, but in my real time UAV flight control software, I have gone to a python-based property tree with a C++ interface for the performance critical code (like the EKF.) Most of the logging and communication code is python. The entire mission manager and task system is pure python. Going forward I envision more python and less C/C++. This system holds a solid 100hz on a beaglebone (similar to a raspberry pi) and flies aircraft successfully in real time.

The reason I haven't done detailed performance profiling of the property system is because things like logging (writing to a file) and crunching the 15-state kalman filter are by far the bigger performance hogs. And to head a few comments off at the pass, the logging is handled by a kernel thread, but at some point that thread needs to wake up and do the work of writing to onboard flash, and it's a lot of work and take a lot of time (relative to other things.) Communicating to the sensor head via uart (serial) and sending actuator commands via uart also takes up a big chunk of time. This is a different application, but property tree access happens in similar proportions to FlightGear. FlightGear's big overwhelming bottle neck of course is drawing the scene on the graphics hardware.

In my own code, I have typically gone to a mindset where I keep all the data in the property tree without binding it to C++ variables for direct C++ access ... again, the added complexity just doesn't buy much (if any) win with performance.

So all that to say, moving the property system over to a script engine outside of the C++ world (and hosting most of the important data directly in the property tree) might not be much of a hit on actual FlightGear performance. But again, that could be a huge construction project if we also took the opportunity to freshen up the property system API a bit and drop a bit of the historical cruft.

One more comment as a P.S. The property system doesn't lend itself to multithreaded or multi-process systems very well. Personally, that's not a big deal to me, because I'm not a big advocate of threading and multiprocessing as a first pass programming technique ... especially in a group project. I think of threads a little like dental work ... they are painful and you use them only when necessary. When necessary, then you go deal with it and do what you have to do, but threads are never my first choice. I have seen too many people not get what they expected all the time because they really didn't understand all the implications of threading, or didn't have strategies to test all the edge cases of their system, or maybe didn't even think to test things. I've seen 40hz loops run at 25hz when I finally convinced someone to check the timing. I've seen 100hz loops run at 200hz-4hz and everything in between ... but average 100hz over time. I've seen tons of random thread crashes over the years. The older I get the less interested I am in "ideology" and more interested in keeping things practical, simple, and straight forward. And I have always suffered from a need to understand what I'm doing before I get any energy to do it ... drives my bosses nuts.

People have discussed how to make the property system thread friendly. Hooray speaks of serializing property tree access with is exactly right ... but how to do that outside of making the property tree something like a standalone process that serves requests (almost like a web server model)? Serializing with mutex's or locks on individual property element accesses with multiple threads will almost certainly lead to instant deadlock. Serializing at the calling layer might make more sense (i.e. a module loop might be: lock property tree, read inputs, release property tree, execute main loop, lock property tree, write outputs, release lock, repeat...) But that puts the burden on every module that uses properties and it would be really easy to break as development forges ahead and new contributors come on line and different people poke at code not fully understanding how the whole system works together. Which (in my mind) sends us back towards keeping things single threaded as much as possible to keep things simple, straight forward, and avoid unintended breakage.
Aerospace Engineering and Mechanics
University of Minnesota
curt
Administrator
 
Posts: 1174
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: Nasal must go

Postby Hooray » Sun Oct 09, 2016 8:48 pm

actually, the point was having one private property tree instance per subsystem - such as the AI traffic manager having its own property tree, to which it manages access in a sequential fashion by processing read/write requests using a queue - imagine it like setting/getting properties via the props/telnet interface. At that point, certain subsystem could be having their own private property trees, that would never show up in the global tree in any directly accessible fashion other than going through some form of IPC/RPC, even if it's just as simple as using the existing telnet or httpd code to update/read a property, that will force people to think about process/thread boundaries, because there is no other way to access the corresponding property. Thus, a subsystem like the AI system could have its own private property, just like quite a few other subsystem (think Canvas) - any subsystem wanting to deal with those systems would have to go the extra mile and dispatch an event/signal to get/set a property from the corresponding system, which like you say, would work analogous to a webserver handling requests one by one.

Note that this concept is already in use in some parts of FlightGear, e.g. the multiplayer system, but also other I/O options (think generic protocol, telnet) - it's just that we don't commonly expect other C++ code running in the same address space to go that extra mile to fetch/set a bunch of properties.

But like I said previously, I am not convinced that property I/O is really a bottleneck on modern hardware - if anything, it has to do with other stuff running in the main loop, but an idle core (i7 I think in Thorsten's case?) should be able to satisfy hundreds of updates per 1/60 second easily - keep in mind that we're dealing with PODs here, i.e. ints, doubles, floats and a bunch of strings.

So, it would be interesting to come up with a really reduced test case that reproduces the problem Thorsten is seeing and than actually look under the hood to see what is going on.

Anyway, if we cannot find the original benchmarks, it should be trivial to write a piece of C++ code that samples how many property reads/updates it can easily process per second, and per frame on average hardware, and then take it from there - i.e. by profiling the code. But I really believe any such issues to be due to other abstraction layers (think Nasal, listeneers) and potential "indirection accidents" (think the original effects leak that Torsten nailed down).

I believe the real issue is that we don't have any tools to measure easily what is taking place behind the scenes, i.e. seeing an actual backtrace would be interesting under some circumstances - just like the number of total updates per second, and number of registered listeners, and how that relates to GC objects (SGPropertyNode ghosts)

Anything specific to Nasal/Canvas can be addressed over time, we have done that repeatedly (remember those KNUQ taxiways ... :P )
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal must go

Postby bugman » Sat Oct 29, 2016 3:32 pm

As requested, I have split out the ADI ball discussions:

Regards,
Edward
bugman
Moderator
 
Posts: 1698
Joined: Thu Mar 19, 2015 9:01 am
Version: next

Re: Nasal must go

Postby Lydiot » Sat Oct 29, 2016 3:55 pm

Hooray wrote in Sun Oct 09, 2016 4:51 pm:Another possibility


Just wanted to say that I greatly enjoy reading your posts. I think you're taking an admirably productive and creative approach to this. It's very nice to see.



Can I also just throw out a though for you to comment on: On the topic of the usage of cores/threads, my understanding of the CPU industry is that the rate of increase of cycles per seconds has dropped in favor of more cores (speaking of average CPUs purchased). So to me it would seem that if we (well, "you" really) wanted to take advantage of both future but also current computational capacity utilizing cores efficiently should be the thing to do. After all, for how many more years will we see people whose "average" computers aren't really multi-core computers often running two threads per core?
Lydiot
 
Posts: 985
Joined: Tue Oct 22, 2013 10:50 pm

Re: Nasal must go

Postby Hooray » Sat Oct 29, 2016 4:57 pm

That is a long-standing suggestion and challenge - dating back to the mid 2000s - if you are into a non-heated view of the situation, I would highly recommend reading the following paper, authored by several long-term FlightGear contributors back then (none of whom were/are any core developers): http://wiki.flightgear.org/Technical_Re ... _Simulator

In layman terms, the thing as it stands is that FlightGear is/was designed with a mainloop assuming a really fast single core - so that tons of code has been added without any of it being written with multiple cores in mind. As time moved on, more and more people suggested the use of multi-threading (for all the reasons you mention), many of whom spoke up on the devel list - and Curt made some rather elaborate statements why threading would be problematic in a real-life application like FlightGear.

Ultimately, what took place is that people politely decided not to argue with Curt, but still added additional threads to FG - unfortunately, Curt ended up being right, we're seeing tons of threading related problems these days, i.e. segfaults/crashes that can definitely be linked back to FlightGear related threaded code.

The thing is, there is a fairly massive legacy code base that was written with the fixed/single-core assumption in mind, and as things moved on, more and more threading was added - especially in the light of the PLIB/SG->OSG migration - OSG can be extremely aggressive, and good, at multi-threading - but for that to work properly, you need to make use of certain coding patterns - absent that, you don't automatically benefit from any of that functionality, but you add semi-functional (=crippled) functionality that sometimes works, and sometimes just crashes - see the warning at: http://wiki.flightgear.org/Howto:Activa ... PU_support

Multi-threading is something that is extremely difficutl to get right, and basically impossible to add well after the fact - especially in an unorganized codebase like FlightGear with tons of subsystems treating the property tree as a global dump space for all sorts of state.

The OSG port was initiated over a decade ago, and has never really been completed - so there is tons of legacy functionality that people care about (as in legacy code) that simply cannot be removed, but that is massively incompatible (conflicting) with the way OSG wants to see things done - this is severely limiting FlightGear's evolution.

Nasal is definitely contributing to this, but it's far from the only culprit - I have posted patches that demonstrate how Nasal can be entirely disabled, but there is tons of other legacy code that ends up running in the main loop where it simply doesn't belong.

To some extent, the Canvas system can help us getting rid of certain code, such as the legacy 2D panels code, hard-coded MFDs, HUDs or GUI - because the Canvas system can reimplement those features using a fraction of the size of the legacy C++ code, while internally falling back to just a handful of so called Canvas "elements", all of which are using OSG natively - which is why a number of core developers originally suggested using this approach to help unifying the 2D rendering back-end: http://wiki.flightgear.org/Unifying_the ... via_canvas

If this should end up being pursued/picked up again, it would be much easier to get rid of tons of legacy rendering code, that can be simply yanked - at the mere cost of writing Nasal parsers that turn the legacy definition files (think 2D panels, GUI dialogs, HUDs) into the corresponding Canvas representation, which is what the pui2canvas parser is doing:
http://wiki.flightgear.org/Howto:Proces ... ing_Canvas

Curt nicely summed up the multi-threading challange FlightGear is facing in one of the 2015 sourceforge interviews:

https://sourceforge.net/blog/november-2 ... lightgear/
Curt wrote:Defining a multi-threaded client-server architecture from the beginning would make a big difference today but, of course, that wasn’t a priority back then.
[...]
Currently we don’t take good advantage of multi-core CPUs.
[...]
We’re also planning to use HLA/RTI, which will allow FlightGear to integrate with distributed simulation environments (important for industrial applications) and make better use of multi-core processors.



And this is backed up by many discussions in the archives: http://wiki.flightgear.org/Multi-Thread ... FlightGear


It is hard to come up with something that could be presented as a consensus given the plethora of differing views/approaches, but one of the most commonly mentioned suggestion is decoupling/partitioning the simulator at the sgsubsystem/property tree level, by having one subsystem-specific and private property tree per subsystem, and allowing each subsystem to subscribe to state/events from other subsystems, dispatched using a global property tree - where each property tree would maintain a queue of read/write requests, and only ever handle those in a single-threaded fashion, possibly exposed using an XDR-based PUB/SUB mechanism to get/set properties in different subsystems, where each subsystem would be running in its own/separate thread - here's an article written by James summarizing that idea here:
http://wiki.flightgear.org/Property_threading

At that point, you could decouple the simulation from rendering with reasonable effort, but would still need to explicitly stop using/calling any APIs from "outside" subsystems without using serialization.

This is particularly important for subsystems that make heavy use of other subsystems - and Nasal is the main suspect here, because it's sole purpose is using a bunch of existing code/functonality without people having to know C++, so it makes sense that Nasal is the most obvious candidate to optimize when it comes to threading/IPC, because it is touching a ton of places in a very chaotic fashion, which is further complicated by FlightGear -so far- using a single global property tree for all subsystems, instead of having private, and subsystem-specific property trees that are inaccessible from other subsystems at the C++ level without going through an IPC mechanism like networking (heck, even Torsten's websockets would be better than the current mess).

This is where RPC (remote procedure calls), and "IPC" (inter-process communciation) come in - because you inevitably need a way to call functionality/code living in another process/thread at some point, as well as get something back from that. This why some folks have been actively working towards supporting HLA: http://wiki.flightgear.org/High-Level_Architecture

Anyway, before Nasal can go, or even just alternatives be considered, people need to understand why FGPythonSys would suffer from the same restrictions.
HLA is all great and dandy, but for that to actually work and be beneficial, certain groundwork has to happen first - and that involves making Nasal entirely optional, so that we can get rid of the hard-coded assumption that Nasal is always up and running.

The next step is making subsystem initialization incremental in its entirety - which also entails stopping to load all sorts of subsystem-specific Nasal scripts despite some subsystems having become optional these days (thanks to James reset/re-init work).

In other words, imagine a scenario where ATC-chatter is loaded but sound is not running/disabled, or the tanker scenario/script with AI disabled (and even moreso Bombable), or gui.nas loaded without PUI being active etc.

And like many people have stated previously, the extreme use of Nasal embedded in GUI dialogs has to stop, too - it's hugely chaotic and almost perverse, it is adding tons of problems, and making it next to impossile to come up with other UIs.

Equally, even the way people code avionics is problematic from the standpoint of wanting a simulation without Nasal in the main loop - as it doesn't scale, it also assumes that Nasal is always running in the main loop - so once/if people shuffle it out of the main loop, there would be all sorts of race conditions (=crashes) caused by such code.

The way Nasal is used in FlightGear is certainly very problematic, but before FGPythonSys (or other alternatives) can be considered, there are a handful of things that need to happen first, e.g.:

  • providing an option to disable Nasal entirely: http://wiki.flightgear.org/Howto:Disable_Nasal_entirely
  • allow Nasal to be stopped/restarted and removed/re-added at runtime using the subsystemFactory/fgcommands
  • providing an option to initialize Nasal much earlier and better formalize subsystem/script dependencies: http://wiki.flightgear.org/Initializing_Nasal_early
  • get rid of Nasal hacks commonly used in GUI dialogs and MFDs, i.e. introduce dedicated XML tags for common hacks (think widgets) and instead of having blobs of Nasal code, introduce fgcommands (possibly just mapped to Nasal code), so that other GUIs (think Phi/qt5) can support those files without them having to support Nasal
  • Integrating IPC/RPC functionality, so that Nasal interpreters can access other subsystem specific property trees without causing race conditions
  • Allow multiple instances of the scripting interpreter to be instantiated, so that these can run in different threads - e.g. with aircraft/scenery scripts running separately from GUI or weather/environmental scripts.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11329
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal must go

Postby kamikaze » Wed Mar 08, 2017 2:52 pm

+1 for Python
kamikaze
 
Posts: 7
Joined: Wed Mar 08, 2017 2:44 pm

Re: Nasal must go

Postby adamstyl » Fri Mar 17, 2017 6:14 pm

tl;dr While nasal is not flexible, rewriting is painful and costly.

Nice point, Nasal must go. That's what I thought when I first heard of it although I haven't written a single line of code -the fact that it's js sibling repels me. Now since I want to be pragmatic, let's see the pros and cons. Ok using a "general purpose" language that somebody invented back in 2003 and is only used in 1 project (yes that is FG) it's not a good idea. Who maintains it? Who makes bug fixes? Who evolves it?

Especially for the evolution part: if js was chosen back then, now (and assuming for a moment there are no licensing issues) V8 could have been harnessed to FG and scripting performance could have rocketed sky high. Then there is type script and when you have to write gazillions of lines of code, you don't want to get your errors in runtime, do you? So you could harness type script on top of V8 who is harnessed on FG. NodeJS did it, so others could do it as well. But now with Nasal: ouch you simply can't.

On the other hand you have to think one other factor: the existing codebase. And I think that someone in the thread mentioned 40K lines of nasal code. Ouch (again). If you ask me as a professional software developer, I wouldn't dare to rewrite a single line of these. There will be pain, there will be blood, there will be cursing. But I'm not sure if there will be a result. And also you have to bear in mind that FG, as an open source project, is developed by people who love it, usually in their spare time. And these people won't be very willing to rewrite things that already work.

So in my humble opinion, Nasal must go, but it won't go anywhere. For a long time.

Disclaimer: I'm a professional C# and Java developer and aviation enthusiast who just couldn't resist taking position on the matter. I hate scripting and dynamic languages so please take all the above with a grain of salt.
adamstyl
 
Posts: 6
Joined: Fri Mar 17, 2017 5:30 pm

Re: Nasal must go

Postby Thorsten » Fri Mar 17, 2017 6:28 pm

Ok using a "general purpose" language that somebody invented back in 2003 and is only used in 1 project (yes that is FG) it's not a good idea. Who maintains it? Who makes bug fixes? Who evolves it?


The one project which actually needs it does all that. Having it lean, focused towards what's needed and not dependent on external development is a very big asset, not a problem.

V8 could have been harnessed to FG and scripting performance could have rocketed sky high.


No, it would not, because scripting performance is 'sky high' right now - it's the interface between scripting and the rest, things like property I/O, terrain altitude queries etc. which are slow. And that wouldn't change if you had Python, or Lua, or whatever language you prefer.

Did you even read any of the previous arguments?
Thorsten
 
Posts: 10953
Joined: Mon Nov 02, 2009 8:33 am

Re: Nasal must go

Postby adamstyl » Fri Mar 17, 2017 6:41 pm

Oh someone is getting fretful about Nasal... Relax mate. As I said. My 2 cents...

PS I don't believe that nasal performs faster than V8 which compiles to machine code. But suit yourself.
adamstyl
 
Posts: 6
Joined: Fri Mar 17, 2017 5:30 pm

Re: Nasal must go

Postby Thorsten » Fri Mar 17, 2017 6:52 pm

PS I don't believe that nasal performs faster than V8 which compiles to machine code.


As I said above, I don't think that's particularly relevant, as it's the bottlenecks which determine execution speed of the flightsim - so I couldn't care less whether one system which has a marginal performance impact can be made 10 times faster, I care whether the bottleneck can be made 2% faster, because that you're going to see.

As I said. My 2 cents...


Not reading through a discussion and repeating arguments that have been made sure is one way to introduce oneself to a community.

Welcome anyway.
Thorsten
 
Posts: 10953
Joined: Mon Nov 02, 2009 8:33 am

Re: Nasal must go

Postby adamstyl » Fri Mar 17, 2017 7:08 pm

Thorsten, irony and aggressiveness is also another way to say hello.

Thanks for welcoming me anyway, though I'm sure it will be some time before I speak my mind again.
adamstyl
 
Posts: 6
Joined: Fri Mar 17, 2017 5:30 pm

PreviousNext

Return to Nasal

Who is online

Users browsing this forum: No registered users and 1 guest