Board index FlightGear Development Nasal

Nasal week :-)

Nasal is the scripting language of FlightGear.

Nasal week :-)

Postby Hooray » Sat Oct 15, 2011 2:58 pm

Together with flug, I spent some time reviewing, extending/reorganizing and hopefully improving the existing Nasal documentation in the wiki.

The goals are simple:
  • make Nasal more accessible and increase its user base
  • improve the documentation
  • document best practices
  • document new techniques

Thus, I would like to invite all people interested in learning more about Nasal to help me review the Nasal documentation, so that we can hopefully further improve it to make it more accessible to new FlightGear users but also people who are generally new to programming.

I hope to be able to spend the next week answering your questions by incorporating the requested info in the Nasal article, if that shouldn't be possible, I will catch up with your feedback later on!

For people who don't know anything about programming and/or Nasal, that means: your input is valuable and appreciated!!

Please, ask your questions here (or over at the wiki), so that I can clarify things as needed. For example, if there is anything that you don't understand (terminology, phrases etc), PLEASE leave a comment here! That's the only way to ensure that FlightGear scripting becomes more accessible to end users.

None of my recent additions are intended to become full fledged tutorials, most of this is still introductory material. So, this is not the right place to ask for full tutorials, if that's what you are interested in, please add your ideas to the list here: wiki.flightgear.org/Nasal_introduction

Finally, I would obviously appreciate any help from people who have a "black belt track record" of writing Nasal code, such as mfranz, AndersG, flug, Thorsten, xiii, syd, skyop and others. The truth is, by sharing and documenting our knowledge, we can ensure that other people can learn from it and create new things in FlightGear. Even just 10 minutes spent at documenting things, has the potential to be of good use to many FlightGear users, now and in the future!

For instance, some of the additions I've made, were purely copied/pasted from the mailing list or the forum here.
Yes, it is easy to point people to the forum search or the mailing list archives or tell them "read the source" (I keep doing that myself unfortunately), but the truth is we can do better by adding important info right away to the wiki, and I noticed that many of my Nasal related postings/comments could be directly copied to the wiki, too. Also, a number of messages that I exchanged with Thorsten regarding his local weather code, were also added to the wiki because they may be useful to others eventually.


PS: Yes, I am absolutely aware that the original Nasal article has become rather sizable now, I plan to restructure things eventually and move them to dedicated pages, but for the time being I don't consider this a major problem or priority at all, I think it's a great "challenge" to have actually, I'd rather have too much information, than too little - restructuring things isn't as complex as writing up new contents, and can even be done by people who don't have to be proficient in Nasal necessarily. So please bear with me and be a little patient, thanks!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12164
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal week :-)

Postby Thorsten » Mon Oct 17, 2011 11:10 am

Well, it's a joy to be used as the example of bad coding so often... :D

So, let me make a provocative statement here: I don't really care how many lines code has - I care about two things: 1) time it takes me to write and debug the code 2) time it takes to execute the code. For many of your elegant suggestions, I'd like to see evidence that they're in fact not slowing down the system.

I've noticed one other thing - you should explain how props.xxx() works - you're using it in several examples, but you're not really telling what it is and does.
Thorsten
 
Posts: 11995
Joined: Mon Nov 02, 2009 8:33 am

Re: Nasal week :-)

Postby Hooray » Mon Oct 17, 2011 1:12 pm

Thorsten wrote:Well, it's a joy to be used as the example of bad coding so often... :D


shit, I really hope you are kidding!

It's just that it occurred to me that we exchanged so many PMs, that others may possibly benefit from - that's why I copied them directly to the Nasal page. Honestly, I didn't feel inclined to rework the code or provide different examples... My own laziness, I guess - I simply copied/pasted everything directly, sorry if you feel sort of offended now (even if just slightly) ...

I never meant to imply that there's "bad coding" involved at all. Obviously, you code works and gets things done. As you have surely noticed, most of these are just about brevity and expressiveness. Actually, I am pretty sure that you know already ALL suggestions, and I _think_ that we talked a while ago about adding those suggestions to the wiki, too. I don't think I ever said anything about "bad coding" at all!

So, let me make a provocative statement here: I don't really care how many lines code has - I care about two things: 1) time it takes me to write and debug the code 2) time it takes to execute the code. For many of your elegant suggestions, I'd like to see evidence that they're in fact not slowing down the system.


Really, no need to argue at all ... the point of those additions to the wiki was not at all to change your code, just to improve the Nasal documentation by providing some more examples. I am really sorry if that could be misunderstood. Obviously, if you want me to provide performance figures now, I'd rather spend the time on improving the documentation.

As you may remember, the point was originally that 10,000+ lines of code are not necessarily easy to understand (a point repeatedly brought up by others) - thus the idea to use separate modules (files), more compact functions and loops, as well as overall more expressive code.

On the other hand, I understand and share your philosophy completely (it's obviously very efficient after all for a single developer), my perspective is just that it's hard for anybody -besides yourself obviously- to easily work with that code base, no matter if it's about providing bug fixes or creating new features.

And part of this is not just caused by the complexity of the architecture but also simply by the wealth of code. Nasal being a functional programming language has the potential to be really succinct and clear, when a non-procedural programming style is adopted. That, I was trying to illustrate.

None of the comments in the wiki were directly aimed at LW. Reworking the examples/code would just take more time.

Actually, I am convinced that you understand that, given that you have adopted a number of similar suggestions in the past already, without asking for performance comparisons ;-)

Again, sorry ... I think, I'll have to clarify this in the wiki, too.

I've noticed one other thing - you should explain how props.xxx() works - you're using it in several examples, but you're not really telling what it is and does.

okay, will do - props.nas.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12164
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal week :-)

Postby curt » Mon Oct 17, 2011 2:51 pm

From an engineering perspective, every tool has it's place. Every choice has strengths and weaknesses. No choice is perfect. The goal of a good engineer is to balance all the factors and influences and pick an approach and make choices that best suit your project. And then with your decisions established, maximize the good aspects of your choices, and find work arounds to minimize that bad aspects of your choices.

People can "blame" nasal, but it is really just one part of a trio of tools that make FlightGear a really interesting development platform:

XML + the "property system" + nasal.

These 3 things go hand in hand, and none of them would be nearly as useful by themselves. These tools give tremendous power and internal access to end-users and "non" developers. (And by non-developers I mean those who don't dream every night about compilers and cmake and linux and whitespace and semicolons and scoping rules, etc.) :-)

The property system is our internal data structures mapped to english names and made accessible in a variety of ways.
XML is our way to represent static configurations (model animations, instrument panels, sound configuration, autopilot, etc. etc. etc.) all of which get mapped directly into the property system for easy data sharing.
Nasal gives run-time programming ability to anyone -- and scripts can be tied to particular aircraft or models.

Now is nasal+properties+xml the best way to attack every problem on the planet? Of course not, but I think we can show example after example of how these tools bring power and freedom to more people to do what they want or what they need in the software world. Can nasal+properties+xml solve every problem on the planet (even though it might not be the most optimal approach) -- probably the answer is yes. :-)

Is nasal a great fast-prototyping tool for working out the logic and details of a task? Absolutely!
Is nasal a great way for new comers and non-developers to add code to the project to do new and never-before-anticipated things? Absolutely!
Is Andy Ross a coding genius and one of my all time heros? Absolutely!

So I say let's all nasal away, but let's also be good engineers and keep an eye on performance issues. Once we have our logic figured out and our task prototyped, it is worth looking at pushing the more performance critical items into core code.
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: Nasal week :-)

Postby Hooray » Mon Oct 17, 2011 5:45 pm

Curt, I think you brought up a number of excellent points, and your posting could (and actually should!) be copied to the wiki, as is. Because it is an excellent introduction to the flexibility of FlightGear, and the reasons for it. And I think, I will exactly do that if you don't mind it.

Obviously, FlightGear's power and flexibility doesn't just come from Nasal, the use of XML and the property tree are definitely two other significant decisions and pillars that make all this possible in the first place.

On the other hand, personally, I am not sure if I agree with your final sentiments, this is because I have used SimGear/Nasal in a non-FG related context (embedded) at work, where it doesn't show many of the issues that are more prominent in FlightGear:

curt wrote:Once we have our logic figured out and our task prototyped, it is worth looking at pushing the more performance critical items into core code.


I think, we do agree that there is a steadily increasing amount of Nasal source code being used in FlightGear, no matter if it's tiny bits of Nasal sprinkled all over the place, or literally complex systems such as the local weather package or the bombable addon, both of which are creating significant new features on top of FlightGear.

For instance, just yesterday I ran find/xargs/wc against the base package to see how many lines of Nasal code we have at the moment in order to update the wiki metrics, it's 350 kloc now (that's just in *.nas files, not counting any embedded scripts-not sure if previous metrics counted those, too?).

What I am trying to say is: the amount of Nasal code in FlightGear is huge, and it is pretty unlikely to stagnate or even decrease. I think, both of us agree that it is a good thing for FlightGear and the degree of flexibility it provides.

As you said already, Nasal is already extremely clever and efficient, but on the other hand, let's face it: more complex systems being implemented in Nasal space, bring new issues to our attention, situations where Nasal could obviously do better than it is doing right now.

The truth is, most of these issues have only recently become really obvious and sort of "significant", due to heavier use of Nasal to create more complex systems.

In other words, projects like the local weather system or the bombable addon are just showing SYMPTOMS of a deeper-lying issue here.

Thus, it is in my opinion not doing justice to Nasal to suggest that a different scripting language should be employed, or that a different programming language in general should be used to implement certain features, this includes C++ in my opinion.

I brought this up a number of times before: As you surely know, Mozilla/Firefox has become the de facto standard browser these days, yet much of the high level features are purely implemented in JavaScript.

I consider Nasal's role in FlightGear similar to JavaScript's role Firefox, where it is also used to make basically the entire host application scriptable and allow people to implement completely new features and even applications on top of Firefox (think Thunderbird).

That is the reason why I think it would be better to acknowledge the fact that the Nasal subsystem has recently shown some issues, these are actually just a handful of issues that show up under certain pretty well-defined circumstances in FlightGear. This in and of itself is pretty remarkable actually, and a testimony to the Nasal implementation.

Assuming that the Nasal adoption rate, and thus its use in FlightGear, is going to increase at least linearly in the years to come, it would in my opinion be better to look into fixing Nasal as a subsystem in FlightGear, rather than suggesting the use of different programming languages to implement certain features "because of Nasal problems".

This is part of the reason, why I spent 2 hours yesterday browsing the mailing list archives and looking for previously discussed Nasal issues and created a corresponding wiki page: http://wiki.flightgear.org/Improving_Nasal

The summary is: while there are a handful of issues in the lexer/tokenizer code (i.e. bugs in the parser), the Nasal VM itself is actually pretty mature and robust, there are actually only two "real" issues that I could find, that affect FlightGear code at run time:

The global interpreter context (a legacy leftover leaking memory and causing GC issues, according to Andy Ross himself), and the GC related issues discussed a couple of months ago, identified by ThorstenB.

Both issues are actually related, and both may have a more or less noticeable effect on frame rate (latency), depending on your CPU power.

According to earlier discussions, AndersG has already provided a patch for a threaded GC implementation, and there were talks about incremental garbage collection algorithms, too.

Interestingly, according to the Nasal design document, Andy Ross himself intended to implement a "return early" mechanism for the GC, for latency-critical applications (i.e. like FlightGear!):

http://www.plausible.org/nasal/doc.html
The current implementation is a simple mark/sweep collector, which should be acceptable for most applications. Future enhancements will include a "return early" capability for latency-critical applications. The collector can be instructed to return after a certain maximum delay, and be restarted later.
An outside application may have realtime constraints, and Nasal can be instructed to run for only a certain number of "cycles" before returning. Later calls will automatically pick up the interpreter state where it left off.


In other words, this looks like an idea that might be worthwhile to look into, it would allow the GC to be run regularly, while ensuring that the frame rate doesn't suffer.

The new "worst frame latency" property could in fact be used to link Nasal/GC invocations dynamically to the frame rate, to ensure that a certain latency is never exceeded.

The most obvious issue/challenge here would actually be picking a starting point for GC, which might need to be semi-random to ensure that all references get searched eventually.

IMHO, It is preferable to improve the overall FlightGear/Nasal system so that all Nasal code can automatically benefit from it.
ThorstenB has implemented support for on demand loading of Nasal sub modules already, which is another great feature.

Finally, it is obviously easier to re-implement certain code in C++ space if you need higher performance.
But moving code into C++ space just because "C++ is faster" isn't necessarily the right thing to do: Most of the Nasal limitations are not so much due to Nasal, but rather due to FlightGear's architecture and the way Nasal needed to be embedded in it. I don't see ANY of this in my work project at all!

Besides, there are many features supported by Nasal that are not currently used by FlightGear, features that FlightGear and its users may in fact benefit from some day.

This includes Nasal's thread safe design, but also its support for functional programming. Both of which having the potential to have a noticeable effect on FlightGear once adopted, including its performance.

For instance, at the moment it is obviously much more straightforward to make use of worker threads in Nasal space, than it is in C++ space. However, the FlightGear interface itself cannot currently be considered to be thread safe unfortunately.

Once this is fixed, more complex systems implemented in Nasal could obviously run some computations/tasks asynchronously without having an effect on the frame rate.

So the problem really isn't that FlightGear has outgrown Nasal, the problem is that SOME Nasal systems have outgrown FlightGear's architecture and the way Nasal has been embedded in it.

I absolutely understand that it is hard to think beyond the knee-jerk reaction that scripting is just for slow stuff, and that multi-threading should always be frowned upon.
But it is absolutely possible, just think about all the multi-threaded JavaScript code that is running in your firefox browser, right now!

For instance, AndersG has repeatedly suggested that subsystem-level parallelism should be exploited before main loop parallelism is even considered:

viewtopic.php?f=6&t=9338#p93219
AndersG wrote:I also think that it would be preferable to first look for potential parallelism inside subsystem (while maintaining a sequential interface with the main


I share that view. Fact is that this is significantly facilitated by Nasal, worker threads and functional programming in scripting space.
I mean, let's face it: basically, it has taken almost 10 years and probably about 500 kloc for Nasal to become a noticable bottleneck in FlightGear, under CERTAIN circumstances.

Suggesting now that Nasal is perfect for prototyping stuff that should be moved into C++ space eventually isn't really fair, IMHO.
Moving algorithms into C++ space to achieve higher performance, is just curing symptoms and means moving problems to the C++ code base instead.

The truth is, that the C++ code in FlightGear also has issues, too. Much of it hasn't been actively maintained for years, you don't need to look very hard to find legacy code that was scheduled for removal years ago, according to code comments.

Similarly, many people have been asking to introduce more parallelism into FlightGear, too.

For *years*, people who were asking how to speed up FG, were advised to switch off main loop systems, such as the AI traffic system.
So, the question remains if moving more code into C++ space and thus the main loop, doesn't complicate things?

As a C++ developer, I would be happy about every bit of code that is moved out of the core code base, because that means less maintenance work for me, no?

Maybe, it just time to revisit and reconsider the way Nasal is embedded in FlightGear, to fix the most obvious issues?

Some of these problems would probably not exist if Nasal were actively maintained, while others would be more prominent if we had an active maintainer for the Nasal library in $FG_ROOT/Nasal.

</SOAPBOX>
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12164
Joined: Tue Mar 25, 2008 8:40 am

Re: Nasal week :-)

Postby Thorsten » Tue Oct 18, 2011 6:57 am

shit, I really hope you are kidding!


Yes, I tend to be fairly relaxed about these things most of the time nowadays, except when I come from a heavy debugging session.

But there's some serious comment here as well: You aim to

document best practices


and from having read through a number of your suggestions, I have some idea about what you think they are - you like very elegant code. Here's my list of preferences:

Code needs to be...

1) ... there
2) ... working
3) ... free of major bugs
4) ... efficient and low impact on a single frame
5) ... accessible
6) ... elegant

Just to give examples of what I mean: If there's a single person who doesn't write Nasal code because he doesn't understand elegant OOP structures, that's a bad day for us.

Or, props.globals.getNode.doSomething(); has very elegant functionality allowing OOP style coding. Since I have figured out that setprop(); works 3 - 10 times faster (dependent on circumstances) I remove it from my code wherever I find it.

Or, you once advised me that using

Code: Select all
var number_to_compare = 30;
(some other code)
if (i == number_to_compare)  {do stuff();}


would be preferable over

Code: Select all
if (i==30) {do stuff();}


But I find the second variant much more accessible - you immediately see what i is compared to. The first variant can make for spectacularly difficult debugging if number_to_compare isn't what you think it is, because its name prevents you from thinking that it could be anything else. Similarly with your hash lists vs. my explicit comparisons. I may do in 100 lines what takes you 20 - but my 100 lines are easy to read, you see exactly what property gets what value, so in case of debugging it's much faster to debug the 100 lines than the 20.

Good practice involves always some judgement - some code needs to be efficient and dispense with accessibility and elegance if necessary. Other parts of the code should be deliberately accessible as interfaces for addons, and some loss of efficiency is acceptable. Some code must be bug-free at all costs, so it should be written as dumb and straightforward as possible to minimize the chance of errors.

Having said that - there should perhaps also be a section on writing efficient code. I've come across some spectacularly inefficient schemes - the favourite mistake is calculating quantities which change slowly and could be updated per minute in a per-frame loop.
Thorsten
 
Posts: 11995
Joined: Mon Nov 02, 2009 8:33 am


Return to Nasal

Who is online

Users browsing this forum: No registered users and 1 guest