Board index FlightGear Development New features

FGPython an propose for Python as an nasal alternative

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Dec 28, 2015 1:04 am

This is a follow up of a discussion that I have with one of the devs at fg weekend 2015.
My propose is that we include python 3 as an alternative for nasal.

This mean not that nasal is bad, but I can writing code in python faster than nasal. (this is only my opinion and experience)
Another plus for python re can create a standalone interface that connect to fgfs or play a flight from an recode session for cockpit building or testing new code.

What is your considers about Python as an nasal alternative?
Sorry this is short post, more in the future.
www2
 
Posts: 249
Joined: Thu Apr 16, 2009 1:58 pm

Re: FGPython an propose for Python as an nasal alternative

Postby wkitty42 » Mon Dec 28, 2015 1:20 am

while you did talk about it with a dev at fgweekend, it might be a good idea to also post this to the dev list where the majority of the devs hang out... then maybe consolidate the two topics in both places so that both groups know what's going on...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5502
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Dec 28, 2015 1:07 pm

This has been discussed, on and off, for the past couple of years.
Some of the remaining key developers are indeed very much fond of Python.

Python isn't going to magically "solve" certain scripting related challenges we are seeing in FG (think Garbage Collection vs. Python's GIL).
However, Python obviously is great for a number of reasons, mainly excellent libs, tools, documentation, support and maintenance status.

That said, Python support is probably going to be added sooner or later - not so much in the form of a dedicated SGSubsystem, like FGNasalSys, but instead in the form of a HLA federate - HLA is going to make such things increasingly feasible, without introducing tons of bloat/dependencies.

Adding Python in the form of a SGSubsystem would be rather simple, but cause even more issues down the road.
However, even in a HLA-enabled build, adding support for more and more scripting solutions may cause interesting challenges, think "addons" requiring modules in Nasal, Python, Lua, Ruby, Perl etc - all of that will become a possibility, and it could be facilitated by the fgaddon/package manager (catalog) developments, because there will be much less oversight than there is currently when it comes to Nasal.

The way people are using Nasal is already hugely problematic and causing tons of challenges for the project, especially since the departure of mfranz, with nobody remaining to review contributions and ensure that modules are sufficiently generic in nature, and maintainable in the long term - right now, the situation with Nasal code is an increasing mess, and it is likely that this will not improve much by supporting even more scripting languages.

Technically, it is trivial to support something like Python - from a feasibility standpoint, there are other issues to be considered though.
I suggest you look at this wiki article, and links listed there: http://wiki.flightgear.org/Nasal_success_stories

You will find, that we've had a number of related discussions over the years, i.e. about providing additional scripting support and/or replacing Nasal in its entirety - e.g. see: http://sourceforge.net/p/flightgear/mai ... sg21686158

I would encourage you to look at postings from Melchior Franz, Andy Ross or other core developers with a track record of using/extending/maintaining Nasal (i.e. the core engine or even just fgdata modules).

In general, this is touching on the "plugins" discussion, too - because, ultimately, supporting more scripting languages will mean that people can use different technology stacks to accomplish their goals: http://wiki.flightgear.org/FlightGear_Plugins

With Python you should keep in mind that Python comes with a huge community and tons of resources, including different package managers - FlightGear however is only just about to reinvent the package manager concept (from scratch): http://wiki.flightgear.org/FlightGear_Package_Manager

It would obviously be great to support Python, but most people seem to forget/ignore that this is causing tons of new challenges - imagine installing an aircraft via the package manager, that depends on a certain Python version, which in turn requires certain Python modules (think OpenCV, facetrack etc), with some of these being platform specific.

The FlightGear project is a hugely disorganized project and it is generally not very good at tackling such issues before they turn out to become real showstoppers - in other words, a few years from now, this (supporting even more technology stacks) will undoubtedly add to the workload of those shouldering much of the project currently.

If you are willing to "wait" for HLA to materialize, this will be the "more correct" path, but it still won't prevent people from implementing features (think aircraft) in a way that dependency bloat will sooner or later become a real issue.

When people say that they "prefer" language "X", what they're really saying is that they don't mind sytactic differences, but that they want to have more/better 3rd party modules.

If you were to provide a Python-based replacement next to FGNasalSys, the underlying FG related APIs would almost certainly (have to!) look the same, i.e. setprop/getprop etc.

And the next issue you would be facing is that people only care about certain features (think SGSubsystems), so that some stuff may end up not being supported by another scripting engine - imagine Python not having access to Canvas for instance (just intended as an example).

You do have a point regarding the "standalone" interpreter though - but even that can be accomplished using Nasal, in fact, there is a standalone Nasal interpreter that links to SimGear (headless) - something like that could easily be integrated with Stuart's HLA work to run /certain/ Nasal scripts as HLA federates (think AI tanker).

Supporting Python (or even Lua/JavaScript) could be great for the project (in theory) - but it is a step that needs to be carefully considered and discussed, and it would make sense to look at what people have previously stated in similar discussions, e.g. referring to: http://sourceforge.net/p/flightgear/mai ... sg21686158

Almost certainly, you don't just want to write your code in Python, but you also want to have access to certain/all FG APIs, and sooner or later people will also want to use certain Python packages, which may include binary packages (DSOs/DLLs).

But you should be also prepared to deal with these challenges, i.e. there is now an increasing trend towards online deployment of aircraft, I think we don't want to cripple such efforts by introducing dependencies to yet another scripting language and binary modules, possibly even platform/OS specific ones.

Such a step may affect/cripple FlightGear's multi-platform nature in fairly interesting ways - imagine addons like Bombable/Advanced Weather, Walker, Wildfire, Dual Control or FGCamera were implemented in other scripting languages, like Python, and what that could mean for the project from a workload/maintenance standpoint.

To be fair, there has been some discussion among a few core developers to entirely convert Nasal code in $FG_ROOT to Python code and replace the Nasal VM in its entirety - but many of these discussions don't make it to the devel list for understandable reasons :)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: FGPython an propose for Python as an nasal alternative

Postby curt » Mon Dec 28, 2015 1:39 pm

I think the first hurdle towards python integration would be an interface to the property system. I have another project (a UAV autopilot system) that is built using the FlightGear properties, and I have a longer term interest in python integration there too. For the FlightGear project specifically, it would be important to proceed in a way that doesn't create headaches for the build system ... i.e. we need to be able to build on Windows, Mac, and Linux. Typically we are adverse to adding new build dependencies. We are less adverse to adding 'optional' build dependencies.
curt
Administrator
 
Posts: 1173
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Dec 28, 2015 4:33 pm

Wrapping setprop() and getprop() functionality via Python bindings would be trivial, and Python is exceptionally well supported by cmake, too - on all platforms (via findPackage(PythonLibs)):
https://cmake.org/cmake/help/v3.0/modul ... nLibs.html

In fact, it is probably safe to say that Python, and its underlying cmake tooling, are supported on tons of platforms that not even FlightGear supports, the number of FlightGear targets would be a tiny subset of those.

Thus, even Jenkins (the build server) should be easy to update accordingly.

The C/C++ code to provide setprop() or getprop() equivalents would be under 50 lines of code, and Python has much better support for automatically created bindings than Nasal used to have (prior to cppbind), so using boost/bind, it would not be too difficult to even re-implement props.nas entirely, without any scripting space overhead at all (minus the GIL obviously).

Those are straightforward things in comparison to the arguments brought up by Melchior, Andy and others in the original "Nasal alternatives" thread on the devel list.

And the property tree is trivial to hook up to Python thanks to it not being thread-safe at all - other FG APIs would be more interesting, while still looking very much like their Nasal equivalents.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: FGPython an propose for Python as an nasal alternative

Postby Thorsten » Mon Dec 28, 2015 5:47 pm

More scripting languages - more potential problems, more maintenance load, more fiddling with dependencies. If the only advantage is really that people feel more comfy with Python than with Nasal, then I frankly don't see much of a gain. So is there something that really can be done with Python which we can't do in Nasal?

My quick two cents.

We could also link FORTRAN-written libs to FG in case someone is more comfy coding FORTRAN than C++ - it's quite doable :-)
Thorsten
 
Posts: 10693
Joined: Mon Nov 02, 2009 8:33 am

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Dec 28, 2015 6:07 pm

Like I said, I tend to agree - but the momentum/support provided by Python is hard to beat, it could be a great technology enabler, similar to how OSG adoption has literally boosted core development, and similar to Qt5 adoption having the potential to significantly improve/accelerate core development.

For the time being, Nasal must be considered "unmaintained", and whatever is available, is usually written and maintained by the FlightGear community, no matter if that means Nasal code or C/C++ code related to Nasal.

With Python, the situation is hugely different/better - and like I mentioned previously, most people will not be bothered by the syntax being different, but by Nasal not having the degree of 3rd party support/tooling and modules/libraries.

Which means that the real question is one of accepting more dependencies, optional or not - like Curt briefly mentioned.

There is definitely tons of stuff for which we cannot currently use Nasal, due to the sheer lack of modules for those things - even just thinking in terms of recent development (think Stuart's HLA work, using Python modules prototyped by Mathias).

The situation is very similar to the whole Qt5 UI debate, i.e. from a manpower/resource standpoint, it makes absolutely sense for the FlightGear project to stop reinventing the wheel, and use established "standards" - no matter if that means using Qt5 and/or HTML5/JavaScript (Phi) for a UI, or a more standard solution for embedded scripting, just like the adoption of OSG has been a great technology enabler.

However, in actuality, it is going to cause new challenges - especially in the light of the package manager development, and how this will be affected by Aircraft (and possibly other resources) having potential dependencies to optional features/module, like Python itself - but also an infinite number more or less platform/OS specific modules to implement certain features.

For the time being, FlightGear neither has the infrastructure in place to deal with it, nor are there any plans on the roadmap to prevent the potential chaos arising from supporting different scripting solutions.

Even just the whole HLA development is going to be a huge challenge, i.e. it opening up a whole new can of works (think licensing).

But the deployment argument is kinda moot, especially for Qt5-enabled builds, because Python can be pulled in via Qt5.

A few years ago, many of the key contributors familiar with Nasal internals (Andy, Melchior etc) were opposed to the whole idea pretty much for the reasons stated by Thorsten - these days, it seems that many of those are no longer as actively involved.

And Thorsten is right, it is likely that "bit rot" is going to cause interesting challenges somewhere down the road - we could/can see that in other areas that have become optional, and which are using non-standard languages (e.g. think the Ada FDM).

From a consistency standpoint, it would be good to address such challenges before they become actual problems, even if just by augmenting the "project roadmap" document accordingly, which is already missing so many crucial developments (think reset/re-init).

It would not be funny to see people expend their time on improving Nasal internals, only to see it replaced at some point... we are already fairly bad at attracting/keeping contributors.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11317
Joined: Tue Mar 25, 2008 8:40 am

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Dec 28, 2015 6:28 pm

Thorsten wrote in Mon Dec 28, 2015 5:47 pm:More scripting languages - more potential problems, more maintenance load, more fiddling with dependencies. If the only advantage is really that people feel more comfy with Python than with Nasal, then I frankly don't see much of a gain. So is there something that really can be done with Python which we can't do in Nasal?

That depent on the goal.
One of the things that python can do and nasal not is using NumPy for matrices what normal is done in Matlab.
And be site this there is cython that is an python compiler for compiling python code in c/c++ code for more speed using pyx extension.
This can be done on Jenkins.
Thorsten wrote in Mon Dec 28, 2015 5:47 pm:We could also link FORTRAN-written libs to FG in case someone is more comfy coding FORTRAN than C++ - it's quite doable :-)

LOL FORTRAN in fgfs that is good for custom fdm.

EDIT:
Some of the stuff that I miss in nasal and i can easy do in python is adding an WebSocket client for example to connect to blitzortung for real time lighting data.
Or parsing Global Forecast System data for more realistic weather simulator using pygrib. (https://github.com/jswhit/pygrib)
Last edited by www2 on Mon Dec 28, 2015 7:00 pm, edited 1 time in total.
www2
 
Posts: 249
Joined: Thu Apr 16, 2009 1:58 pm

Re: FGPython an propose for Python as an nasal alternative

Postby Thorsten » Mon Dec 28, 2015 6:47 pm

One of the things that python can do and nasal not is using NumPy for matrices what normal is done in Matlab.


Should I ever need it, it'd cost me 20 minutes to code matrix algebra into a Nasal module. Like I have my set of vector operations, time utility functions and rotation handlers for the Shuttle code.
Thorsten
 
Posts: 10693
Joined: Mon Nov 02, 2009 8:33 am

Re: FGPython an propose for Python as an nasal alternative

Postby curt » Mon Dec 28, 2015 7:04 pm

I'm nervous right now because I feel like I'm slightly agreeing with hooray ... but that said, it's not just that some of the original nasal proponents are less involved now. Python is much more prominent and well known today than it was 10-15 years ago. When Nasal was first imagined, perl was still a pretty big deal and python was still more of the new-comer trying to break onto the scene... at the time it seemed like just another scripting language ... and slightly weird because of it's rigid white space usage. Lua was/is also a viable option. (So let's not nitpick exact impressions/timelines ... much of this is subjective to a greater or lesser degree.) :-) At the time, a really stripped down, lean custom scripting language for FlightGear made quite a bit of sense ... versus rolling in a massive perl or python dependency or venturing into Lua/Javascript waters where most of the developers at the time where quite unfamiliar. Today python makes quite a bit of sense for a number of reason. It is quite well known, it carries with it a tremendous body of supporting library code ... things like numpy, scipy, sympy, opencv, etc. offer extremely sophisticated functionality. If I were to start writing terragear today, I would definitely start down that path in python. Obviously we can't do anything too intensive per-frame in a scripting language.

I've grown to be a big python fan over the past 5 years or so and have used it for two pretty serious projects (and a bunch of smaller things) ... 1. a fully parameterized 3d modeler and part layout nester (madesigner.flightgear.org) and 2. an open-source image stitching pipeline that does some of the same things as pix4d or agisoft: https://github.com/UASLab/ImageAnalysis

Interesting (to myself at least) is that the aerial image stitching tool chain touches on some similar concepts to terragear including DEM simplification and surface approximate, point cloud triangulation, and probably several other things.

I personally haven't looked into binding C/C++ code with python code myself, so it may be as easy as Hooray suggests ... 50 lines of code isn't too much, although I'm sure it's important to get the exactly correct 50 lines of code, which can sometimes be a bit more work. :-)

So from my own perspective at least, I think python integration would be an interesting path to run down. And for sure, early adopters would need to understand they are following a highly experimental path and any python dependent models or scripts would have zero backwards compatibility with currently existing or previous versions of FlightGear.

Here I will ask a question I don't know the answer to: what's the situation with python garbage collection? I think nasal garbage collection performance issues is the biggest strike against nasal right now. But is python any better?

To address Thorsten's question of what can python do that nasal can't? Probably nothing that is super compelling. I imagine that even if we did include support and hooks for python scripts, nasal would still be the standard/recommended/best-documented FlightGear scripting language for many years to come. This would fall more into the category of supporting additional sandbox's because we can and there is enough community desire, not because there is a specific need. Repeating myself: access to things like numpy's matrix and vector math libraries and fast data structures might be attractive for people who are scripting more sophisticated behavior ... again, not that we can't implement the same math and logic in nasal, but python offers some great functionality right out of the box, well known, well documented, well debugged, well optimized.
curt
Administrator
 
Posts: 1173
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Dec 28, 2015 7:05 pm

Thorsten wrote in Mon Dec 28, 2015 6:47 pm:Should I ever need it, it'd cost me 20 minutes to code matrix algebra into a Nasal module. Like I have my set of vector operations, time utility functions and rotation handlers for the Shuttle code.

I bereave this is slower than numpy that use an c/c++ library as an back end.
www2
 
Posts: 249
Joined: Thu Apr 16, 2009 1:58 pm

Re: FGPython an propose for Python as an nasal alternative

Postby curt » Mon Dec 28, 2015 7:10 pm

www2 wrote in Mon Dec 28, 2015 7:05 pm:
Thorsten wrote in Mon Dec 28, 2015 6:47 pm:Should I ever need it, it'd cost me 20 minutes to code matrix algebra into a Nasal module. Like I have my set of vector operations, time utility functions and rotation handlers for the Shuttle code.

I believe this is slower than numpy that use an c/c++ library as an back end.


Yes, that's exactly the difference. Thorsten could easily spend 20 minutes coding up a functional equivalent for sure (I think I've done something similar for some basic quaternion math), but the numpy routines are designed on top of fast C code and fast C data structures. You still have to use the numpy functionality intelligently to get a performance benefit, but if you understand the library a bit so you stay on the optimized path, the results can be pretty equivalent to raw C performance.
curt
Administrator
 
Posts: 1173
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: FGPython an propose for Python as an nasal alternative

Postby curt » Mon Dec 28, 2015 7:13 pm

Hmmm, www2 brings up a good point that's worth considering before going too far down the python path. If we can easily pull in any and all python library functionality, like web servers (for example), then we have some huge security concerns to factor in. A rogue aircraft could potentially turn your computer into a spam bot, or worse. One of the benefits of nasal is that it is carefully sandboxed for what it can or can't do ... would we have the same level of control over security with full blown python integration? I don't know?
curt
Administrator
 
Posts: 1173
Joined: Thu Jan 01, 1970 12:00 am
Location: Minneapolis, MN

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Dec 28, 2015 7:20 pm

curt
Here I will ask a question I don't know the answer to: what's the situation with python garbage collection? I think nasal garbage collection performance issues is the biggest strike against nasal right now. But is python any better?

That was one of the first question that was ask to me when I put this idea on IRC and make some benchmarks about this.
Code: Select all
node   time in sec
5      0,000411
50      0,00041
500      0,000414
5000   0,00044
50000   0,000741
500000   0,003672
5000000   0,035334

note: node is the size of the an array
time in sec: is the time that is use for garbage collection

source:
Code: Select all
import gc
import time
def gctestrun(count):
   l = []
   for i in range(count):
       l.append(l)
   l = None
   t1 = time.time()
   gc.collect()
   t2 = time.time()
   return (t2 - t1)
   

def gctest(count,numOfTest):
  print('number of objects: %i' % count)
  t = 0
  for i in range(numOfTest):
    t += gctestrun(count)
  mt = t/numOfTest
  print('gc time: %f' % mt)
  print('framerate cg %f' % (1.0/mt))
  gc.collect()

gctest(5,1000)
gctest(50,1000)
gctest(500,1000)
gctest(5000,1000)
gctest(50000,1000)
gctest(500000,1000)
gctest(5000000,1000)
www2
 
Posts: 249
Joined: Thu Apr 16, 2009 1:58 pm

Re: FGPython an propose for Python as an nasal alternative

Postby sanhozay » Mon Dec 28, 2015 7:23 pm

I like Python but I don't see a huge benefit in replacing one scripting language with another, or worse, having a mixture of two.

I think we'd be better off thinking about domain specific languages to reduce the verbosity of XML: property rules, JSBSim systems, autopilot controllers, model animations, checklists, sound effects ...
sanhozay
 
Posts: 1207
Joined: Thu Dec 26, 2013 11:57 am
Location: EGNM
Callsign: G-SHOZ
Version: Git
OS: Ubuntu 16.04

Next

Return to New features

Who is online

Users browsing this forum: No registered users and 1 guest