Board index FlightGear Development New features

FGPython an propose for Python as an nasal alternative

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Mon Feb 01, 2016 11:02 am

Just a quick FGPythonSys update. I'm currently working on a proper test suite for FlightGear and I will convert all the tests in the py-ogel test.py script into real unit tests. This is not easy - think along the lines of eliminating the 'globals' class and making FGPythonSys complete independent of the fgfs binary, which I have done!

@Hooray:

Once I have the test suite up and running and all the current features of FGPythonSys have 100% test coverage, I'll look into all your great suggestions.

@www2:

For sandboxing, note that you can replace these modules in the sys.modules data structure. This means that typing 'import os' will import the replacement located in sys.modules. However there are ways around this using Python's extremely powerful introspection mechanisms. I did work out how to get to the original 'os' module after replacing sys.modules['os'], and this was in a deep place where the original module could not be replaced. Unfortunately I didn't write this down, so I would need to look again (there's plenty of info on the web on how to do this). A second problem is that Python will load Python/C modules. This means that you can just write a replacement 'os' module and bundle it with the aircraft.

Therefore sandboxing is not a good security solution. Note that these issues will never be listed as a security issue by CERT - this is just a mechanism for delivering trojan horses.

A better solution is to build up the concept of trust. You would do this by pre-parsing of the Python scripts, checking for all 'import' or equivalent statements. You would then follow the imports to see if it is from $FG_ROOT, from the aircraft, the Python standard library or elsewhere, following all paths in sys.paths (not just the standard load order). Then if there is any import of the 'os' or 'sys' modules, using introspection methods as well, or any non Python standard path loading of a C module (i.e. not a text file), then the pre-screening can label the aircraft (or other content) as unsafe and refuse to load it. The pre-parsing would follow all non-standard imports and check those too. We could then create a command line option such as --trusted-path to allow aircraft to be labelled as safe by the user, for example for all FGAddon aircraft, so that these load faster.

All of this pre-parsing could be locked into stone via the test suite I am developing. Each import mechanism and its blacklisting by the pre-parsing can be tested in separate unit tests.

Regards,

Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Feb 01, 2016 12:32 pm

Once I have the test suite up and running and all the current features of FGPythonSys have 100% test coverage, I'll look into all your great suggestions.


Actually, my postings were primarily intended to be "constructive feedback", based on our experiences with Nasal/FGNasalSys and the existing SGSubsystem/Mgr infrastructure/main loop- so I would strongly suggest not to work on anything related to my feedback without first raising this with people/core developers willing to help review/commit any related patches.

Also, keep in mind, that people's priorities, but also opinions, may differ and change over time.

However, FGPythonSys is in a pretty good position, because there isn't any legacy code base that it needs to support, unlike FGNasalSys - which has to continue working with tons of existing code/features in $FG_ROOT, so that we need to tread very carefully when it comes to refactoring things.

Some of your ideas/interests are admittedly overlapping with my own interest to move certain Nasal/scripting functionality to a different thread/core - and like I mentioned, "tasking" could be really useful to establish as a concept in scripting space.

So I may provide the occasional patch trying to ensure that your work aligns well with my own agenda :D

Thus, before Curt or others start again yelling at me for speaking in a "self-appointed" fashion on FlightGear core development matters, I suggest to prioritize your work based on what actively involved core developers have to say, i.e. those willing to get your work reviewed and committed, and those familiar with the FGNasalSys/SGSubsystem side of things, and the limitations resulting from that.

On the other hand, I also suggest to keep asking yourself if you are getting the impression that your efforts are rewarded, because the backlog of patches not yet committed is still growing... while I do realize that you are considering this an "experiment" for the time being, you must have already spent several hours prototyping all this, and probably doing at least as much research/networking on the forum and the devel list.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Fri Feb 05, 2016 12:44 am

To day i have port geodtocart() and carttogeod() from nasal api to python api.
The only c++ function on my todo list is current geodinfo().
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Fri Feb 05, 2016 8:10 pm

Not sure what you have done there, but it would make more sense to expose the underlying C++ code in SimGear instead of re-implementing such functions in Python space
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Fri Feb 05, 2016 8:37 pm

Hi,

I'm still working on a test suite and hunting down every last memory leak (you'll find some initial code in my cppunit/r* branches). I guess you would like to replicate the following Nasal function (I really don't know why this is Nasal specific and not elsewhere):

Code: Select all
// For given geodetic point return array with elevation, and a material data
// hash, or nil if there's no information available (tile not loaded). If
// information about the material isn't available, then nil is returned instead
// of the hash.
static naRef f_geodinfo(naContext c, naRef me, int argc, naRef* args)
{
#define HASHSET(s,l,n) naHash_set(matdata, naStr_fromdata(naNewString(c),s,l),n)
  if(argc < 2 || argc > 3)
    naRuntimeError(c, "geodinfo() expects 2 or 3 arguments: lat, lon [, maxalt]");
  double lat = naNumValue(args[0]).num;
  double lon = naNumValue(args[1]).num;
  double elev = argc == 3 ? naNumValue(args[2]).num : 10000;
  const simgear::BVHMaterial *material;
  SGGeod geod = SGGeod::fromDegM(lon, lat, elev);
  if(!globals->get_scenery()->get_elevation_m(geod, elev, &material))
    return naNil();
  const SGMaterial *mat = dynamic_cast<const SGMaterial *>(material);
  naRef vec = naNewVector(c);
  naVec_append(vec, naNum(elev));
  naRef matdata = naNil();
  if(mat) {
    matdata = naNewHash(c);
    naRef names = naNewVector(c);
    BOOST_FOREACH(const std::string& n, mat->get_names())
      naVec_append(names, stringToNasal(c, n));
     
    HASHSET("names", 5, names);
    HASHSET("solid", 5, naNum(mat->get_solid()));
    HASHSET("friction_factor", 15, naNum(mat->get_friction_factor()));
    HASHSET("rolling_friction", 16, naNum(mat->get_rolling_friction()));
    HASHSET("load_resistance", 15, naNum(mat->get_load_resistance()));
    HASHSET("bumpiness", 9, naNum(mat->get_bumpiness()));
    HASHSET("light_coverage", 14, naNum(mat->get_light_coverage()));
  }
  naVec_append(vec, matdata);
  return vec;
#undef HASHSET
}


I would suggest not touching globals here, as Nasal does. This is rather convoluted! Instead directly call FGScenery::get_elevation_m(). For the test suite branches, I have made FGPythonSys 100% independent of 'globals', and I would like to keep it that way.

For designing a Python version of geodinfo(), I would first write some Python code to demonstrate its expected input and output. This behaviour can then be locked into a unit test.

Regards,
Edward


Edit: Documentation could also be written on the wiki, similar to the Nasal geodinfo() function.
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Fri Feb 05, 2016 8:56 pm

If the Python "thing" should really take off it might make sense to unify useful APIs and re-implement them using a wrapper like SWIG (for which a Nasal patch exists) - i.e. to ensure that APIs are using common code/signature.

By the way, Zakalawe and others once mentioned that such Nasal APIs would ideally be implemented as objects using lazy computation - unfortunately, the geodinfo() API pre-dates the object-oriented Nasal bindings, so is kinda difficult to change without breaking legacy code.

In other words, you would probably not want to port geodinfo "as is", but make it return an object with methods to compute things on demand, which should be quite a bit more efficient than the current geodinfo() implementation, whose results are often discarded anyway.

PS: It's not at all Nasal specific, the APIs is indeed used elsewhere - e.g. see the hard-coded terrain presampler that is property-driven (Torsten's work).

EDIT: Regarding those unit testing related commits, it might be easier to just look at fg_props.cxx and copy the whole shebang of fgSet*/fgGet* helpers to the SGSubsystem class, and then move your "setRootPropertyTree()" method to the SGSubsystem class, too - that way, basically none of the existing code would need to be changed, because class members would take precedence over free-standing functions, i.e. the "local" getters/setters would be used automatically by existing code using those APIs. Equally, that would make it straightforward to provide each SGSubsystem not only with a pointer to the global tree, but also with a private property tree for its own properties, which it may register in the global tree, so that the SGSubsystem could be invoked in a SGThread, and all read/write requests could be dispatched to the local/private tree, without much needed in terms of synchronization, because there would only ever be a single writer. Besides, the whole step of decoupling a SGSubsystem from globals has already been done previously (e.g. see FGRadar, which is using a SGSubystemMgr loop to reuse existing SGsubsystems, including FGNasalSys)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Fri Feb 05, 2016 9:59 pm

Hooray: I work on a c++ library for python that use some code that i port from the c++ code that use for nasal api.
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Sat Feb 06, 2016 4:05 pm

sorry for the double post.
I have some feed back and a idea for ruing code in the main thread:
1. I found out that the code run in <python><foo><script> run in the global interpreter.
Were <python><bar><script> can access the data if <foo>
Code: Select all
    <python>
      <foo>
        <!--All the code run current in the global interpreter -->
        <script>
          a = 'foobar'
        </script>
      </foo>
      <bar>
        <script>
          print(a)
        </script>
      </bar>
    </python>

A better way is:
Code: Select all
    <python>
      <foo>
        <!-- code run in foo-->
        <script>
          a = 'foobar'
        </script>
      </foo>
      <bar>
        <script>
        <!-- code run in bar-->
          print(foo.a)
        </script>
      </bar>
    </python>

2. I think we can move from setlisener/settimer to a class base system:
Code: Select all
import fgloop

class LoopTestClass(fgloop.loop):
    def __init__(self):
      self.propdata = ''
      self.timerdata = 'false'
      self.timeloop = 0
     
    def propfunc(self, propdata):
        # this function is call whene propetry is chance and past the new value to the functuon
        self.propdata = propdata
       
    def timerfunc(self):
        # this functuion run once
        self.timerdata = 'True'
       
    def timeloop(self)
        # this functuion run every second
        self.timerloop += 1
       
    def frameloop(self):
        # this is the main loop of this script
        print(self.propdata)  # print the current data from the prop tree
        print(self.timerdata) # print ture if the have running once
        print(self.timerloop) # print time that is eleps from the moment that self.run is execude
   
    def run(self.looptickrate):
       self.maintimeloop = self.settimeloop(self.timeloop, 1) # this function be execude one pre second
       self.mainframeloop = self.setframeloop(self.frameloop, 25) # this function be execude once in 25 franes
       self.proploop = self.setprop(self.propfunc, "/foo/bar") # this function execude when fg decete a chance in the /foo/bar propertree
       self.timer(self.timerfunc, 42) # this function run after 42 sec once
       """
       this are the control api settimeloop, setframeloop and setprop:
       setrate: this the update rate in seconds or frames this is not available if using setprop
       stop: Supents the function from execution in the loop
       start: resume execution of the function
       """

LoopTest = LoopTestClass()
LoopTest.run()
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Sat Feb 06, 2016 5:12 pm

bugman mentioned already that he doesn't want to support timers in their form - anyway, if you'd like to know more about their implementation and how they relate to Nasal callbacks invoked via timers, as well as the corresponding fg_init.cxx/SGSubsystem magic, you may find this interesting: viewtopic.php?f=36&t=23458#p214028

This basically demonstrates how to create a new SGEventMgr instance of the event manager and hook it up to be interleaved with the FDM/AP to run interleaved - while you probably don't want to do that, you may still a few more things about timers, Nasal callbacks and the FlightGear initialization sequence and how SGSubsystems can be hooked up to existing subsystems.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Sat Feb 06, 2016 7:03 pm

bugman wrote in Thu Jan 28, 2016 3:06 pm:Do you think it's really worth adding Python version info to the About dialog? I have HLA/RTI 1.3 compiled into my builds, but this info is not present in the about dialog. It might be better for Stuart to add that instead.



Just for the sake of completeness: https://gitorious.org/fg/hoorays-flight ... eb3b99176f

Subject: FGPython an propose for Python as an nasal alternative
www2 wrote:First i work on an python interface prototype that use Telnet/prop protocol for access the prop tree.



Note that there's a python module to access the property tree via telnet, somewhere in contrib.
If you are interested in implementing protocols directly in Python, you may want to look at the 3-5 commits at, which basically demonstrate how to expose the underlying SG/FG APIs to Nasal space: https://gitorious.org/fg/hoorays-flight ... fgprotocol
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Sun Feb 07, 2016 2:38 pm

Regarding incremental subsystems re-initialization in scripting space, you may find this interesting: https://gitorious.org/fg/philosophers-f ... s/fgcanvas

And concerning scripted bootstrapping, this should be interesting, too: https://gitorious.org/fg/philosophers-f ... -bootstrap
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Feb 08, 2016 11:07 am

I found out that python execute to early for testing geodinfo.
And i wand to see a setlistener or settimer as a temporary debugging tool or a way to call python on demand from a <script>
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Feb 08, 2016 3:22 pm

I am kinda inclined to agree with bugman in that timers and listeners should not be used for such purposes, which is one of the most problematic use-cases for them due to the sheer amount of Nasal system "integrated" with the main loop via timers and listeners.

In the meantime, you could the fdm-initialized property (signal) to wait for the main loop to be up and running, and/or use a Python timer to wait a few seconds before running your tests.

Overall, if FGPythonSys is about learning from FGNasalSys, we really don't want to introduce the same coding patterns - exposing SGSubsystem/Mgr is a much cleaner way to accomplish that.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby www2 » Mon Feb 08, 2016 4:37 pm

@hooray i say that settimer is only as an temporary debugging tool before an better way is crate.
But my preferably way and a better solution is an python terminal.
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Mon Feb 08, 2016 6:06 pm

the issue would then still be that your python interpreter is running in a separate thread, so that it would need to be synchronized/serialized with access to other subsystems, like the tile manager/cache - or it will just segfault.

The "python terminal" can be implemented using the plib netchat class, which is what we are using for the telnet/props daemons in FlightGear - so, implementing a corresponding netchat channel that provides a python terminal would be relatively straightforward
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

PreviousNext

Return to New features

Who is online

Users browsing this forum: No registered users and 7 guests