Board index FlightGear Development New features

FGPython an propose for Python as an nasal alternative

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Tue Mar 27, 2018 9:13 am

This is a crazy idea, but I was thinking of implementing the aircraft-side Python to be a real FG subsystem object with bind(), init(), reinit(), shutdown(), update(dt), unbind(), etc. An aircraft developer would then have direct access to update(dt), so would timers (and their callback implementation) be necessary? Would this lead to sloppy aircraft development weighing down the main loop, by over using update(dt) rather than using listeners? The aircraft-side Python subsystem would be separate from the fgfs-side Python subsystem, though probably would run serially on the same Python instance until OpenRTI support is improved and we can offload the two subsystems into two different processes.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby curt » Tue Mar 27, 2018 12:49 pm

2c (or less) worth of my thoughts:

By whatever mechanism (call backs, timers, threads, or directly inserted into the main loop) code has to run when it needs to run and takes as much time as it needs to take. You can gain some efficiency (possibly) by putting code that blocks on I/O into it's own thread and letting that thread eat the blocking wait time. And of course, you can possibly push tasks into kernel threads to take advantage of multiple cores (but every one's system has a different # of cores.) I've seen a long list of threading and real time system #fails over the years, so (in my opinion) threads and multiple cores and real-time operating systems aren't just magic sauce that makes everything work. If you want something to happen at the right time or right interval, you have to make sure it happens! Threads and cores are tools, but they are difficult tools to use. It is hard work to get things exactly right (and keep things working exactly right when new code tweaks are pushed to the repository.)

So if some code needs to run every frame, then (personally) I prefer to just put it in the flow of the main loop so that when, where, and what happens is clear and direct. [Of course exceptions for gui style coding for which an event driven model with call backs seems to be the best arrangement ... many cockpit instruments have a user-interface component. And exceptions for code that would have unacceptable blocking wait times, and can't be restructured with non-blocking I/O.]

Again just personally for my own philosophy of programming: simple, clear, direct, linear, fewer lines of code, more readable, == better. I don't always (ever?) fully succeed at creating simplicity in my own work, but more and more this is on my mind. I struggle with modern C++ and some of the crazy looking syntax and code that gets written ... but maybe that's me. It's human tendency to build elaborate things, and then continue adding and refining and making them even more fancy and impressive (but less comprehensible to anyone else.)

So for me: python (or something similar) is a path towards simpler code, simpler structures, simpler concepts while still producing the desired outcome. Modern C++ seems to be adding features that nibble around the edges of what scripting systems already do. Developers recognize the value of the features that scripting languages offer. But scripts do their work with a simple syntax, while modern C++ nibbles on the edges of these concepts and generates crazy looking syntax! I remember the good old days when I could look at some C++ code and actually read it. Modern C++ seems to have lost that human readable aspect. Sorry, just complaining! :-)

I think this is an interesting article in the same vein of what I'm trying to say:

http://cppdepend.com/blog/?p=744

I probably don't agree with every point, but I definitely agree with the underlying train of though.
Aerospace Engineering and Mechanics
University of Minnesota
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: FGPython an propose for Python as an nasal alternative

Postby CaptB » Tue Mar 27, 2018 1:13 pm

curt wrote in Tue Mar 27, 2018 12:49 pm:But scripts do their work with a simple syntax, while modern C++ nibbles on the edges of these concepts and generates crazy looking syntax! I remember the good old days when I could look at some C++ code and actually read it. Modern C++ seems to have lost that human readable aspect. Sorry, just complaining! :-)


Agreed. Since there was no news on this:https://forum.flightgear.org/viewtopic.php?f=4&t=31628#p304944 I wanted to see if I can do the work on my own, but what I saw looked sufficiently unfamiliar to get me discouraged even though I used to program in pure C and a little bit in C++ in the old DOS Turbo C++ days.
Ongoing projects(3D modelling): A320, MD-11, A350, B767
FG767: https://fg767.wordpress.com/
CaptB
 
Posts: 685
Joined: Thu May 23, 2013 7:36 pm
Callsign: EKCH_AP
IRC name: CaptB
Version: next
OS: Xubuntu

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Tue Mar 27, 2018 7:39 pm

It's not such a crazy idea at all - a number of core devs have previously pointed out how timers and listeners are pathetic tools to implement complex subsystems on top of them running inside the fgfs main loop. Furthermore, a few people suggested that had Nasal had SGSubsystem/Mgr wrappers exposed to scripting space, more Nasal features could be aligned much better with the main loop, and especially main loop events ("signals") - especially referring to reset/re-init, but also saving/loading and resuming flights (sessions)

However, as things are standing currently, there still is a significant bunch of C++ code that isn't even using the SGSubsystem/Mgr APIs properly (or even not at all).
Besides, the multi-instance use-case is one of the most important ones (FSWeekend/LinuxTag-type setups, but also anything involving MP/dual-control or other setups having a need to selectively share/replicate certain state), but this is something that isn't designed into the SGSubsystem/Mgr architecture
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Tue Mar 27, 2018 8:09 pm

@Curt: My coding philosophy is also simplicity and I strongly avoid programming language specific idioms not present in other languages (e.g. in Python, no lambdas!). But sometimes simplicity comes with new programming features and I'll use that straight away. I like this C++11 way of iterating over a std::vector:

Code: Select all
for (auto i : my_vector) {
    // Code
}


@Hooray: Exposing update(dt) means that you essentially create the timer on the aircraft side. There are no callbacks. How you run a function at a given rate is up to you (track "now" like in SGEventMgr" or sum "dt"). Or a callback system independent of SGEventMgr could be added in the Python aircraft subsystem.

Hooray wrote in Tue Mar 27, 2018 7:39 pm:However, as things are standing currently, there still is a significant bunch of C++ code that isn't even using the SGSubsystem/Mgr APIs properly (or even not at all).


We'll have to fix that then ;)

Hooray wrote in Tue Mar 27, 2018 7:39 pm:...but this is something that isn't designed into the SGSubsystem/Mgr architecture


The subsystem manager is pretty basic at the moment. I will need to expand this significantly for the now integrated test suite (which came from the first FGPythonSys branches). Once I manage to make subsystems more independent, support for multiple copies of subsystems or multiple subsystem managers isn't a difficult task.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Tue Mar 27, 2018 8:33 pm

Mathias' latest public statements regarding HLA/RTI (SimKit at that point) suggested that he was going to overhaul the SGSubsystem accordingly, i.e. so that subsystems could be trivially moved to separate threads, but also processed, using the same code.

But even without looking at HLA/RT, there is quite a bit of existing stuff that could be used to make this happen - for instance, any SGSubsystem could have its own private SGPropertyNode, to provide a subsystem-specific property tree. Equally, there's existing stuff like SGSocket which would make it possible to hook up a simple RPC interface to SGPropertyNode, so that properties can be remotely updated/read, analogous to Erik's original idea: http://wiki.flightgear.org/Remote_Properties

At that point, you'd have a socket-enabled property tree, which could in theory also be mounted in another property tree, at the mere cost of adding a friend class to SGPropertyNode to dispatch read/update events to another SGPropertyNode instance. Which, in turn, would mean that you could run different subsystems in different threads (think SGThread) and processes - all using existing SimGear APIs.

(Note that this use-case, remotely getting/setting and updating properties, is a fairly common one - and we have meanwhile at least a handful of different methods to do just that: the generic protocol, telnet protocol, multiplayer protocol, Phi (via AJAX/JSON) as well as a number of other hard-coded protocols doing exactly that (think native, fdm, master/slave etc)- it's just something that keeps being re-discovered and re-implemented due to lack of coordination among people interested in basically the same functionality - a "remote property tree" implementation of top of sockets.

For the sake of completeness (and maybe for inspirational purposes), you could also look at Richard's Emesary framework (which is not specific to Nasal at all - in fact, Richard mentioned a few times how he's considering to add C++ bindings, too): http://emesary.codeplex.com/
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Tue Mar 27, 2018 9:01 pm

Hooray wrote in Tue Mar 27, 2018 8:33 pm:But even without looking at HLA/RT, there is quite a bit of existing stuff that could be used to make this happen - for instance, any SGSubsystem could have its own private SGPropertyNode, to provide a subsystem-specific property tree.


This has been present in FGPythonSys for two years now, but I am writing it again for the test suite.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Tue Mar 27, 2018 9:46 pm

FWIW, James originally came up with the PropertyObject<> template that binds to the default root node in fgfs to get rid of tying (tied properties), but this template could be just as well moved to SGSubsystem/Mgr scope, so that any SGSubsystem could trivially use property objects that refer to the private SGPropertyNode/property tree: http://wiki.flightgear.org/Howto:Use_Pr ... ee_Objects

The same applies to the various fgSet*/fgGet* templates in $FG_SRC/Main/fg_props.cxx - that way, they'd overshadow global stuff automatically, so that global APIs would be overridden at the SGSubsystem/Mgr level.

Likewise your notion of an extended SGSubsystemMgr API could also entail moving the subsystemFactory stuff back into the top-level manager - at which point any subsystem (manager) could handle its own reset/re-init without having to maintain a separate factory, which is touching on your idea of having self-contained test suites that handle their own inter/intra-dependencies transparently.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Wed Mar 28, 2018 10:07 am

Hooray wrote in Tue Mar 27, 2018 9:46 pm:FWIW, James originally came up with the PropertyObject<> template that binds to the default root node in fgfs to get rid of tying (tied properties), but this template could be just as well moved to SGSubsystem/Mgr scope, so that any SGSubsystem could trivially use property objects that refer to the private SGPropertyNode/property tree: http://wiki.flightgear.org/Howto:Use_Pr ... ee_Objects


This is very similar to how the property tree is handled in FGPythonSys. I'd forgotten about this. I think the number operators I have implemented for the Python property tree objects is greater, compare:


It might be interesting to convert the 242 Python property tree unit tests (which gives 100% test coverage of the Python property tree) into PropertyObject unit tests and implement all the rest of the operators.

The same applies to the various fgSet*/fgGet* templates in $FG_SRC/Main/fg_props.cxx - that way, they'd overshadow global stuff automatically, so that global APIs would be overridden at the SGSubsystem/Mgr level.


As mentioned on the mailing list, I planning on implementing the fgSet*/fgGet* functions as virtual functions of the SGSubsystem base class. These could then operate on the locally stored property tree, falling back to the global root if not present, and also handling the case of no property tree gracefully.

Likewise your notion of an extended SGSubsystemMgr API could also entail moving the subsystemFactory stuff back into the top-level manager - at which point any subsystem (manager) could handle its own reset/re-init without having to maintain a separate factory, which is touching on your idea of having self-contained test suites that handle their own inter/intra-dependencies transparently.


I have to see, but that was also my idea. To have a smart subsystem manager that can set up a string of subsystem dependencies on demand, as this will make it much easier to write system tests in the new CppUnit test suite framework. So you could write a FGPythonSys system test setUp() function, initialising a new subsystem manager, and asking it for FGPythonSys and X number of other subsystems to be initialised.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Wed Mar 28, 2018 8:12 pm

bugman wrote: To have a smart subsystem manager that can set up a string of subsystem dependencies on demand, as this will make it much easier to write system tests in the new CppUnit test suite framework.

note that we're currently handling CLI startup options in a single location only, i.e. subsystem-specific options are not registered/added via the proper SGSubsystem, so that anything involving standalone testing of certain subsets of subsystems would end up duplicating such code - in other words, it may be beneficial to move the corresponding options.cxx/fg_init.cxx stuff that is subsystem specific into the corresponding subsystems, so that these would expose a virtual interface to register startup options/property mappings at the subsystem level. Referring to: https://sourceforge.net/p/flightgear/fl ... .cxx#l1580

This should simplify any standalone unit testing efforts significantly.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Wed Mar 28, 2018 10:14 pm

@Hooray: I don't know if that code would be useful. The subsystem dependency chain is hardcoded in the fgCreateSubsystems() function of fg_init(). What I need is more flexibility. A subsystem returns a list of essential dependencies (and optional dependencies). Then the subsystem manager asks those dependencies for their lists of dependencies, and so on. Then the init order can be dynamically determined by the subsystem manager based on what is asked for. For the testing though, you need to say please only start X, Y and Z, i.e. the dependency resolution code is not triggered. These ideas are not present in FG yet, but James and I have talked a little about it on the mailing list.

Regards,
Edward

Edit: That gives me ideas for starting a new mailing list thread :)
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Thu Mar 29, 2018 5:31 pm

You missed my point, I am aware of how subsystems are currently initialized in a hard-coded fashion - as a matter of fact, I mentioned previously, that I touched that code a while ago to move subsystem initialization into scripting space to make it configurable/scriptable analogous to run-level scripts, the set of patches/diffs can still be found here (gitorious links): http://wiki.flightgear.org/Initializing_Nasal_early

I'd suggest to consider this outdated, and not in line with your cppunit based approach, but conceptually it may still be interesting to look at, because it's a straightforward mechanism to have our cake and eat it, and would also work without going through Nasal explicitly (say using XML or property rules instead).

However, my actual point was that registration of subsystem specific CLI options should/could also be moved into the corresponding subsystems, too.
In other words, this would encapsulate all house-keeping stuff, including even if/how subsystems are re-initialized, as long as you deal with "groups" of subsystems (SGSubsystemMgr)

One major issue here is that even code "already" using the legacy SGSubsystem/Mgr interfaces, still doesn't have any notion of supporting multiple instances of a subsystem using different names/indices - which has been highlighted in numerous discussions over the years, dating even back to times when David Megginson, Erik and Curt were still hoping to support multiple instances of a "fdm" by having fdm[0], fdm[1] and fdm[2] branches in the property tree - which could in turn also be used for other in-sim purposes (think AI traffic).
The original idea was to also "bind" separate autopilot/route-manager instances to such fdm[x] trees, to make it all work "magically".

Pointers/quotes to be found here: http://wiki.flightgear.org/Status_of_AI_in_FlightGear

Obviously, none of this ever happened - yet, a number of people interested in working on HLA/RTI did independently come up with the conclusion that this would be a worthwhile goal to aim for - Durk's most recent summary being available here: http://wiki.flightgear.org/FGTraffic
Agreed on everything else you said.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby Hooray » Sat Mar 31, 2018 10:49 am

Referring to: https://sourceforge.net/p/flightgear/ma ... /36280483/
bugman wrote:I think I need to experiment a little. A nice initial
isolated subsystem set would be useful to find.


Again, it may be worthwhile to look at some of the previous work in that area (especially to help you understand "idle states") - for instance, the topic branches initializing Nasal earlier I mentioned previously (which were created in response to James asking for that originally) are able to dump such information to the console (again, I am still not suggesting that you use any of this, but it may still help you make heads and tails of the subsystem "architecture" currently in place, I really don't care about the patches/code, what I do care about however is the functionality, excerpts from relevant commit messages shown below):


alternate/subset startup modes, e.g. a FGCanvas mode or a headless mode. We can now also display the
Aircraft Center right after starting.

Mainly, these patches will:
- move FGNasalSys out of the global SubsystemMgr into the FGGlobals class, along with getters
- update fgMainLoop() to explicitly call the Nasal::update() method there
- fix up all globals->get_subsystem("nasal") lookups to use globals->get_nasal() instead
- move initialization of subsystem-specific APIs into each subsystem's bind() method
- the interpreter is now instantiated at the beginning of the init routine
- currently, it's right after fgcommands are added, as those are likely to be needed anyway
- reset/re-init handling is augmented to directly access the instance or allocate a new one

For testing purposes, there some CLI flags added, these will probably disappear once $FG_ROOT/Nasal is fixed up:

--enable-scripted-init - passes control to a configurable boot script in $FG_ROOT/Boot
--mode=filename - allows the boot file to be specified, defaults to default.boot

--enable-early-nasal-init - initializes Nasal earlier, to see which scripts still need to be fixed

We're hoping to re-implement bootstrapping in Nasal space using Philosopher's code, which would then
be moved into $FG_ROOT/Boot/default.boot - once that works, we'll still have to fix up $FG_ROOT/Nasal
to make subsystem dependencies more explicit there.

Once that is the case, the two --enable-... options can go, and we can easily implement additional startup
modes by adding other *.boot files

Hopefully, this will also allow us to more easily do regression testing in an automated fashion-for
example by adding a new "unit-test" startup mode that will run certain APIs to do a self-check.

Likewise, we could also use this to do headless benchmarking/profiling and debugging, possibly even
right on the build server (minus all the visuals obviously).

Future changes will be mainly about:

- moving fgCreateSubsystems() into scripting space
- extending the subsystemFactory accordingly
- extending fgIdleFunction() to allow some of it to be scripted (e.g. splash screens)

Additional info at: http://wiki.flightgear.org/Initializing_Nasal_early



- move initialization of event_mgr into fgIdleFunc() because Nasal timers depend on events
- main.cxx/fg_init.cxx extract init logic into standalone helper functions and expose them to Nasal
- fg_init.cxx split up fgCreateSubsystems() and use a separate function for "early" subsystems
- preparations for creating & initializing a subset of subsystems earlier than the others
- add a few new "signals" after intializing subsystems, so that Nasal listeners can pick those up for dependencies
- expose idle-state as a propertyObject so that Nasal listeners can be registered
- re-order idle-state increments to the bottom of each block, because using listeners will otherwise invalidate things

NOTE: This patch will now assume that there's ALWAYS at least a default boot script in $FG_ROOT/Boot/default.boot
The script can be empty though - an empty file will do.

But make things more interesting, add this to the file:

var p = "/sim/startup/current-idle-state";
var update = func(n) {
print(p," changed:", var n =getprop(p));
}

_setlistener(p, update);

When booting/resetting, You should get notifications for each change in idle-state, such as:

Scripted initialization, now passing control to:Path ".../fgroot/Boot/default.boot"
/sim/startup/current-idle-state changed:3
Nasal interpreter initialized
/sim/startup/current-idle-state changed:4
/sim/startup/current-idle-state changed:5
/sim/startup/current-idle-state changed:7
/sim/startup/current-idle-state changed:8
/sim/startup/current-idle-state changed:9
/sim/startup/current-idle-state changed:10
/sim/startup/current-idle-state changed:900
/sim/startup/current-idle-state changed:1000

There's a new _boot namespace exposed that contains a handful of init routines so that initialization can
be easily re-implemented and customized using a few lines of Nasal in each boot script.

For the time being, FGNasalSys::postinit() will default to running the hard-coded bootstrapping
scheme - but boot scripts can override this via:
setprop("/sim/startup/disable-hardcoded-nasal-bootstrapping",1);

This option will disappear once scripted bootstrapping works properly.

The next set of patches is going to make fgCreateSubsystems() better configurable.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Tue Jun 18, 2019 11:22 am

Yet another rebasing update to my FGPythonSys branches to 'next'. Again there have been no significant changes to FGPythonSys itself, only those changes required for keeping the branches up to date:


As the CppUnit branches have been merged to next, these are instead now rebased onto my subsystem refactoring branches.


Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: FGPython an propose for Python as an nasal alternative

Postby bugman » Thu Jan 09, 2020 11:08 am

Here is the next rebasing update to my FGPythonSys branches to keep them alive (again with no significant changes):


Note that Python is no longer built by default, so the CMake option "-DENABLE_PYTHON=ON" should be used. These are rebased directly to the official repositories' 'next' branches as of today. To summarise, these build the embedded Python interpreter into the fgfs and fgfs_test_suite binaries and currently only implement a Python-object-based property tree (supporting all of the Python operators). Adding Pythonised listeners and timers would be the next infrastructure to implement.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

PreviousNext

Return to New features

Who is online

Users browsing this forum: No registered users and 4 guests