Board index FlightGear Development Aircraft

Note to aircraft maintainers - 2019.X

Questions and discussion about creating aircraft. Flight dynamics, 3d models, cockpits, systems, animation, textures.

Note to aircraft maintainers - 2019.X

Postby Thorsten » Sun Nov 17, 2019 9:35 am

To all who aren't following the mailing list:

I suspect there's currently a regression in the core code from 2018.3 - I don't know whether 2019.1 already is affected, 2019.2 surely is.

The problem I've tracked down has to do with channels in JSBSim, but possibly it is more general.

The phenomenology of the issue are random crashes of a Nasal system at state changes which go hand in hand with systems changes (AP switches to a different mode, seemingly ignores the command and becomes inoperable). The reason is that there appears to be a change in the way properties are initialized by the system channels, which leads to a racing condition with Nasal - if the systems win, they're ready when Nasal requests a property, but if Nasal wins then it gets and empty property node and exits - which is why the system ceases to work.

Because this is a racing condition, it might affect faster hardware more badly than slower hardware (the systems computation runs at fixed speed, but Nasal is usually faster on faster HW) - so this can lead to a situation where a user reliably experiences a bug which the aircraft maintainer can not reproduce at all.

Also, if the aircraft is tested and developed on 2018.3, everything is fine and only a user who has a more recent FG version will experience these changes.

At this point I am not sure of any details, I have reported my findings to the mailing list - this message is to alert aircraft maintainers to what is going on in case they have to deal with bug reports which seemingly can't be reproduced at all.

I have reported the issue to the mailing list along with my preliminary analysis - so far not much reaction - my recommendation at this point would be to do a regression testing in case such vague and erratic problems are reported, i.e. try whether they persist when going back to 2018.3 or not.
Thorsten
 
Posts: 11720
Joined: Mon Nov 02, 2009 8:33 am

Re: Note to aircraft maintainers - 2019.X

Postby Hooray » Sun Nov 17, 2019 9:42 am

Not sure if this helps or not, so feel free to disregard (I am not even lurking on the mailing list ...)

From a regression testing standpoint, doing a git bisect would obviously be useful - but given that JSBSim can also be linked to fgfs in standalone mode (external fdm option), that option might be more straightforward to see if a different JSBSim version exhibits the same problem or not, unless the changes are of course on the fgfs side of things ?

I am not sure if the build server provides jsbsim binaries or if there is any other source that could be used to help troubleshoot the problem ?
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11925
Joined: Tue Mar 25, 2008 8:40 am

Re: Note to aircraft maintainers - 2019.X

Postby Thorsten » Sun Nov 17, 2019 9:44 am

I doubt we have a test case of Nasal interacting with an external FDM (not sure whether that is even possible) - I believe JSBSim standalone is fine, because there the system gets always initialized before needed.
Thorsten
 
Posts: 11720
Joined: Mon Nov 02, 2009 8:33 am

Re: Note to aircraft maintainers - 2019.X

Postby Hooray » Sun Nov 17, 2019 9:51 am

As far as I remember, this is using something called "FDMShell", which is an abstraction that dispatches local property accesses via an I/O mechanism (protocol) to the standalone FDM - thus, it should not look to Nasal like anything special, it should basically "connect" to the FDM and obtain a copy of the new /fdm state vector, and then copy that into the local (global) fgfs property tree.

Again, this could very well be a red herring, but we should have details about interfacing fgfs and jsbsim in standalone mode - I believe a number of folks have been using that to run jsbsim out of the main loop.

Back in the early days of the project, that used to be a rather common use-case, not sure how much of this still works today - but I believe Curt was the one to originally come up dedicated protocols to dispatch FDM inputs and outputs between different fgfs related processes. But, I would have to check the docs to see if/how this is supposed to work. I guess someone involved in jsbsim, e.g. Erik or AndersG would be in a better position to judge if any of this makes sense from a troubleshooting standpoint or not.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11925
Joined: Tue Mar 25, 2008 8:40 am

Re: Note to aircraft maintainers - 2019.X

Postby Thorsten » Sun Nov 17, 2019 10:14 am

Again, I see no particular reason to assume that a racing condition in a standard FG use case is readily reproducible using an external FDM - if you see any benefit in investigating this, please do, but admittedly I don't get the idea.
Thorsten
 
Posts: 11720
Joined: Mon Nov 02, 2009 8:33 am

Re: Note to aircraft maintainers - 2019.X

Postby bugman » Sun Nov 17, 2019 10:39 am

We don't currently have a test case set up that way, but it is certainly possible. The test suite binary contains everything required (it is essentially the fgfs binary with nothing initialised due to not having the boostrapping and init sequence running). I just would need to set up Nasal and JSBSim in a test fixture setUp() function. The bigger problem is first identifying more precisely where the problem is happening so that the problem is possible to reproduce. If that could be done, it might be fun test code to write.

Regards,
Edward
bugman
Moderator
 
Posts: 1797
Joined: Thu Mar 19, 2015 9:01 am
Version: next

Re: Note to aircraft maintainers - 2019.X

Postby Hooray » Sun Nov 17, 2019 10:46 am

According to Thorsten's findings, this would be somewhere at the FDMShell layer - I believe, JSBSim is still using tied properties, so adding some SG_LOG() statements or logging tied properties under /fdm might be a good starting point to see if Thorsten's ordering theory can be reproduced.

From a troubleshooting standpoint, it would be interesting to know if this can be nailed down to any particular properties, i.e. /autopilot vs /fdm ?
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11925
Joined: Tue Mar 25, 2008 8:40 am

Re: Note to aircraft maintainers - 2019.X

Postby wkitty42 » Sun Nov 17, 2019 3:12 pm

FWIW: i just sent this message to the list... not sure if it helps or not...
wkitty42 wrote:On 11/17/19 7:07 AM, Thorsten Renk wrote:
>
>> Which change are you talking about ?
>
> The ones which cause properties inside a channel not be initialized before the channel goes active. I don't have a commit identified, I have a rather elusive and mean bug I'm trying to chase here and the only solid bit of evidence I have for it is
>
> * in 2018.3, the TAEM-specific AP channel is probed by the HUD even when not active and returns valid properties at all times
>
> * in 2019.2, the TAEM specific AP channel is likewise probed by the HUD even when not active and returns empty nodes when not active but valid values when active


IF i've done this properly, this should return the list of commits from when 2018.3 was cut till when 2019.2.0/2019.1.1 was cut... if i'm reading properly, 2019.1.1 and 2019.2.0 were both cut within a second of each other...

~/flightgear-dev/next/flightgear$ git log 518d3cbd..f007dd3e

looks like only 173 commits to look at... 12 of those from Bertrand...

~/flightgear-dev/next/flightgear$ git log 518d3cbd..f007dd3e | egrep -A1 -e "^commit [a-z0-9]{40}" | egrep -B1 -e "Bertrand" | egrep -e "^commit "

commit f57d7329c2c3a91ae81d75852f09d3dd8019c724
commit 9c7243e1c432bb26fbcc0fad87c2e6ff7acff560
commit e704d589f219c6cb605ea905f1516d480a789196
commit 466fb0979dcaf3172d016364521c6e8941dc54b2
commit 27ddcedad277cae08008364ad004372ce7765466
commit 3267aaf5e032b8083138c36de9a5b5064a4ae066
commit 8742fee23d28ab98238b04aaa2560a98230b151e
commit d3aa4b19a1315bd80911255924dd47d850a37722
commit 4c7402fb23520a0fbd3a5701de3d391f0136915d
commit 04eb0459314558199411a54c9bb8000ca2eebe2b
commit 94e1cdc5514d862d0b1e4a2e59cba2d99078d0b4
commit 8e05816b43e257d7bb429bcedcb5a5dd707d7473

but looking at them and knowing that i know nothing other than what is written in the commit message, i see nothing in them that jumps out at me... i suppose it could easily be something else not specific to JSBsim...


again, if i've done this properly, this will return the 78 commits in simgear just as above...

~/flightgear-dev/next/simgear$ git log f4cad429..f9643740

"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 6603
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Note to aircraft maintainers - 2019.X

Postby Hooray » Sun Nov 17, 2019 4:30 pm

FWIW, if this is about a property value being nil in a numerical context, the usual idiom/workaround is defaulting any getprop calls, as per:

http://wiki.flightgear.org/Nasal_Condit ... Defaulting
Code: Select all
var x = getprop("/sim/foo/bar") or 0.00;


If safe-guarding all property accesses like that, gets rid of the problem, the corresponding properties are n/a at the access time, which could indeed be said to be akin to a "race condition".

The "cheap solution" would be overloading the getprop API to make sure that this behavior is the default behavior - of course getprop() itself accepts a default value, too - it's just that this isn't commonly used.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11925
Joined: Tue Mar 25, 2008 8:40 am

Re: Note to aircraft maintainers - 2019.X

Postby Thorsten » Sun Nov 17, 2019 5:54 pm

FWIW, if this is about a property value being nil in a numerical context, the usual idiom/workaround is defaulting any getprop calls, as per:


I know what the workaround is - I'm just not keen to apply it to a couple of hundred situations which are randomly triggered and distributed across an unknown number of aircraft in the repository.

And changing getprop might have fatal consequences of its own...
Thorsten
 
Posts: 11720
Joined: Mon Nov 02, 2009 8:33 am

Re: Note to aircraft maintainers - 2019.X

Postby legoboyvdlp » Wed Dec 04, 2019 6:29 pm

Has - anyone - reported any problems with 2019.1 or is it only 2019.2?
User avatar
legoboyvdlp
 
Posts: 7750
Joined: Sat Jul 26, 2014 1:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: Note to aircraft maintainers - 2019.X

Postby wlbragg » Wed Dec 04, 2019 6:35 pm

Good question, I have been running 2019.1 and haven't noticed any issues. Not until I built 2019.2.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
Intel i5 3570K AMDRX480
User avatar
wlbragg
 
Posts: 5734
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/AMDRX480

Re: Note to aircraft maintainers - 2019.X

Postby Hooray » Wed Dec 04, 2019 7:32 pm

If that's so, it would make sense to look any Nasal related commits since then and make those optional to see if the issue disappears or not - property stuff (props or setprop) is using so called Nasal ghosts, which are "objects" that wrap a C/C++ pointer, this should show up via "git bisect". If it only involves property nodes, I'd suggest to look at any SGPropertyNode related changes since then.

And thinking about it, since there was an initialization related issue involving jsbsim aircraft, it might be interesting to check whether non-JSBSim aircraft/FDM are affected to or not ?
If they are not, it could be an initialization order thing again, which would make sense in the given context, because the jsbsim folks started phasing out tied properties, which are all about initializing things once you think about it.

That should be cheaper to check for than running git bisect ...

Background: There is something like the "FDMShell" (or ShellFDM) mechanism used, and JSBSim has been using tied properties for almost two decades, Andy implemented YASim to simply copy properties instead. All other/non/pseudo FDM stuff should also not be using tied properties. However, tied properties are all about initializing a property by binding it to the pointer/address of the POD - thus, not getting a handle to the property could plausibly yield a nil/Nan as long as fdm shell stuff isn't updated accordingly. Again, that's off of the top of my head, that's all code dating back to the early 2000s, back when Nasal and YASim where still in their infancy. But, personally, I would certainly confirm first that the issue affects non JSBSim aircraft (unless that's already been established and I missed that, still only skimming ...so sorry for any noise).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11925
Joined: Tue Mar 25, 2008 8:40 am

Re: Note to aircraft maintainers - 2019.X

Postby Thorsten » Wed Dec 04, 2019 8:33 pm

This

viewtopic.php?f=87&t=35079&start=120#p357443

seems to refer to 2019.1, but I don't know whether reverting to 2018.3 fixed it or not.
Thorsten
 
Posts: 11720
Joined: Mon Nov 02, 2009 8:33 am


Return to Aircraft

Who is online

Users browsing this forum: No registered users and 2 guests