Board index FlightGear Support

Crash - deleting still referenced object

All general support: help on flying, installation, hardware, getting online etc. There are lots of users and developers to help you out.
Forum rules
In order to help you, we need to know a lot of information. Make sure to include answers to at least the following questions in your initial post.

- what OS (Windows Xp/Vista, Mac etc.) are you running?
- what FlightGear version do you use?
- what graphics card do you have?
- does the problem occur with any aircraft, at any airport?
- where did you download your aircraft/scenery from?
- is there any output printed to the console (black window)?
- copy&paste your commandline (tick the "Show commandline box on the last page of FGRun or the "Others" section on the Mac launcher).

If you experience FlightGear crashes, please report a bug using the issue tracker (can be also used for feature requests).
To run FlightGear on old computers with bad OpenGL support, please take a look at this wiki article.

Note: If you did not get a reponse, even after 7 days, you may want to check out the FlightGear mailing lists to ask your question there.

Re: Crash - deleting still referenced object

Postby wkitty42 » Mon Jan 07, 2019 6:34 pm

legoboyvdlp wrote in Mon Jan 07, 2019 4:12 pm:sure - will do - although I can't really see how the autopilot flight plan makes much difference...

because he doesn't have to manually try to reproduce your flight... he can play the tape and go exactly as you did while he's watching his debugging tools showing the code and variable values ;)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5018
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Crash - deleting still referenced object

Postby legoboyvdlp » Mon Jan 07, 2019 7:00 pm

Ah, I see...

Regardless, shouldn't it be possible to catch these errors and keep going? The only thing I can think of is that if it occurs something inside Flight gear or OSG is so messed up it would be of no benefit to continue....

I also couldn't get OSM roads to work, no idea what's going on there. Only the buildings ever showed up. Somewhat strange, but I expect unrelated...
User avatar
legoboyvdlp
 
Posts: 6252
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Crash - deleting still referenced object

Postby wkitty42 » Mon Jan 07, 2019 7:20 pm

WoodSTokk wrote in Mon Jan 07, 2019 5:03 pm:Manual navigating from LOWW to LNZ VOR to SBG VOR to RTT NDB. On the leg from SBG to RTT FG crashed.

i'm testing this now with FG 2018.04... using this linked flightplan which is built from your waypoints...
https://www.dropbox.com/s/9t410ou384nxs2a/LOWW-LNZ-SBG-RTT-LOWI.fgfp?dl=0

currently i'm at 20000ASL @475kts in the UFO approaching LNZ...

[time passes]

now approaching SBG... flight settings still as above...

[time passes]

at half way between SBG and RTT i dropped to 10000ASL... if nothing happens, i'll fly all the way in to LOWI and be there in another 10 minutes or so...
Last edited by wkitty42 on Mon Jan 07, 2019 7:33 pm, edited 1 time in total.
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5018
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Crash - deleting still referenced object

Postby wkitty42 » Mon Jan 07, 2019 7:31 pm

legoboyvdlp wrote in Mon Jan 07, 2019 7:00 pm:Regardless, shouldn't it be possible to catch these errors and keep going?

yes... the thing is to find out where they occur and then put in the catcher...

legoboyvdlp wrote in Mon Jan 07, 2019 7:00 pm:The only thing I can think of is that if it occurs something inside Flight gear or OSG is so messed up it would be of no benefit to continue....

exactly... and it happens sometimes if something isn't coded properly and memory contents get stomped on...

legoboyvdlp wrote in Mon Jan 07, 2019 7:00 pm:I also couldn't get OSM roads to work, no idea what's going on there. Only the buildings ever showed up. Somewhat strange, but I expect unrelated...

yeah, probably not related...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5018
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Crash - deleting still referenced object

Postby iamzac » Mon Jan 07, 2019 7:38 pm

After a few hours of running my script no crash so I was wrong, this is not a "quick" way to make this happen.
But I will still let it run for a day or so to see what happens.
I increased the complexity, now it's looping in 10 steps through 10 busy airports (including some where I had crashes) with 10 different settings and 2 aircrafts.
iamzac
 
Posts: 96
Joined: Fri Nov 07, 2014 7:52 am
OS: Windows / Kubuntu

Re: Crash - deleting still referenced object

Postby Richard » Mon Jan 07, 2019 11:50 pm

legoboyvdlp wrote in Mon Jan 07, 2019 7:00 pm:Regardless, shouldn't it be possible to catch these errors and keep going? The only thing I can think of is that if it occurs something inside Flight gear or OSG is so messed up it would be of no benefit to continue....


It's not possible to catch the error and continue because the scenery object that has just been loaded in the database pager thread will have parts of itself destroyed by the main thread and so pretty much the next time it is rendered parts of it will be pointing to memory that is no longer allocated and could contain anything that reuses this memory. So it might work for a little while but almost certainly it will crash.

Also saved flightplans help as it means that I don't have to try to match where you flew it just helps to recreate the circumstances of a crash and therefore helps with the debugging process; I started using saved flightplans to find the problem and it helped me enormously to have a consistent point of failure (although in the first instance this was after only 4h30m)


-----------------------------------
If you want to ignore the error and continue (and can build from source) then open simgear\debug\OsgIoCapture.hxx and comment out line 26;

Code: Select all
            //throw new std::string(message);


on older simgear (2018.3) it will be the two lines
Code: Select all
            //int* trigger_segfault = 0;
            //*trigger_segfault     = 0;
Richard
 
Posts: 661
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Re: Crash - deleting still referenced object

Postby Hooray » Tue Jan 08, 2019 12:12 am

Yes, if possible, please always include at the very least a flight plan so that the portion of the flight can be reproduced more easily.

By the way, that's also an excellent way for people to easily test if an issue is specific to a certain aircraft or environmental setting (especially in conjunction with increased simulation time).

Originally, we were using pre-recorded flights to stress-test fgfs and try to reproduce certain issues, then Instant replay got added, and meanwhile there is the "flight recorder" subsystem.

When it comes to troubleshooting, it really makes sense to use these features.

Note that if people generally shared such flights, it would be possible to grow a library of flights (plans) and in theory they could be executed automatically in a scripted fashion on some build server (think jenkins), e.g. in a headless fashion running inside a gdb session with an increased simulation rate, for troubleshooting purposes.

Over time, something like this could become a huge asset (think regression testing), not just referring to bugs, but also to performance troubleshooting (profiling) - imagine being able to gather the memory requirements for the same flight (plan) using different fgfs versions, or the frame rate/spacing.

In other words, it would be an excellent idea for people to always create, provide and share such flights when making bug reports - likewise, it would be an excellent idea for core developers to commit such files to /some/ repository so that we can use them in the future for regression testing purposes.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11292
Joined: Tue Mar 25, 2008 8:40 am

Re: Crash - deleting still referenced object

Postby iamzac » Tue Jan 08, 2019 4:38 am

After 12 hours of continuous running of the script finally a crash, this time at PHNL after 354 loops of the 10 airports script.
Now removed the other 9 airports and I will only repeat the loop with PHNL that caused the crash to see if it will repeat faster now.
I have the 3 crash report files in case they are necessary.
iamzac
 
Posts: 96
Joined: Fri Nov 07, 2014 7:52 am
OS: Windows / Kubuntu

Re: Crash - deleting still referenced object

Postby Richard » Tue Jan 08, 2019 12:39 pm

I've made changes to OSG (using a ReadWriteMutex), so that the database loading thread acts as a reader and the expiry a writer; what this means is that it will only be able to expire objects from the cache when no loading is underway.

Initial testing is promising; although only a few hours of testing so far the debug log is showing the right sort of activity. Each of the "...cannot check for expiry as something is accessing the object cache" messages is a potential gateway to the bug occurring.

Code: Select all
6996.78 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.81 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.85 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.87 [INFO]:terrain    Loading tile 3253640.stg
6996.88 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.88 [INFO]:terrain    Loading stg file Path "i:/flightgear/project3000/Objects/e010n60/e018n60/3253640.stg"
6996.89 [INFO]:terrain    Loading stg file Path "i:/flightgear/terrasync/Objects/e010n60/e018n60/3253640.stg"
6996.89 [INFO]:terrain    Loading stg file Path "i:/flightgear/terrasync/Terrain/e010n60/e018n60/3253640.stg"
6996.92 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.95 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6996.98 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
6997.05 [WARN]:OSG        ObjectCache::removeExpiredObjectsInCache expired cannot check for expiry as something is accessing the object cache
7007.06 [ALRT]:nasal      set tgt Ttype multiplayer mdl=c172p
Richard
 
Posts: 661
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Re: Crash - deleting still referenced object

Postby legoboyvdlp » Tue Jan 08, 2019 2:51 pm

The fact that it's not able to remove expired objects until it loads those tiles suggests it might be tile loading at fault - trying to remove them too early?
User avatar
legoboyvdlp
 
Posts: 6252
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Crash - deleting still referenced object

Postby Richard » Tue Jan 08, 2019 3:30 pm

legoboyvdlp wrote in Tue Jan 08, 2019 2:51 pm:The fact that it's not able to remove expired objects until it loads those tiles suggests it might be tile loading at fault - trying to remove them too early?


You probably should read my message on the devlist as that's got my diagnosis of the problem (although I may have badly explained it); and my most recent post is about a possible fix for this by imposing order on the threads that are both using the object cache.
Richard
 
Posts: 661
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Re: Crash - deleting still referenced object

Postby Richard » Tue Jan 15, 2019 1:12 pm

Thanks to the folks over at OSG reviewing and rejecting my pull request with a good explanation of what we should be doing this is something that needs fixing in simgear.

I've made the changes and I'm doing one of my standard long route tests to verify it - assuming that there are no problems the fix will be in simgear shortly and we'll backport it to 2018.3.2
Richard
 
Posts: 661
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Re: Crash - deleting still referenced object

Postby legoboyvdlp » Tue Jan 15, 2019 1:27 pm

Just to be absolutely certain, there is no need to recompile OSG?
User avatar
legoboyvdlp
 
Posts: 6252
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Crash - deleting still referenced object

Postby Isaak » Tue Jan 15, 2019 2:00 pm

I 'm so grateful that you took the time to fix this bug. I encoutered it again today (suddenly had random crashes again in the last month) for the second time when I was flying from LEBL to EHAM (it also happend Sunday). First time it happened somewhere over France, while I was cruising. The second time it happened while descending into EHAM, on the DENUT1A arrival shortly before RIVER. I thought it had something to do with the (quite unstable) VATSIM bridge, which was why I didn't report it yet, but I 'm glad there 'll be a fix shortly.

Flightplan is here.

Windows 10 with FG 2018.3.1 here.
User avatar
Isaak
 
Posts: 250
Joined: Sat Jun 04, 2011 2:52 pm
Location: Leuven, Belgium
Callsign: OO-ISA
Version: 2018.3.2
OS: Windows 10

Re: Crash - deleting still referenced object

Postby wkitty42 » Tue Jan 15, 2019 6:22 pm

legoboyvdlp wrote in Tue Jan 15, 2019 1:27 pm:Just to be absolutely certain, there is no need to recompile OSG?

no... no need to recompile OSG... the fix is to change some calls in SG to use the "ref" related ones... that's if i understood what i read in the topic thread in the mailing list and their forums when reading the report...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5018
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

PreviousNext

Return to Support

Who is online

Users browsing this forum: No registered users and 1 guest