Board index FlightGear Support Flying

Stop crashing, fgfs.exe  Topic is solved

Controlling your aircraft, using the autopilot etc.

Re: Stop crashing, fgfs.exe

Postby Hooray » Thu Mar 06, 2014 8:13 pm

adrian wrote in Thu Mar 06, 2014 7:19 pm:
Hooray wrote in Thu Mar 06, 2014 6:37 pm:But ultimately, we're lacking the tools to determine what's really happening behind the scenes.


Eh, it's not actually that dramatic, the tools exist, it may just take some time to run a callgrind or a gdb trace, and it requires some users experienced and willing to do this. Unfortunately, unlike a couple of years ago, there's been a constant stream of Windows users who can't or don't have the tools to do so.

Actually, I wasn't at all referring to "external" tools - because those are obviously available and can be run by people who already know how to build from source, and how to make working c++ changes.
You mention ThorstenB, it's true he was triaging bugs and actually regularly ran things through valgrind apparently and helped track down threading issues, too. But he also provided another extremely integrated tool: the built-in system monitor for instrumenting SGSubsystems.

That's basically exactly what's now needed for ALL subsystems (including the pseudo-subsystem stuff in simgear/scene), but we would ideally also be able to track memory use, too. And right now, we have now good way to tell what's going on threading-wise, but our use of threads is contributing in major ways to segfaults that we've been seeing recently.

Obviously, this is not about existing "tools" not being available - it's about the lack of integrated tools and measures to track resource usage per subsystem. Tools like valgrind, gdb,gDebugger etc need to be first of all understood - and many people around here are obviously not able to use these tools, not even the integrated google-perftools profiler, so we need to make things much more accessible.

And then again, valgrind/gdb/cachegrind/perftools may be too low-level to be meaningful in many cases, i.e. just look at what's going on in the scenery/aircraft department, here just identifying "hot spots" in terms of code/functions would be a red herring, because the real culprit is typically not at all in the runtime code, but the underlying scenegraph data, such as highly complex textures, 3D models or terrain/scenery.
This can now be easily verified by using Zakalawe's new draw-masks to switch off aircraft/scenery/clouds or AI model rendering.

Having acess to gdb/valgrind etc is one thing, being able to actually run and interpret the results is a completely different thing. Thus, it is more promising to optionally expose certain info at runtime, so that users can access it to see how their aircraft/scenery/feature/configuration behaves in comparison to some other aircraft/scenery/feature/configuration.

Also, when it comes to using tools like gdb -and especially valgrind- the main challenge is that FlightGear is primarily a GUI application, and it doesn't lend itself it being debugged/profiled or leak-tested in an isolated fashion at all. For example, for valgrind runs to be useful and straightforward, we would need to be able to minimize what's done inside fg_init.cxx so that we can disable things not needed, but also even run FlightGear in a "headless" mode without a visible GUI window, so that such tests could be run directly on the build server as part of a regression test. Otherwise, even just running fgfs in valgrind for 10 minutes may turn out to take hours due to the nature of valgrind, i.e. it being an emulator.

valgrind et all are great if you know how to use them, but for end users, these tools are too fine-grained, we would ideally support some per-subsystem or per-feature granularity for end users.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Stop crashing, fgfs.exe

Postby Gijs » Thu Mar 06, 2014 8:27 pm

MasteroftheSKIES wrote in Thu Mar 06, 2014 8:01 pm:Here is the code for the skyscraper_01 model.

So?

PS: Notice that the .stg file referes to generic_skycraper_01.xml. Is that file present in your Models/ directory? Yes or no?
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9363
Joined: Tue Jul 03, 2007 2:55 pm
Location: Amsterdam/Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Stop crashing, fgfs.exe

Postby Philosopher » Thu Mar 06, 2014 8:31 pm

I think I tend to agree with Hooray here – it's nearly impossible to see what actually goes in in FG these days simply because there's too many disparate parts involved here. For example, in the rare instance that Nasal could lead to a segfault (don't know of an instance currently, other than using naCall() too recursively and running out of C stack frames, or too much on the operand stack, or something similar), we could potentially see where the C code was using gbd – but that wouldn't tell us where we were in the Nasal code. And that similarly applies to many things, especially things that "jump" boundaries here. Another prime example is listeners - Nasal listeners don't have a backtrace beyond the C code, even if it is triggered directly from a Nasal setprop()/.setValue(), because it passes through C and C++ stack frames and then back into Nasal.

And then also just info like memory usage per subsystem would be yet another helpful data point - yet we don't have it, and need experienced (and not busy!) developers who are really familiar with C++ and can get around the FG codebase to add these things, even though it would help out with several problems - particularly this.

Anyways, I think Saturday night or Sunday would be a good day for running GDB. I'm currently working on Nasal bootstrapping and would have to revert to a ~cough~ working setup ( :wink:) to see if disabling things helps out any. :D
Thanks,
Philosopher
(inactive but lurking occasionally...)
Philosopher
 
Posts: 1590
Joined: Sun Aug 12, 2012 6:29 pm
Location: Stuck in my head...
Callsign: AFTI
Version: Git
OS: Mac OS X 10.7.5

Re: Stop crashing, fgfs.exe

Postby MasteroftheSKIES » Thu Mar 06, 2014 9:07 pm

See my post on page three, maybe some of you missed it because it was at the bottom.
I would use FSX but I use FlightGear because it's a project with goals.

Windows 7; Processor: 8 core 4 GHz; 8GB RAM; Graphics: NVIDIA GeForce GTX 770 (a pretty good graphics card if you're wondering)
MasteroftheSKIES
 
Posts: 216
Joined: Tue Oct 22, 2013 11:10 am
Version: 3.2
OS: Windows 7

Re: Stop crashing, fgfs.exe

Postby Philosopher » Thu Mar 06, 2014 9:12 pm

You posted the .ac file, which is the actual 3D model, not the wrapper FG usually uses in XML form.

P.S. Flying around with UFO across many scenery tiles did nothing. The DHC3F model didn't trigger any crashes, either - even on the same route as the A6M2. Obviously this is incomplete, but suggests that it may be partly aircraft related. The A6M2 is pretty old still, so maybe related to complexity vs new features?
Thanks,
Philosopher
(inactive but lurking occasionally...)
Philosopher
 
Posts: 1590
Joined: Sun Aug 12, 2012 6:29 pm
Location: Stuck in my head...
Callsign: AFTI
Version: Git
OS: Mac OS X 10.7.5

Re: Stop crashing, fgfs.exe

Postby MasteroftheSKIES » Thu Mar 06, 2014 9:34 pm

OK, there is no generic_skyscraper_01.xml in C:\Program Files\FlightGear\data\Models\Commercial. Only the .ac file. That's probably the problem I have. Where can I download the models?
I would use FSX but I use FlightGear because it's a project with goals.

Windows 7; Processor: 8 core 4 GHz; 8GB RAM; Graphics: NVIDIA GeForce GTX 770 (a pretty good graphics card if you're wondering)
MasteroftheSKIES
 
Posts: 216
Joined: Tue Oct 22, 2013 11:10 am
Version: 3.2
OS: Windows 7

Re: Stop crashing, fgfs.exe

Postby Hooray » Thu Mar 06, 2014 9:37 pm

Philosopher wrote in Thu Mar 06, 2014 8:31 pm:I think I tend to agree with Hooray here – it's nearly impossible to see what actually goes in in FG these days simply because there's too many disparate parts involved here.


That's true, on the one hand, all the information is too low-level, and on the other hand, there's so much SG/FG-specific abstraction going on, that meaningful results are hard to obtain (listeners, timers, Nasal, XML). Gdb or valgrind would not help us very much in such instances - or rather, you'd need to be really familiar with FG/SG internals to still extract useful information. Valgrind in particular is known to yield may false positives with OSG (osg::ref_ptr) but also with GC schemes like the ones used by Nasal.

And then also just info like memory usage per subsystem would be yet another helpful data point - yet we don't have it, and need experienced (and not busy!) developers who are really familiar with C++ and can get around the FG codebase to add these things, even though it would help out with several problems - particularly this.

I don't even think that adding this to most subsystems would be particularly tricky - the main culprit here is that we need to agree that this needs to be done, and subsequently adopted by others. For example, Mathias and Tim have stated pretty clearly years ago, that they consider ANY C++ code still making use of raw memory management via new/delete broken if it doesn't assign allocated memory to smart pointers (SGSharedPtr/osg::ref_ptr) directly - and that they'd refuse to review/commit such code - then again, code committed by other core developers doesn't typically go through any kind of review where such things would be flagged.
Thus, given such examples, I am not sure if we're even in a position to discuss this, let alone enforce such best practices.
SGSubsystem and SGPropertyChangeListeners were around for over a decade before they started being adopted in any significant way, and even still today, there's a ton of code not using these APIs.

On top of all this, threading adds another can of worms that cannot be easily inspected at runtime obviously ...
We cannot really expect people to follow best practices unless we give them tools to understand what exactly is going on.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Stop crashing, fgfs.exe

Postby Gijs » Thu Mar 06, 2014 9:54 pm

MasteroftheSKIES wrote in Thu Mar 06, 2014 9:34 pm:OK, there is no generic_skyscraper_01.xml in C:\Program Files\FlightGear\data\Models\Commercial. Only the .ac file. That's probably the problem I have. Where can I download the models?

You can get the entire package at http://scenemodels.flightgear.org/downl ... Models.tgz
When you use TerraSync, you'll get it automagically.
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9363
Joined: Tue Jul 03, 2007 2:55 pm
Location: Amsterdam/Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Stop crashing, fgfs.exe

Postby MasteroftheSKIES » Thu Mar 06, 2014 11:10 pm

Just successfully completed the flight that FG always crashed on, after I deleted that line. Gijs, Hooray, adrian, Philosopher, and everybody else, a personal thanks for all your help. Gijs, thanks for giving me the model folder, I just installed it, which probably means no more crashes.

So, in the end, the problem was that I did not have the models I should have had. Not sure why this problem occurred, even after a reinstall. I should configure TerraSync soon, so this problem won't happen again. Developers, just have this in your mind if/when other pilots report crashes. Again, thanks for all help, and devs, you can continue to discuss the stuff you're talking about which I don't really understand. :)

I will mark this thread as solved.
I would use FSX but I use FlightGear because it's a project with goals.

Windows 7; Processor: 8 core 4 GHz; 8GB RAM; Graphics: NVIDIA GeForce GTX 770 (a pretty good graphics card if you're wondering)
MasteroftheSKIES
 
Posts: 216
Joined: Tue Oct 22, 2013 11:10 am
Version: 3.2
OS: Windows 7

Re: Stop crashing, fgfs.exe

Postby Hooray » Thu Mar 06, 2014 11:21 pm

please let's not draw wrong conclusions here - does the crash occur again once you re-add those lines or once you remove those files ?
Normally, missing models should not be able to crash the sim - if that's still the case, this is a bug that needs to be fixed.
Maybe you could use your current setup to use a flight plan / flight recorder tape to come up with an easily reproducible test flight and then see if the error shows up after removing the models/references to it ?

I really don't see why a missing 3D model could be causing a crash, unless something weird is happening, like Nasal code running and trying to get a handle to the model which isn't available !?
We really need to get to the bottom of this to understand what's going on here, so that this can be fixed in 3.2, or you'll have to go through this over and over again...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Stop crashing, fgfs.exe

Postby MasteroftheSKIES » Thu Mar 06, 2014 11:30 pm

Later tonight I will do a flight with the new model set in an area that used to crash; an area in which I have not modified the files. I hope that will allow us to know for sure if the problem is fixed.
I would use FSX but I use FlightGear because it's a project with goals.

Windows 7; Processor: 8 core 4 GHz; 8GB RAM; Graphics: NVIDIA GeForce GTX 770 (a pretty good graphics card if you're wondering)
MasteroftheSKIES
 
Posts: 216
Joined: Tue Oct 22, 2013 11:10 am
Version: 3.2
OS: Windows 7

Re: Stop crashing, fgfs.exe

Postby MasteroftheSKIES » Fri Mar 07, 2014 1:35 am

Update: just finished the flight I was talking about. No crashes, so I guess I have solved my problem; I didn't have all the models I should have.
I would use FSX but I use FlightGear because it's a project with goals.

Windows 7; Processor: 8 core 4 GHz; 8GB RAM; Graphics: NVIDIA GeForce GTX 770 (a pretty good graphics card if you're wondering)
MasteroftheSKIES
 
Posts: 216
Joined: Tue Oct 22, 2013 11:10 am
Version: 3.2
OS: Windows 7

Re: Stop crashing, fgfs.exe

Postby chrisb » Thu Mar 13, 2014 3:53 am

Wouldn't missing models cause it to crash at a regular spot consistently?
Mine keeps crashing randomly, anytime from 2 seconds after start onwards, it could last hours

Do you maintainers get crash reports, Ive sent in lots, e.g. the last 10 minutes I ve already sent in 2
chrisb
 
Posts: 136
Joined: Mon Oct 21, 2013 8:46 pm
Location: YLIL, YMML
Callsign: cnb123
Version: git
OS: Win7 64bit

Re: Stop crashing, fgfs.exe

Postby adrian » Thu Mar 13, 2014 5:32 am

I think poor James gets them all, I wonder how he manages to sift through all the data. Then again, he did it with his very own hand :)

There's been repeated mention of referenced pointers in this thread. I'm not against using them per se, but I wouldn't consider manual memory allocation broken code. After all, this normal practice in 99% of all C/C++ code. They are certainly useful in some shady areas of simgear, but otherwise there's and added level of complexity that is not always desirable, especially when dealing with simple code.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: Stop crashing, fgfs.exe

Postby chrisb » Thu Mar 13, 2014 5:41 am

Well I've 2 more at least
The fix suggested above made no difference
chrisb
 
Posts: 136
Joined: Mon Oct 21, 2013 8:46 pm
Location: YLIL, YMML
Callsign: cnb123
Version: git
OS: Win7 64bit

PreviousNext

Return to Flying

Who is online

Users browsing this forum: No registered users and 0 guests