adrian wrote in Thu Mar 06, 2014 8:19 pm:Hooray wrote in Thu Mar 06, 2014 7:37 pm:But ultimately, we're lacking the tools to determine what's really happening behind the scenes.
Eh, it's not actually that dramatic, the tools exist, it may just take some time to run a callgrind or a gdb trace, and it requires some users experienced and willing to do this. Unfortunately, unlike a couple of years ago, there's been a constant stream of Windows users who can't or don't have the tools to do so.
Actually, I wasn't at all referring to "external" tools - because those are obviously available and can be run by people who already know how to build from source, and how to make working c++ changes.
You mention ThorstenB, it's true he was triaging bugs and actually regularly ran things through valgrind apparently and helped track down threading issues, too. But he also provided another extremely integrated tool: the built-in system monitor for instrumenting SGSubsystems.
That's basically exactly what's now needed for ALL subsystems (including the pseudo-subsystem stuff in simgear/scene), but we would ideally also be able to track memory use, too. And right now, we have now good way to tell what's going on threading-wise, but our use of threads is contributing in major ways to segfaults that we've been seeing recently.
Obviously, this is not about existing "tools" not being available - it's about the lack of integrated tools and measures to track resource usage per subsystem. Tools like valgrind, gdb,gDebugger etc need to be first of all understood - and many people around here are obviously not able to use these tools, not even the integrated google-perftools profiler, so we need to make things much more accessible.
And then again, valgrind/gdb/cachegrind/perftools may be too low-level to be meaningful in many cases, i.e. just look at what's going on in the scenery/aircraft department, here just identifying "hot spots" in terms of code/functions would be a red herring, because the real culprit is typically not at all in the runtime code, but the underlying scenegraph data, such as highly complex textures, 3D models or terrain/scenery.
This can now be easily verified by using Zakalawe's new draw-masks to switch off aircraft/scenery/clouds or AI model rendering.
Having acess to gdb/valgrind etc is one thing, being able to actually run and interpret the results is a completely different thing. Thus, it is more promising to optionally expose certain info at runtime, so that users can access it to see how their aircraft/scenery/feature/configuration behaves in comparison to some other aircraft/scenery/feature/configuration.
Also, when it comes to using tools like gdb -and especially valgrind- the main challenge is that FlightGear is primarily a GUI application, and it doesn't lend itself it being debugged/profiled or leak-tested in an isolated fashion at all. For example, for valgrind runs to be useful and straightforward, we would need to be able to minimize what's done inside fg_init.cxx so that we can disable things not needed, but also even run FlightGear in a "headless" mode without a visible GUI window, so that such tests could be run directly on the build server as part of a regression test. Otherwise, even just running fgfs in valgrind for 10 minutes may turn out to take hours due to the nature of valgrind, i.e. it being an emulator.
valgrind et all are great if you know how to use them, but for end users, these tools are too fine-grained, we would ideally support some per-subsystem or per-feature granularity for end users.