Board index FlightGear The FlightGear project

Helping users in the long term

Questions about the FlightGear organisation, website, wiki etc.

Helping users in the long term

Postby Johan G » Sun Mar 16, 2014 12:43 am

Hooray wrote in Sat Mar 15, 2014 5:20 am:...However, the main issue is quality of bug reports, i.e. people not being able to provide backtraces and good diagnostics usually...
chrisb wrote in Sat Mar 15, 2014 8:21 pm:Problem with that is most players are not programmers and have no idea how or what to provide

This post was originally a reply to the above, but was too off topic so I created this topic.

I think you might have hit the head of the nail there.

I think users can be hypothetically be put in three large categories: (1) Those that to a very large degree understand what the developers/maintainers need to get a grasp of the problem and preferably bring a bit more to the table; (2) those that at least understand that the need to supply the symptoms of what went wrong, what OS and version of FlightGear they were running; and finally (3) those that seem to think that fixing a software problem could not be harder than harder than to hammer in a nail that last bit... or is it that they are expecting a flawless polished product much enough that they just have went their frustration?

There is also a fourth category: (4) Those that understands and can modify parts of FlightGear, let that be the source of fgfs or the data (in particular the aircraft), and can propose and motivate a certain change to fix a problem.

I actually do not think much can be done about the third category, at least not with an immediate effect (or without brawls :roll: ). But through documentation and wiki articles, not to mention this forum one can move users from the second category to the first, and maybe even moving a few users from the first to the fourth category, but it will require curiosity from the users (many of us just could not resist modifying an aircraft, right? :wink: ).

Question is how to further entice the users curiosity and how to improve the wiki and documentation (remember it is not all about quantity). One good way to get users more curious are to release early and release often and show the progress on the forum, but then you have to shield yourself against the third category... :wink:
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5480
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: Helping users in the long term

Postby Thorsten » Sun Mar 16, 2014 6:45 am

I dunno. I deal with the somewhat less complicated problems of Nasal and GLSL shader errors which tend to either produce a report or have visual effects. A fairly typical example of an interaction with a user is here.

My primary concern in debugging is to be able to reproduce the issue. If I can, it's usually a done deal. In the majority of cases, I'm dealing with (3) and am unable to get a straight answer what steps I need to do to see the problem even for asking. As long as users can answer questions and follow suggested simple diagnostics, I'm usually doing fine in debugging.

My secondary concern is the systematics if I can't reproduce the issue myself, i.e. I need to see under what conditions it occurs and under what not.

For the issues I'm dealing with, I'm not sure how helpful general debug checklists or wiki pages are - I usually want very specific information, just a subset of what's usually asked. I need answers to questions like 'Describe the sequence of buttons you press' or 'What happens if you reduce model shader quality to minimum?' or 'Is there a GLSL compilation error in the console?' and I often need a screenshot. General things like the OS are not helpful for that particular kind of problems. A fair share of bug reports remains unanswered because I can't get answers to such questions. By its very nature, debugging tends to be adaptive and unpredictable and needs to be made up. But in order to do it, you need familiarity with the system. I can debug GLSL because I know what it does, I can't debug YaSim because I only have a foggy of its internals.

So I guess we're operating just fine with (1) and (2) users because they do respond in a meaningful way to questions which is really all it takes, and really for my ease there's not so much difference between the two categories since ultimately I need to hunt the bug anyway myself, but (3) are a problem.
Thorsten
 
Posts: 10820
Joined: Mon Nov 02, 2009 8:33 am

Re: Helping users in the long term

Postby Hooray » Sun Mar 16, 2014 7:14 am

right, troubleshooting is a matter of presenting the steps required to reproduce/verify a certain issue, and that's usually a long-winded and tedious process unfortunately.
And it also doesn't scale particularly well. Which is why I tend to come up with wiki articles and simply post links that detail all steps and questions that people need to follow/answer.

Otherwise, I would not get anything done. Then again, I fully realize that most users (especially new ones, with primarily MSFS/FSX and XP experience) are unlikely to even respond to such enquiries. Admittedly, I also don't find this particularly intuitive or user friendly. But given our "manpower" there's only so much we can do about this. I think the recent addition of integrated crash reporting support is a good step - but as mentioned elsewhere, it doesn't scale too well as long as only a single developer has access to these reports, and as long as only a single guy actually knows how to process those reports. At the moment, that's our main bottleneck when it comes to troubleshooting crashes.

Then again, not all issues are necessarily segfaults/crashes - but for the time being, I'd consider those to be of primary importance, so personally I'd focus on those, rather than some obscure usability issues.

Now, when it comes to troubleshooting end-user crash reports, there's only so much that we can do, short of asking people to provide GDB back traces, and that alone isn't very appropriate for the majority of non-segfaults.

What I've been contemplating for a while is making things better reproducible by using FlightGear's existing support for replicating runtime settings via serialization (user-archive attribute). Basically, we could use the existing flight recorder/replay system and extend it to also plot additional properties, such as for example those that are relevant for rendering/shader settings. The main prerequisite for this to work would be that the corresponding properties are runtime-configurable, i.e. take effect without requiring a restart. For most rendering related settings this already seems to be the case.

Ultimately, that would allow us to ask people to provide a "replay tape" with their rendering settings, possibly even including a flight or other runtime settings, so that developers/contributors could more easily download such an XML file, and apply it to their own installation at runtime, to see if they can reproduce a problem. Personally, I would be much more likely to do something like this, instead of having to spend 2 weeks asking dozens of questions to "approach" the nature of the problem.

The replay/flight recorder subsystem is flexible to support most of this, even without modifying its C++ code - it's just some usability "glue" that needs to be added, i.e. GUI dialogs.

The good thing about this approach is that it would be very scalable as long as FlightGear primarily deals with settings in the form of properties - we could even log startup stuff to such a file, such as OpenGL/GLSL version info, operating system, amount of RAM, number of CPUs etc

The recent reset/re-init work makes this increasingly feasible, so we may want to reconsider this once all the building blocks are in place.

It would be a good project for someone interested mainly in XML and GUI dialogs, but most of us are already juggling too many other projects - so I'd rather not provide another "prototype" currently:

Subject: Mac 2.10 RC Atmospheric light scattering bug
Hooray wrote:we should add a menu item to dump the current position and all rendering/environment settings to an XML file, so that we can more easily reproduce such things, just by loading a config from a file.



Subject: Mac 2.10 RC Atmospheric light scattering bug
Hooray wrote:
someguy wrote in Sun Feb 03, 2013 5:39 am:That's a super idea! It wouldn't surprise me if some of these glitches are peculiar to specific hardware configurations, either, so perhaps that might be part of the report as well.


Yes, even just knowing that certain issues only occur with some GPUs would be VERY good to know. But obviously we would need a sane way to easily reproduce a certain configuration, including all startup settings, but also the runtime rendering settings.

I'll paste XML into forum posts all day if it helps the devs fix bugs.

We could obviously use a pastebin, but probably using the issue tracker would be a better idea then.
After all, having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion.

We could add a dialog so that people could even describe the problem - so that the XML files would become self-contained and could be easily checked by different people without having to ask tons of tedious questions...

Thinking about it, the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog), which should give us a way to reproduce settings fairly well.



Subject: Flightgear-specific benchmark
Hooray wrote:Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult ....

ericolon wrote in Thu Feb 21, 2013 4:35 am:I could provide/specify a list of settings and record some flights (if said feature exists)


We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings:

New properties can be easily added by editing an XML file, see $FG_ROOT/Docs/README.flightrecorder (plain text file, can be opened using notepad or wordpad on Windows): http://gitorious.org/fg/fgdata/blobs/ma ... htrecorder
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Helping users in the long term

Postby chrisb » Mon Mar 17, 2014 1:12 am

I dont think its that simple, I fit somewhere into 2 AND 4
4) in that I can make basic changes to obvious problems in things like aircraft xml
NOT 1) I dont know what the devs actually need to debug and fix a problem
2) I have an understanding of the process involved and can provide basic info

But im guessing 3) makes up the majority of the FG player base
chrisb
 
Posts: 136
Joined: Mon Oct 21, 2013 8:46 pm
Location: YLIL, YMML
Callsign: cnb123
Version: git
OS: Win7 64bit

Re: Helping users in the long term

Postby Johan G » Mon Mar 17, 2014 1:33 am

I see that my (original) intent was a bit obscure here. :oops: I should have expanded on that last paragraph, and perhaps have chosen a better title.

What I really was after and should have expressed further was finding what ways we could spark the curiosity and interest of our fellow users so that they progressively may be more able to help themselves, and at some point even trigger them to contribute to this community and of course to FlightGear.

Note that there are more ways to do this than improving documentation, or getting better bug reports, though they will both help. ;)


All that being said, the posts above are still interesting.


Thorsten wrote in Sun Mar 16, 2014 6:45 am:My primary concern in debugging is to be able to reproduce the issue. If I can, it's usually a done deal. In the majority of cases, I'm dealing with (3) and am unable to get a straight answer what steps I need to do to see the problem even for asking. As long as users can answer questions and follow suggested simple diagnostics, I'm usually doing fine in debugging.

My secondary concern is the systematics if I can't reproduce the issue myself, i.e. I need to see under what conditions it occurs and under what not.

For the issues I'm dealing with, I'm not sure how helpful general debug checklists or wiki pages are - I usually want very specific information, just a subset of what's usually asked.
...
By its very nature, debugging tends to be adaptive and unpredictable and needs to be made up. But in order to do it, you need familiarity with the system...

Reproducible issues are indeed easier to troubleshoot. No one likes mysterious intermittent problems. And yes, often the path to a solution can have "interesting" twists, but at the other hand those twists sometimes reveal bigger issues...

Thorsten wrote in Sun Mar 16, 2014 6:45 am:In the majority of cases, I'm dealing with (3) and am unable to get a straight answer what steps I need to do to see the problem even for asking.

I am sometimes quite surprised how even native English speakers can at times be asking for help, but still seems to not read trough questions by the one trying to help, ignore advice that is repeated and a few times even end up flaming/scorching the helping hand. :roll:


Hooray wrote in Sun Mar 16, 2014 7:14 am:right, troubleshooting is a matter of presenting the steps required to reproduce/verify a certain issue, and that's usually a long-winded and tedious process unfortunately.
And it also doesn't scale particularly well. Which is why I tend to come up with wiki articles and simply post links that detail all steps and questions that people need to follow/answer.

Otherwise, I would not get anything done. Then again, I fully realize that most users (especially new ones, with primarily MSFS/FSX and XP experience) are unlikely to even respond to such enquiries. Admittedly, I also don't find this particularly intuitive or user friendly...

Debugging and troubleshooting indeed does not scale. Helping people in particular does not.

chrisb wrote in Mon Mar 17, 2014 1:12 am:I dont think its that simple, I fit somewhere into 2 AND 4
...
But im guessing 3) makes up the majority of the FG player base

I think most people are in different user categories depending on the context. I consider myself being (1) and sometimes (2) or (4) when it comes to aircraft and joystick configurations, but when it comes to anything that touches more Nasal than short statements I am in (2) and for the FlightGear source I am in (2), and I guess sometimes (3).
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5480
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit


Return to The FlightGear project

Who is online

Users browsing this forum: No registered users and 3 guests