Aircraft Rating System

[Hooray]

you can always come up with rating systems for different types of aircraft, we'd just need people interested in exploring this

[Jabberwocky]

The point not entirely clear is, is this a comptetition or just an information structure. As information structure, it works well for me. I look at Systems because I like flying from inside the cockpit and there I am, happily picking my planes. And honestly, if I would, for example download a warbird with good systems by accident, and I am not interested in warbirds in the first place, all comparing to other warbirds won't help.
So, what is this discussion about? How many abstract points can a plane get in a sum that, because it's the sum, has lost the information content?

[Michat]

Dear Hvengel the point is the user vote for the experience factor, not for the FDM and technical issues as I mentioned ( apart of the FDM and technical issues ).

About ranking orderer by flight time, that's not a race of who flies more, among the airliners category there will be a rank (users choice-like factor), same thing for each aircraft category, I mean the roster will show us by itself what are the most used aircrafts by category in a natural way. The number of offline players I think doesn't matter if we consider that is in self a representation of the online mass, users at last.

Dear Thorsten. True, I'm volunteer. I can show some sketches with general ideas, also I can help with graphics, just a tiny part

[hvengel]

I really don't think that we need different ratings criteria for different aircraft types for our current rating system. The current system automatically compensates for type and complexity differences. The rating reflects how complete and accurate the aircraft is regardless of type or complexity and takes most of the subjectivity out of the equation. What it does not do is reflect how much work went into getting things to that level and I don't think it should. Something very simple like a J3 can get to a full 5s rating level with a lot less work (big undertaking) than getting a jetliner to the same level (massive undertaking). But I think that is a given and that everyone understands that getting a very simple aircraft fully modeled is a lot easier than for a complex aircraft.

The person who started this was suggesting that total flight time be used to create some sort of rating/ranking. And I just pointed out how different aircraft types had very different operational profiles. As examples airliners are used for very long duration flights and aerobatic aircraft usually have very short flights. Because of this total flight hours is a very poor indicator of the actual "fun factor" except perhaps if this was used only for ranking with in specific aircraft types. If we have 2 airliners and airliner A has 100,000 total hours this month and airliner B has 20,000 total hours then you can probably conclude with some degree of certainty that airline A is more fun or more rewarding to fly. But if you compare airliner A's 100,000 hours to aerobatic aircraft B's 20,000 you can't conclude anything about the fun/reward factor because the airliners average flight might be 4 (IMO mostly boring) hours and the aerobatics aircraft average flight might be 15 very fun packed minutes (IE. a 16 to 1 duration difference) with the aerobatic aircraft doing over 3 times as many flights during that period. So which is more fun/rewarding? The total flight time number tells us nothing about the comparative fun/reward levels of these two aircraft.

The other thing I was getting at is that user ratings tend to be highly subjective and highly variable. Someone who is into WWII warbirds will likely rate a nicely done WWII warbird very differently than someone who is into airliners. They are coming at it from very different view points and with very different expectations and are likely to assign very different "fun factor" rating to the same aircraft. For example a WWII aircraft person would likely have a lot deeper historical background in these aircraft and have a better understanding of the systems and technology of that era. As a result he is likely to be more objective when rating the aircraft where as the airliner person will not understand the WWII warbird at all (IE. "Why do I have to flip 3 switches before I can fire the guns?") and this lack of context will likely garner a much lower rating even if the aircraft is very well done. Basically the more outside of someone comfort zone the more subjective the rating is likely to be.

User experience with flight sims also enters into this. New FG users tend to prefer the less realistic more game like feel of aircraft that are less developed where as more experienced users tend to prefer more accurately simulated aircraft. They want to be challenged by the aircraft and to learn new stuff about it and flying. Where as new users tend to find this frustrating ("I can't can't even get the dam thing started...."). Because of this I believe that we would see very different user ratings on the same aircraft from new and experienced users.

[Michat]

Dear Hvengel

The fun factor or user experience is a vote from user that nobody knows the reason he likes, but he votes for that.


Total flight time will be that, total flight time. So you we can rank the most used aircrafts (1rst-2nd-3rd) by categories. Airliners, Classics- propeller, etc etc. So we compared flight time/categories, not user experience or FDM accuracy. Those numbers will reflect the total virtual flight time accumulated with some statistics.

So two points, give power to the user to say I like that one aircraft, and show some statistics about what are the aircrafts that most users are flying by category.

" User experience with flight sims also enters into this. New FG users tend to prefer the less realistic more game like feel of aircraft that are less developed where as more experienced users tend to prefer more accurately simulated aircraft. They want to be challenged by the aircraft and to learn new stuff about it and flying. Where as new users tend to find this frustrating ("I can't can't even get the dam thing started...."). Because of this I believe that we would see very different user ratings on the same aircraft from new and experienced users."

Agree with you but at the very first moment the user, will try to use the model he likes, the more attractive to him, even the bigger airliner if the new user is a young pilot. At that point is when you mention "Where as new users tend to find this frustrating" and they opt to try others. That's the natural way to discover more about FG .

Amicalement

[Jabberwocky]

Huh? I'm an old newbie as pilot, what does that count for. And if y<ou have no joystick, go big and heavy, it makes the life soooo much easier.
About the fun factor, that's the point. Nobody knows, but statistically, it levels out. I know for example, IH-COL doesn't care too much about the cockpit interior but care for the "historical" aspect of planes. So, he would for example give a 4 to the TriStar and only a 2 to my Shaggy Cow, aka 747-8i. On the other hand, I care about cockpits and not so much about how old a plane is. So, I would give a 2 at best for the TriStar and a 5 for the 747. The Tristar ands up with 6, the 747 with 7 points, the world is all okay.
The real interesting thing comes up, if some planes run off the field. Because if some are leading the pack from far ahead, it means, those are the planes, developers can learn from.

[HJ1AN]

Excellent suggestion. Aircraft by fun factor would be a great indicator and also not that hard to add into the system, I assume. The Ogel would most probably fit the fun factor

[hvengel]

...
The real interesting thing comes up, if some planes run off the field. Because if some are leading the pack from far ahead, it means, those are the planes, developers can learn from.

I don't know if other aircraft devs are like me but all I really care about is making my aircraft as realistic as possible. If that pleases those that try it I'm glade that someone else is enjoying my work. But if they don't like I don't really care although I am more than happy to get constructive feedback and would like to know why to understand what issues I might be able to correct. That some other aircraft is wildly popular with users is something that I don't care about at all but if you see something on another aircraft that could make mine more realistic or easier to interact with please let me know but don't expect me to go digging through some other aircraft's files looking for some unknown tidbit.

I know lots of folks fly my plane and much of the feedback I see here is either about issues that were fixed a long time ago (IE. they are using a old version - simple to fix tell them to upgrade) or the issue is pilot error or equipment issues like bad joystick configs. I would guess that perhaps 10% of the feedback I get here is about issues with the current release that are real (IE. not a pilot error issue or an equipment issue) and the vast majority of these are issues that I already know about and are on my todo list. I always look into any new issues although it can sometimes be difficult to locate and fix the cause.

The best thing that users can do for aircraft devs when there is an issue is to make good reports about what is not working correctly. If you post something that says "it doesn't fly right" and little else this is not very useful. But if you post "The so and so control is supposed to work like blah blah.... according to the pilots manual at page 111 but instead it is doing ..." that give me something I can work with and not only do I not mind that type of report but I make every effort to address the issue. If you really want to see improved aircraft users, who are unable to do actual work on these aircraft, should be focused on providing detailed very specific feedback so that aircraft devs have reports that they can act on. Even better would be to volunteer to help out.

The fact is that there are only a handful of really well done aircraft in FG (perhaps two or three dozen total in all hangers and perhaps a dozen in fgdata git) and even the best of those have many areas that could be improved even though the aircraft dev probably has thousands if not tens of thousands of hours of work into it already. I actively update the wiki page for my aircraft and there is lots of info there for work that is currently underway. So I try to provide feedback to users about what I am doing and what is going on with my aircraft. So far I have not seen a single indication that any user has actually looked at the wiki page which kind of makes me wonder why I am doing these updates to the wiki page.

Most of the recent feedback on the forum for my aircraft for actual issues (not pilot error or equipment issues) has been related to the external 3D model which is a long standing issue on my todo list. The current rating for the external model is a 3 and if it were a 4 or higher the overall aircraft rating would be "advanced production" as everything else is s 4 or 5. I am now at the point were I am starting to prep the new 3D external model for moving it into FG for integration into the FDM and animation after about 500 hours of effort (not a typo - there is not an extra zero). This external model will be a solid 5 when it is finished.

For open source projects, people working on the project, including aircraft devs here, work on things because it pleases them not because users are falling all over themselves to give high ratings in a voting system on-line to their work. This may sound arrogant but many of those users have no idea if they are sitting in the cockpit of a really exceptional aircraft or a mediocre one and the ratings generated by a "voting" type system by these users IMO is of little value. I would not pay any attention to such a system simply because there is way too much noise and virtually no signal to get anything really useful (IE. something actionable). Like I wrote above what aircraft devs need is detailed, very specific feedback not generalized information about how well their aircraft stacks up against other aircraft from a user point of view.

[Jabberwocky]

You are in the P51 model right? I am too clumsy for tail draggers without a joystick. So, how would that rate? But seriously, the point is, that the different tastes will level out each other on a statistical level. That means, all models going off the pack are obviously those looking good in- and outside and fly realistically because they get the three major user groups to vote high

[HJ1AN]

it's a fictional "aircraft" - besides, what else would you like to see improved
And obviously the rating system was not conceived with fictional aircraft in mind.Obviously, the bluebird is another example, but much more developed

It doesn't bother me that the Ogel is rated so highly, because, I can see right away it's not based on something realistic. But, on FDM accuracy, I would rate it a 5/5 if it flew anything like my son's Duplo planes (ie not at all). If there's a fun rating, it would be at least a 4, simply for the model itself.

But you mentioned the Bluebird, well, I took one look (didn't download it yet), and thought, "futuristic-BladeRunner-air-car-transport-thing". Now, if the FDM comes from the UFO's FDM I would probably rate it 0-1, but since it's fictional we don't know how the thing works, but at least if it *feels* realistic, ie affected by turbulance, wind, bounces on landing, scrapes if you bank too much on landing, creates shockwave going mach1, etc.. like what we expect something like that to behave, even though if put into an actual concept it won't be exactly like that, I would give it at least a 4 because of all the other things. That's all my personal opinion of course.

Another minor thing I would point out is that the Cessna 182 Fixed Gear is rated higher than the Retractable Gear version. Having spent some time flying the RG, and then took one look at the non-RG version (plus a few posts on these forums about the FDM of the non-RG), I feel the RG version should be rated higher. Other things being equal (because essentially everything else is the same), the FDM in the RG is better. Even though I suspect the FDM is lifted and modified from the 172P.

[hvengel]

You are in the P51 model right? I am too clumsy for tail draggers without a joystick. So, how would that rate? But seriously, the point is, that the different tastes will level out each other on a statistical level. That means, all models going off the pack are obviously those looking good in- and outside and fly realistically because they get the three major user groups to vote high

Yes the JSBSim P-51D. I just added some new screen shots of renders from within Blender to the wiki this week end showing the current model. See http://wiki.flightgear.org/North_American_P-51_Mustang to have a look. It should be fairly apparent how close I am to moving this into FG.

Tail draggers, particularly ones with lots of power, are not at all fun for those using a mouse unless they are very skilled at it. There are users who fly the P-51D with a mouse but I think most mouse only users will be quickly overwhelmed by it. I can't do it and wouldn't try since I like things to be more realistic. So I have a full controller setup and that makes it relatively easy to fly for it's type.

I work on analytics software (statistical analysis software) for a living and from my experience statistics are not that simple. Too much noise and the stats become meaningless and what you are describing will be extremely noisy.

I guess my concern here is that some are saying that this type of thing will somehow help the aircraft devs and I think this is a fallacious claim. The aircraft devs, at least on the better aircraft, need very detailed aircraft specific feedback in order to make improvements. Very often they will already be aware of the issue being reported and have it in their todo list. They will generally be interested in anything new that is being reported if it is a real issue (IE. not pilot error or some other non-aircraft issue). Keep in mind that for the better aircraft the dev is "sweating the details" at a level that is probably hard for most users to understand and the details, even very small ones, are all that matters at that level.

If users are not providing very detailed issue reports they are generally not useful. So if you report an issue (this applies for anything in FG not just aircraft issues) be precise and very detailed. It is almost impossible to provide too much information. Also if the dev gets back to you to ask for more information please give him everything he asks for and expect him to ask for more. If you do that the dev will generally try his/her best to correct the issue and if he does its a win win situation. Short of actually volunteering to work on the aircraft or some other aspect of FG this is probably the most important thing that users can do to help improve things.

[erik]

Maybe it would be a good idea to provide both realism and difficulty ratings. In the case of the P-51D this would result in maximum ratings for both and uses would know what to expect.

Erik

[Johan G]

Going with what hvengel said above I would also say that subjective ratings are always subjective, and unless there is some specific criteria ratings will be subjective. The presence of objective and specific criteria is what I like with the current rating system.

Just think of advertising and consider that marketing often do more for how a consumer feel and think about some products than the products themselves, though the consumers in general are totally unaware of it (cosmetics, food and clothing comes to my mind).

While I am at that subject I can share an anecdote about one of the earlier vacuum robots (as on floor cleaning) or lawn moving robots (can't remember which, unfortunately). When the product had started to sell, the engineers was very happy with it and some of the algorithms used to optimise battery time and efficiency. Unfortunately marketing got back and reported that many users got the impression that it's work pattern was inefficient and tasked the engineers with finding a pattern that was pleasing the eyes of the customers rather than being efficient.

The same have happened with some of the more well done aircraft, when some users found them "unrealistic" to fly. So far the developers are free to ignore such requests and for example stick with an FDM based on good solid data, and I hope it stays that way.

[HJ1AN]

I was able to fly the P51 fine, although it is a handful. And I never got the landing to align with the runway for long.

The Sopwith, on the other hand , I was never able to take off successfully with a keyboard OR mouse.. Only succeeded in starting large forest fires near the airport off the runway.

I probably need a joystick AND pedals for that to succeed. Nothing a Ctrl+U can't fix to get into the air though.

Anyways back on topic, prob not an issue fixing unless someone can do it

[radi]

Yes the JSBSim P-51D. I just added some new screen shots of renders from within Blender to the wiki this week end showing the current model. See http://wiki.flightgear.org/North_American_P-51_Mustang to have a look. It should be fairly apparent how close I am to moving this into FG.

OT: I wish FG had ten times less airplanes, but of quality comparable to your work! Would also make a rating system redundant.