Board index FlightGear Development New features

Flightgear-specific benchmark

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

Flightgear-specific benchmark

Postby ericolon » Thu Feb 21, 2013 4:35 am

Does flightgear have like a default benchmarking system?
I think the devs should define a set of standard settings and perhaps a flight recording to benchmark flightgear on, to help determine a computer's suitability to run flightgear. Obviously user submitted benchmarkings are pretty different due to different software and settings, so a set of standards like the benchmarking in arma2/just cause2 etc. would be awesome.
eg. low settings benchmark would be a 10 min flight on a low-polygon airplane and simple terrain, and a high settings benchmark would be everything "maxed out" in high res, in heavily complex scenery, in like a thunderstorm with 100 AI aircraft perofrming CPU intensive maneuvers in close proximity etc. (I obviously don't know how fg works and which things are most CPU/GPU intensive)
Obviously a flightgear specific benchmark could be much more suitable for flightgear than a generic gaming benchmark and much more helpful for people figuring out what settings are best for their systems, so what do you guys think of the idea?
I am totally new to flightgear and I haven't seen any indication that fg has a benchmarking system, so I thought that would be nice. I could provide/specify a list of settings and record some flights (if said feature exists), although I doubt I have the fg experience and authority to do so, so I hope you guys can sort this out- besides, a benchmark system should be pretty easy to implement compared to like say adding more realism/ better graphics/ revamping the engine, so I think that including a set of specific benchmarking tools and settings is plausible
just my two cents as a newcomer to the forums
ericolon
 
Posts: 2
Joined: Thu Feb 21, 2013 4:18 am

Re: Flightgear-specific benchmark

Postby Thorsten » Thu Feb 21, 2013 7:16 am

I think the devs should define a set of standard settings and perhaps a flight recording to benchmark flightgear on, to help determine a computer's suitability to run flightgear


Can't a user be bothered with finding out what he wants to run? Some people say they really need 60 fps and not a frame less, others are fine with 15 fps as long as they can get good graphics, for some folks random trees and buildings are terribly important, for others AI traffic is the one thing they absolutely want to have. So I would assume that users just find a compromise between what they want to see and how large a framerate they want to have and stick with it.

From the devel side, I'm usually just interested in how fast a specific subsystem performs (say I want to know how much a change in the cloud fragment shader impacts framerate). I have to design a specific benchmark situation for the task in which the framerate is really dominated by what I want to see and not by anything else. Knowing how a particular standard flight performs on your computer is quite useless to me then.
Thorsten
 
Posts: 10844
Joined: Mon Nov 02, 2009 8:33 am

Re: Flightgear-specific benchmark

Postby Figaro » Thu Feb 21, 2013 7:23 am

I personally am looking at a laptop (and have found one) that will be able to run FlightGear with absolutely all the eyecandy switched on. Now I want it mainly for my YouTube channel, but also to improve my own FlightGear experience.
Having been in the community for a couple of years, I've come to learn what a good system to run it on is, and so I've based my decision on that. I believe there is a recommended specs thing on the main website or the wiki, but I may be wrong.

For those who are wondering, I'm looking at this bad boy. It's received pretty damn good benchmark scores, and as it is equal to or exceeds the FG recommended specs, I'm going for it. Seems I can get it for as little as AUD$1000 or up to AUD$2000 (thanks to Australian retailers charging ridiculously big mark ups on electronics).


Cheers,
Sam.
User avatar
Figaro
 
Posts: 1262
Joined: Fri Feb 25, 2011 9:23 pm
Callsign: 4L-FIG
OS: Elementary 5.0 Juno

Re: Flightgear-specific benchmark

Postby Hooray » Thu Feb 21, 2013 10:32 am

Hi & welcome,

thanks for taking the time to leave feedback here.
We actually talked about that benchmarking idea a while ago, and I even implemented a proof of concept, see: viewtopic.php?f=6&t=7806

Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult to create a simple benchmark framework.

ericolon wrote in Thu Feb 21, 2013 4:35 am:I could provide/specify a list of settings and record some flights (if said feature exists)


We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings:

Subject: Mac 2.10 RC Atmospheric light scattering bug

Hooray wrote:After all, having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion.

We could add a dialog so that people could even describe the problem - so that the XML files would become self-contained and could be easily checked by different people without having to ask tons of tedious questions...

Thinking about it, the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog), which should give us a way to reproduce settings fairly well.


New properties can be easily added by editing an XML file, see $FG_ROOT/Docs/README.flightrecorder (plain text file, can be opened using notepad or wordpad on Windows): http://gitorious.org/fg/fgdata/blobs/ma ... htrecorder

From a regression-testing perspective, I would agree that having a simple benchmark framework in place would seem like a good idea. And a number of existing features could be reused and integrated here, not just the replay/flight recorder subsystem, but obviously also Nasal scripting.

We already have a number of examples doing a simple form of feature-scaling by monitoring certain parameters and then adjusting internal behavior accordingly. The local weather system already uses such heuristics for example. And "property rules" could be easily used to come up with additional rules.

viewtopic.php?f=6&t=7441&p=75977&hilit=feature+scaling#p75977
viewtopic.php?f=6&t=7441&p=75980&hilit=feature+scaling#p75980
viewtopic.php?f=18&t=15578&p=152112&hilit=feature+scaling#p152112
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Flightgear-specific benchmark

Postby Michat » Thu Feb 21, 2013 4:05 pm

I wonder if you are able to create an homogeneous benchmark in multiplatform.

With ATI 4870x2 I was 60fps user, thats was my benchmark. gnewsense cvs jester built /win) Unfortunately I had no AA in GNU/ATI. So I decided to use XP only for FG. :x

Are you guys talking graphic benchmark? Because when I had my power PC, I used to put all resources in max quality. Except level shader <= 3 slider position. with 60fps :) >3 shaders level=crashed, no draw textures.

Furthermore I did tests with all statistics on screen, results were really hard for the machine :oops: . What I understood is that, it was a GPU task different to the GPU work load, anyway I find that internal properties on screen are faster than statistics with a lot of things on screen, internal are faster. :roll:

Months later I got at that 3 level shaders, crazy fans + Bios GPU corruption. It was a casualty that it's happened after decide to remove Skype just after Microsoft buy it, then I received a message from them telling about they gonna change something in my ATi hardware, or it was when I'd uninstalled outerra when I also received a message from ms visual component about my hard....
Or maybe it was just next to writte a very critic comment in you tube about MS Flight fake video (using video to show a non-existent future release, it's non ethic..)

@Connect I will neglect all hardware under or over platformed out of the box, they installed to me a death's clock. Be careful with windows 8 PC's check if restricted boot is present for GNu's.

Have a good bechmark :wink: .

Clear to topic guys.
User avatar
Michat
 
Posts: 958
Joined: Mon Jan 25, 2010 6:24 pm
Location: Spain
Version: 191b
OS: GNewSense

Re: Flightgear-specific benchmark

Postby Hooray » Thu Feb 21, 2013 4:30 pm

Even a very simple benchmark could be useful for feature-scaling purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience.

We already have various building blocks in FG to do most of this, it's really just a matter of combining and integrating existing features to provide such a simple benchmark.

From a troubleshooting perspective this could in fact also be useful, because we could ask users to open a certain dialog and run a certain benchmark and report the results here.

On the other hand, this is an old idea obviously and it should probably be logged as a feature request: https://code.google.com/p/flightgear-bu ... %20request

Now, regarding feature-scaling, there's a long-standing idea to eventually use FlightGear's autopilot system to tune various parameters to achieve a certain target frame rate and frame spacing, which was recently again raised on the devel list: http://www.mail-archive.com/flightgear- ... 39510.html
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am

Re: Flightgear-specific benchmark

Postby Hooray » Thu Jun 20, 2013 9:33 pm

The idea was recently re-invented on the devel list: http://www.mail-archive.com/flightgear- ... 40294.html

So, I have moved everything over to a new wiki article that's all about coming up with a FG specific benchmark system, possibly built on of Nasal scripting and the replay/flight recorder systems: http://wiki.flightgear.org/FlightGear_Benchmark
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11326
Joined: Tue Mar 25, 2008 8:40 am


Return to New features

Who is online

Users browsing this forum: No registered users and 2 guests