Hooray wrote in Wed Sep 23, 2015 3:19 pm:
that's really cool, if we could add the corresponding hooks to do create these kind of statistics/graphs more easily, we could grow a really useful library of regression tests, with frame rate/frame spacing really just being the start of a thorough benchmark suite.
Maybe through the use of Canvas to plot/track properties and then using a native image exporter lib(we already export PNG), that comes with FG?
Hooray wrote in Wed Sep 23, 2015 3:19 pm:
Even though I guess, that it would make sense to use the minimal startup profile (no scenery) as the baseline for all benchmarks, so that the impact of different subsystems (think AI/MP, FDM etc) can be measured against a really "lean" startup profile.
if we could initialize subsystems in order, that would be great for evaluating the impact of each, but for it to be testable, we would need a controlled environment, in the case of AI a "scenario", in the case of FDM a set of instructions to fly along a certain path within a set time-frame, that would gather the footprint for that FDM driving along a similar path, perhaps on CPU usage rather than GPU?
Hooray wrote in Wed Sep 23, 2015 3:19 pm:
Ideally, we would grow a handful of "tests" and then try those with different variables, i.e. aircraft/location.
Given the SIGAR patches we have, we can also add actual RAM/SWAP utilization into the mix, too.
...
In its current form, this is already pretty cool and informative, especially if we could create a table for different airports/aircraft to show the impact of those settings, and how they scale with things like visibility/shader quality.
It would be easier for the user, to select one of pre-recorded files to be parsed with generic "playback" protocol from a GUI, the set of circumstances to be tested is a given, but the use of the system with which to do so is a decision.
i.e using an existing feature like the FGTape, or introduce a new one?
I would like to know the advantages for FGTape, knowing that the generic protocol potentially (might have?) skipped the Nasal-based FDM of the UFO, but it still had the ability to place objects, which might have introduced some Nasal/C++ code into the test, but i don't know how it's implemented, therefore i suggest maybe a new -set.xml to remove this functionality, the UUIC flight model could be sufficient, but it wouldn't allow for different FDM's to be tested for the subsystems idea.
Hooray wrote in Wed Sep 23, 2015 3:19 pm:
To be really useful, such a benchmark should ideally wait for 30-60 seconds after booting FG, to ensure that all subsystems are up and running.
FG was kind enough to only log properties as the scene started, but there still existed the first 3 seconds in the log where the FPS was in the single digits(apparently initializing something), so per standard, all of my input were gathered from the 3-70 seconds range.
I just re-did the scenery_2 test, since that was the last test i left off from yesterday, and it's consistent with yesterday, recording an AVG of 46.27, which is pretty close!
Hooray wrote in Wed Sep 23, 2015 3:19 pm:
My suggestion would be to set up a table of startup settings on the wiki (.fgfsrc), so that people can more easily reproduce "scenery 0...x". Alternatively, we could add a directory to $FG_ROOT, e.g. $FG_ROOT/Profiles and patch FG to load .fgfsrc files from there. That would help ensure that people are actually running the same startup settings.
all my tests were done with the same fashion,
- Code: Select all
--prop:/sim/menubar/visibility=false --prop:/sim/ai-traffic/enabled=false --visibility=20000 --disable-clouds --disable-clouds3d --prop:/sim/rendering/draw-mask/clouds=0 --generic=file,in,30,flight.out,playback,repeat --fdm=external
the first part before the protocol, is very important to maintain consistency, since these would be the ones to introduce random output, clouds for instance can hide the airport, while i thought the draw mask is drawn randomly, so i tried to maintain the same vertex count each time, with no PUI-elements involved.
in this case i didn't set any shader/graphics level from the command-line, but i do it manually and then restart FG to do the tests, do you know a way that i can do that from the command line or .fgfsrc?
Hooray wrote in Wed Sep 23, 2015 3:19 pm:
But I would hope that we can add more metrics, and expose those to the property tree - e.g. stuff like RAM utilization (for which we have working code), but also other stuff like Nasal GC stats, but also listener/callback overhead.
Which brings me to the question, because sometimes just after my computer starts there exists a delay that skews my testing, so does SIGAR have metric for read/write speed, etc?
Edit: What i mean by no_scenery test, is not that there is no terrain, but i was just evaluating the footprint/fps cost of my scenery models,
scenery_0(shader level 0, so model using default-eff) = - 2.55 fps cost
scenery_1(shader level 1, so model using model-combined-deferred.eff) = - 4.8 fps cost, - 7.34 total