Board index FlightGear Development Scenery

Optimizing Scenery

Questions and discussion about enhancing and populating the FlightGear world.

Optimizing Scenery

Postby William » Thu Mar 25, 2021 6:59 pm

Dear developers,

Super job, enjoy FG a lot.

While investigating some stutter I ran performance tests on my PC and observed FG related file read/writes. Tool Nirsoft: FileActivityWatch.
Noticed unnecessary file transactions.

Situation:
Flying above sea, 50 miles from shore (south of Reykjavik) FG is writing many files in TerraSync/Pylons/*.*
While flying over sea I dont expect FG to load files related to land.

Q:
Is this expected behaviour, or does it need a closer look ?

Cheers, William

FG 3.7 RC
Config: i7-4790, GTX 780, 32 Gb
William
 
Posts: 23
Joined: Thu Mar 11, 2021 1:32 pm
OS: W7

Re: Optimizing Scenery

Postby wkitty42 » Fri Mar 26, 2021 2:38 pm

it should be expected because there's a certain amount of surrounding tiles that need to be loaded when you are in range of them... think of it like a ticktacktoe board and you are always in the center square/tile... depending on your (LOD?) settings, it may be that you will also load the 5x5 tiles that surround the 3x3 and maybe even the 7x7 surrounding the 5x5 around the 3x3... i think you can see how this is going...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9146
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: Optimizing Scenery

Postby William » Fri Mar 26, 2021 3:07 pm

Thanks Wkitty42

That makes perfect sense.
Ill see how it depends on LOD settings and direction (to or from) an area

Cheers, William
William
 
Posts: 23
Joined: Thu Mar 11, 2021 1:32 pm
OS: W7

Re: Optimizing Scenery

Postby Hooray » Fri Mar 26, 2021 5:11 pm

IIRC, there are also a handful properties that expose the current tile index/bucket number, and I believe it was Adrian (as part of his radio propagation work) who actually used that info to create an actual in-sim GUI to visualize different LOD schemes, and how certain settings affect memory occupancy (a corresponding UI could also be provided by using Nasal and the Canvas):

https://sourceforge.net/p/flightgear/ma ... /30235494/
adrian wrote:I am presenting an experimental (WIP) method to reduce memory consumption by
scenery with 30%, while increasing the visibility distance 4 times.
This method relies on some kind of LOD system, without mesh simplification.
People smarter than me can come up with a safe algorithm to do that, but I'm
afraid it requires changes to Terragear. Right now mesh simplification was not
a priority.

The need arised from my frequent flights in SRTM + Corine high-res terrain,
where running out of memory was a frequent event (quite terminal on a 32bit
machine unfortunately).
So I took the liberty today and modified the scenery manager a little. It now
works by having an inner circle of high-res tiles, surrounded by an outer
circle of low-res tiles. You will be able to spot that on the map in the
second picture.
Memory savings range from 30% to 50%. I have not yet been able to figure out
why, when leaving a detailed mesh zone, memory consumption does not drop and
remains the same. This behaviour was present before my patch. Re-entering a
high definition zone should not increase memory any further.

I have performed a test with the UFO, setting visibility to 80 miles, LOD
distance bare to 120000 meters, and climbing to 30000 ft aka 10000 meters.
FPS observed was 21, no doubt because of the complexity of the mesh.
Maximum memory consumption after waiting for the loading threads to finish was
2800 MB, out of which 680 were other system apps. Terrain was entirely SRTM +
Corine.

The tile cache behaviour was modified too, it will no longer have a "heavy
tail" history, but stay mostly rectangular around the aircraft. Once I find
out how to deal with the tower view, I plan to disable any tower view as long
as the aircraft is not within a reasonable distance of the tower (say 3 km).
This should prevent terrain loss problems.
Could use some pointers on where to find the tower view settings in the code.

Image
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Optimizing Scenery

Postby William » Fri Mar 26, 2021 5:27 pm

You guys never cease to amaze me :))

Impressive.
William
 
Posts: 23
Joined: Thu Mar 11, 2021 1:32 pm
OS: W7

Re: Optimizing Scenery

Postby Hooray » Fri Mar 26, 2021 5:43 pm

I don't think any of this made it back into the official/mainstream repository though.
So, you will need to look up the original repository to get the corresponding patches and maybe rework/update some of those.
However, I think Stuart made a number of improvements to the way memory is used by FlightGear, I think it all started back when he worked on the random buildings code several years ago.

Note that, if you are able to patch/rebuild FlightGear, it is relatively easy to access/expose such internals - for instance, at the same time, we also patched FlightGear to track CPU, RAM/VRAM, and SWAP utilization:

https://wiki.flightgear.org/Resource_Tr ... FlightGear
Image

Image

More recently, Julian has been working on extending the built-in "replay" system (flight recorder) so that it can be used for regression testing purposes, i.e. to benchmark/stress-test FlightGear with different startup/runtime settings while replaying a certain flight, this can be used to track CPU/GPU/MEMORY utilization - e.g. in conjunction with the built-in performance monitor and/or the OSG stats.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Optimizing Scenery

Postby V12 » Fri Mar 26, 2021 5:56 pm

I used bare LOD 270 km, visibility limit 250 km, flight from EGLL to LIRF at FL530 consumed more than 16 GB RAM, in case of heavy clouds over Alps, memory usage was 20 GB. FG is very memory hungry for realistic visibility.

For example, P3D v4.5 and resource heavy FSL A320 with visibility forced to 320 km does not reach 10 GB on same flight at FL390.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: Optimizing Scenery

Postby Hooray » Fri Mar 26, 2021 6:23 pm

That's one of those things where it would definitely help to expose additional internals in the form of properties that people can track using the property browser and/or some simple scripting, i.e. RAM/VRAM occupancy for different startup/runtime settings and LOD ranges specifically, with different draw masks set.

OSG does have some support for RAM/VRAM tracking since that's a recurring topic on osg-users.

Either way, an improved LOD scheme is obviously in the works.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Optimizing Scenery

Postby Thorsten » Fri Mar 26, 2021 7:24 pm

For example, P3D v4.5 and resource heavy FSL A320 with visibility forced to 320 km does not reach 10 GB on same flight at FL390.


Well, apparently you're happy using P3D - so why don't you enjoy it and stop complaining how FG is so bad? Because - face it - FG is not a commercial product, it can't be optimized like a commercial product, it has no customer service like a commercial product - so you can't treat it as if it were one.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Optimizing Scenery

Postby Hooray » Fri Mar 26, 2021 8:32 pm

To be honest, FlightGear has been lacking in many areas compared to commercial/proprietary products - however, given enough time, FlightGear also has been catching up with some of them (sooner or later). Often, it literally takes a while and features/components need to be reconsidered or even efforts started from scratch. Anyway, there are some examples where FlightGear has managed to acknowledge its strength and where it has come up with features that are empowering contributors, such as for example the property tree, XML, XML based FDMs, open 3D models, Nasal scripting, the effects system or the Canvas system.

Basically, it isn't realistic for FlightGear as a project to catch up with commercial projects in the core department - however, whenever people focused on providing technology enablers to middleware contributors (aircraft, scenery, content/base package/addons), the focus is shifted to content development, so that the onus is no longer on core developers to create contents - with the recent work on the compositor, world scenery 3.0 and the CompositeViewer/Canvas views, it's likely that these developments are -once again- huge technology enablers for content developers in the future.

So, while I certainly wouldn't hold my breath, FlightGear as a project is increasingly focusing on content developers and on enabling options rather than implementing end-users features directly. All of which is to say that exposing a handful internals could go a really long way to enable middleware developers to provide diagnostics so that people can get a better understanding of FlightGear's resource utilization - and in fact, a handful metrics can be easily made available in the form of properties.

And once that is the case, it also is relatively easy to use heuristics like frame rate, frame spacing or the performance monitor to track how memory/vram occupancy scales with different features and settings.
In general, it's very likely that core developers will be hugely interested in such regression tests and findings - whenever people have shared actionable information via the issue tracker or the developers mailing list, potential resource utlization improvements were almost always explored, discussed and implemented.

Thus, the main issue is the lack of bug reports with actionable information (such as valgrind traces) - but once people do provide such information, it's highly likely that people will act upon in - in fact, the commit logs reference literally dozens of commits that fix memory leaks or other resource utilization "bugs".

Apart from that, P3D is obviously a different project, and it's hard to draw any meaningful conclusions, except maybe for memory utllization "trends", but with 90% of the other factors being almost certainly different, it's unlikely that we can draw any important considerations by comparing X-Plane/MSFS or P3D with FlightGear.

People truly interested in understanding/improving FlightGear performance would be well advised to begin using the flight recorder subsystem to create so called "fgtapes" and tinker with replaying those using different startup/runtime settings, while logging internals to an XML/CSV file. And if you are really serious about doing that, consider running FlightGear in gdb/gprof or google perftools - that way, useful data can be obtained much more easily - and it's also much more likely that the corresponding data will be of interest to the core developer community.

Ideally, as a project, we would grow a library of fgtape files with different combinations of "flights" using different airports and aircraft, and the provide a built-in UI to enable people to "run" (replay) those files via the launcher to see how such replays differ for different people/systems and hardware configuration.

It is highly likely that having such a scheme in place will result in massive resource utilization improvements over time, because it would help lower the barrier to entry for less tech-savvy end-users to provide actionable information, while allowing core developers to compare FlightGear performance between 100s of different systems.

More recently, Jules has implemented a mechanism to transparently download/replay fgtape files via the command line - this means that such a mechanism could be used as the foundation for a variety of benchmarks, potentially with tweaking different subsystems or disabling them altogether, which will help track down some of the real culprits.

Either way, if you'd like to make a useful posting, consider forgetting about X-Plane/MS-FS or P3D - and instead, try running fgviewer or osgviewer to see how that performs when rendering your favorite scenery in terms of memory/CPU or VRAM utilization - that's much more relevant to FlightGear than any of the proprietary products.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Optimizing Scenery

Postby V12 » Tue Apr 13, 2021 8:45 am

Dear develoers,
I got chance to test Drzewiecki Design product New York X scenery (http://www.drzewiecki-design.net/prodNYCity_X.htm) on the same machine I have (AMD R7-3700X and GTX 1060). Results was amazing, poor R7 3700X and GTX 1060 were able maintain 30 fps. Same OSM scenery in FG knocks down fps to 10. This is clear evidence of the inefficiency somewhere in the FlightGear or generated OSM geometry or LOD management.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: Optimizing Scenery

Postby vnts » Tue Apr 13, 2021 11:45 am

V12 wrote in Tue Apr 13, 2021 8:45 am:
This is clear evidence of the inefficiency somewhere in the FlightGear or generated OSM geometry or LOD management.

The issue with objects in the old OSM2City format is well understood :) https://wiki.flightgear.org/OSM2City_1st_Worldbuild
StuartB's work on WS3 version of roads should lead to the road object being moved to the better format.
Once all objects are converted, FG will end up with the fast version and the other sim will end up being slow..
V12 wrote in Fri Mar 26, 2021 5:56 pm:For example, P3D v4.5 and resource heavy FSL A320 with visibility forced to 320 km does not reach 10 GB on same flight at FL390.

The plane won't take up much RAM compared to environment/scenery so it doesn't matter for this exercise.

Like Hooray said, FG and P3d aren't doing the same thing. So it's hard to compare:

- Terrain - does P3d have as detailed an elevation & landclass mesh, or are mountains more rounded like at LOWI? P3D also likely doesn't have textures in memory for close up detail - because it doesn't do that level of detail.
- Clouds - FG has very detailed volumetric clouds that respect 3d stucture, that are simulated/tracked.
- Does P3D actually draw land far out as 250km, or does it hide it in a fog...
- You may be loading trees, buildings, roads, etc. at far larger distance (Lod:rough) than P3d does.
- FG can do trees at far higher (i.e. realistic or somewhat realistic) densities. Other sims can't (the new microsoft sim uses fewer trees and a 3d tree shape to increase cover which results in forests that look from above like cupcakes arranged on a tray). This also takes up some memory.
- FG flight was higher so it may have loaded more terrain even though the visibility limits were different (horizon distance goes up with h^0.5).
- You may have had OSM2City on objects
- WS 2.0 doesn't have a LoD system and loads the full resolution entire terrain mesh AIUI. However the memory usage is reasonable, as people post screenshots of ranges far higher than 250km like a good portion of the Earth with the Space Shuttle e.g. this old screenshot at FL3300 : link. How much does FL3300 with draw distances like these take in P3D?
- As Hooray said WS3 will have a very powerful LoD algorithm, but FG can already do massive draw distances under 32 GB.
- As Hooray implied a memory leak could be contributing to RAM usage, on either FG or the other sim (it could be because of some codepath triggered by the specific craft).
Hooray wrote in Fri Mar 26, 2021 8:32 pm:To be honest, FlightGear has been lacking in many areas compared to commercial/proprietary products - however, given enough time, FlightGear also has been catching up with some of them (sooner or later). Often, it literally takes a while and features/components need to be reconsidered or even efforts started from scratch. Anyway, there are some examples where FlightGear has managed to acknowledge its strength and where it has come up with features that are empowering contributors, such as for example the property tree, XML, XML based FDMs, open 3D models, Nasal scripting, the effects system or the Canvas system.

Basically, it isn't realistic for FlightGear as a project to catch up with commercial projects in the core department - [..]

I agree entirely with the main point that FG's data driven approach is very helpful, and makes iteration faster even for people who compile from source as reloading is faster than even an incremental linking, people don't lose the train of thought while waiting/interacting with the compiler, and reloading means FG doesn't need to be restarted each time.

With regards to closed source commercial products: in an ideal world it's easy to imagine that the budgets would translate into better core in a lot of areas. Of course, the reality is quite different and FG's flight fundamentals are way ahead, and it is not the intent or vision of commercial management to catch up with FG.

A lot of the commercial sims historically have been from the game development side (Microsoft), and by companies who are listed in the stock exchange who additionally don't really care about simulation as their main activity. Then there are smaller studios like Laminar Research who are owned by people with an interest in flight, but get massively out-marketed and have to struggle. Until 2-3 years ago every other sim was 32 bit - then X-P 11 came out. X-P still uses a spherical earth model IIRC, none of the sims have a good a weather simulation, and the new MS sim and X-P only develop a runtime art based FDM so they aren't trying to be realistic. You've projects like P3D that have a limited FS-X license and mainly just allow people to replace parts of the engine while relying on the Lockheed Martin name, and FS-X derived projects with the other half owned by companies with no interest in being realistic.

I looked into the new Microsoft sim because of the initial hype videos and this is what I found (quick google/bookmarks from back then): The new Microsoft sim is made by a game studio - with the usual thing where they built an openworld game engine (which only supports DX11 [link]) and then realized it could be used to render a flightsim - but unlike Outerra they didn't give up after realising a sim was more than a game engine. They just bought the rights and like code to the early access version iteration of FS-X that Dovetail, a company formed by train-sim modders, was converting to 64 bit [link]. As the new MS sim is made by a game studio [link] and obviously they had no prior interest in flightsim [[url=https://www.youtube.com/watch?v=FDTT3oxArLI&t=251s
]link[/url]]. Microsoft's interest is in a console exclusive in their competition against Sony's console, so they are fine as long as they advertise massively and get some return. The game studio seems to have kept the game engine cloud plugin Dovetail was using, which only has clouds that look like porridge at close range and turn milky when flying through them, and they just took the fastest route to texturing and populating the world using existing photo maps and AI. The flight dynamics of both X-P and MS sim use runtime flight models to make it easier for art centered teams to release DLC, so they aren't really aiming for realism. That's also the reason why available commercial sims don't even have a decent weather simulation (XP would no doubt like one, but they have other priorities). The only reason the new MS sim had a non-trivial live weather implementation is because the people at the live weather service at least understood and saw possibilities.

I was reminded of this fact by the PR in the interviews for the new sim around launch: Part of the issue is that the game industry that much of these sims seem to come from, appear to massively underpay professionals (software engineers: link), as well as every other area [link]. This prevents software/hardware engineers and scientists with an interest or expertise relevant to flight simulation from being hired by management - at standard wages anyway. The usual games management strategy seems to be throw money at advertising, with one game in this list having a marketing budget of 4x the dev budget :shock: : link. It's easier than arranging all the R&D and nurturing teams. I was reminded of this point with the PR that //continually// mentioned one of the leads [1] went to a university that had a french name with applied mathematics in it [link] - trying to make it sound like the team had a scientist/engineer with a background in hardware as people would expect at least one in a sim project (look at XP or DCS) - it turned out to be just a quirk of the french education system and the maths being referred to was school maths. He turned out to be a (software) game developer before this project as expected, albeit one with an interest in ground vehicles.

So sadly you have a lot of bigger commercial projects which have no interest in being realistic, while the smaller privately owned companies are out marketed, too small, and also focusing on revenue. These projects will mainly be somewhat strong in core areas where they can plug-in existing game tools or re-use parts of their game engines (usually related to graphics), and feed it with art they can buy/make.

(You would know much this. I'm just making the general point that FG does have realism that other sims don't have, and have no real intention of ever achieving - but comparisons seem to be one way.)

Kind regards
vnts
 
Posts: 409
Joined: Thu Apr 02, 2015 1:29 am


Return to Scenery

Who is online

Users browsing this forum: No registered users and 9 guests