Board index FlightGear Release candidates 3.0

fg3.0rc1 memory usage

This is the archive of topics about the 3.0 release candidates.

fg3.0rc1 memory usage

Postby KL-666 » Mon Feb 03, 2014 2:15 am

Watched the fgfs process memory usage, and found out it wants all availabe memory always. Whether there is 4, 8 or 12 Gb, it wil take all available memory eventually. There seems to be no memory leak, because it caps at all available memory. In my case around 10/12 Gb. It does not make a diff which plane i use, whether buildings and trees are on or off, or weather is advanced or basic. Mem goes eventually to max.

But there are problems with that approach:

- is the memory necessary or just stale? Why can somone with 4 Gb fly with a 3 Gb fgfs process, and do i need a 10 Gb process with 12 Gb mem. If i would have had a 100 Gb, would fg fill it up till 95 Gb?

- In multiplayer you need headroom for huge Gb planes when they come in reach. If you get near an airport with 3 1 Gb planes, you suddenly need 3 Gb extra. What will the os say to that, when you have already filled the memory to it's limit? I can make a good guess, something everyone experiences all the time. The os will say: you will not get that much mem extra in the current conditions. I'll just let you crash.

I'd suggest the limit should not be max mem, but maximized to some number or percentage near 50%.
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby someguy » Mon Feb 03, 2014 2:32 am

FG 2.12 ran fine on my Mac with 4 GB. 3.0 would run, but would start swapping to disk (VM) within minutes. I added another 8 GB, and it generally is satisfied with filling 4 to 6. But one time I noticed I had less than one GB left out of 12.

I think it does help that I deleted all the airliners and other fat-assed models. FG has a large folder full of stripped-down models, but it won't load them instead of MP planes. Not sure what they're for, it seems like a wasted opportunity.
User avatar
someguy
 
Posts: 1653
Joined: Tue Nov 25, 2008 5:54 am
Location: USA
Version: 2016.2.1
OS: Mac OS X 10.11

Re: fg3.0rc1 memory usage

Postby Johan G » Mon Feb 03, 2014 2:46 am

KL-666 wrote in Mon Feb 03, 2014 2:15 am:Watched the fgfs process memory usage, and found out it wants all availabe memory always... There seems to be no memory leak, because it caps at all available memory... It does not make a diff which plane i use, whether buildings and trees are on or off, or weather is advanced or basic. Mem goes eventually to max.

I have noticed the same on Win 7. Though in my case it caps at about 90% of 2 GB (I think). I had the "activity manager" (or what it is called in English) open to watch memory usage in a few of the first runs and thought I would get into trouble as the memory usage ramped up, but thankfully it stopped. For me it is ok, at least so far.

I guess as a flight simulator it climbs all the way up to a high cruising altitude. :wink: :lol:
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5406
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: fg3.0rc1 memory usage

Postby Hooray » Mon Feb 03, 2014 2:48 am

We've been discussing this many times over the years, so we're kinda aware of it - we just were not aware of it being still so serious, Thorsten just raised the same issue on the devel list...
And I happened to add a summary to the wiki just a couple of days ago, to make this a bit more prominent: http://wiki.flightgear.org/Subsystem-le ... Background

A while back, I started preparing a patch to add the corresponding diagnostics (RAM usage tracking) to FG so that we can better identify problematic features, and either fix them or at least provide feature-scaling support if necessary. Back then, this was unfortunately ignored and disregarded - recently, this is getting some more attention again, so we'll probably revisit the issue during the 3.2 release cycle. The first step really is getting "live" memory usage details, and exposing this via the property tree - so that we can tell 1) how much RAM is there, 2) how much RAM is being used by fgfs, 3) and: are we swapping ?

These three things are trivial to implement, I provided a patch doing this on Linux and it only took 20-30 minutes - so this could be done for Windows & Mac/OSX, too - and it would be straightforward.
That alone will not yet improve things - it will just show us that there is a problem.

The next step would involve doing the whole thing at a more fine-grained level, i.e. for each subsystem - so that we would have a list of subsystems, analogous to ThorstenB's system monitor, but see how much RAM is being used, allocated and freed per minute/5 minutes/10 minutes etc - analogous to how htop/top work

That should give us a list of problematic subsystems, which would either need to be fixed, or made more configurable.
Basically, we fully understand what's involved - it just hasn't been on anybody's plate recently, and even I had forgotten about it, until some folks around here posted crash reports despite having 10+ gb of physical RAM.

So, yeah - we do have a real problem here, but it surely won't be fixed in time for 3.0 unfortunately .. too late for that I'm afraid, we all worked on different features, and most of us tend to belong to the camp of people with really powerful computers, so that we won't necessarily notice such things - keep in mind that many of the people able to "fix" such problems are also the ones who happen to primarily DEVELOP the sim, not really USE it in the sense of spending many hours each month flying around and letting FG gobble up GBs of memory.

Overall, if you want to help with this, please provide feedback - feel free to use the issue tracker, you have found a real bug here, and a critical one, that needs to be addressed during the 3.2 release cycle - you can also help by updating and maintaining the wiki article, and adding your figures there.

But don't expect this to be fixed anytime soon - people using lots of resource-hungry features need to be aware that even powerful computers may no longer suffice at some point ...

Bottom line being, we need to add some hooks to expose memory usage stats to the property tree, of the whole process, but also of each subsystem - and provide a mechanism to do dynamic feature scaling in response to these numbers, i.e. to tell subsystems "STOP USING RAM NOW" - and we really need to do some kind of regression testing to watch these numbers for each release, i.e. using either prerecorded flights or replay/flight recorder profiles and watch memory usage with different settings (AI/MP, route manager etc)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby Johan G » Mon Feb 03, 2014 3:10 am

My layman impression without knowing what is going on under the hood is that FlightGear loads as much scenery as it dares.
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5406
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: fg3.0rc1 memory usage

Postby KL-666 » Mon Feb 03, 2014 3:15 am

There may be problems i did not notice, but what i noticed is the mem usage caps at max available mem. So that means every part wants much mem, but it is managed by something. And that something gives away mem up to max mem. If just that something in fg could cap at 50% of max mem, then i think a lot of trouble with needing extra mem later is solved.
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Mon Feb 03, 2014 3:24 am

@Johan: If it's really as simple (reproducible) as that, that would be great - because then, we have a at least one suspect/candidate: For example, the random buildings code was recently modified and now uses OSG's PagedLOD IIRC (which is meant to optimize/reduce memory usage), so if there's some bug in it or some related component/library (dependency), that /could/ explain what's going on here (absent any other ideas) - and it should be straightforward to check and fix if that's the case - then again, it would not solve the underlying issue, that is, lack of a dedicated memory tracking method to identify such problems much earlier - we've been seeing a very similar problem after the v2 scenery got released, when Brian Schack found out -using atlas- that the scenery had massive amounts of unnecessary triangles - this went unnoticed for months, so that it made it into the scenery release ...

Now if this should turn out to be such a massive thing in a subsystem/feature that was recently touched, it's at least pretty self-contained and can probably be solved by having some more eyeballs looking at the code in question. Again, I suggest to file a bug report, and post all the info you guys have - so that core developers can have a look, by focusing on recently changed systems like the PagedLOD stuff.

Overall, the real problem remains - no matter if the problem should be fixed in time for the 3.0 release or not, we really need to have tools to better tell how much RAM is being used, and where exactly it is used, i.e. which features are responsible for massive amounts of consumed memory. We had a lengthy discussion about that 2 years ago, not many people agreed back then - maybe it really takes such a massive "bug" to change things for the better ... I don't know.

When you post your bug report, be as specific as you can - i.e. hardware/OS, but also the HELP/ABOUT info - especially OSG version etc.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby adrian » Mon Feb 03, 2014 1:15 pm

Heh, welcome to SRTM3 + Corine scenery boys :D
Some of us had been having this problem since custom scenery appeared. Hence the various patches to load as few stuff as possible in the memory.
I know Stuart has implemented PagedLOD for everything besides terrain, my only question to him would be: is the stuff unloaded at proper time? I know the OSG DB pager thread is taking its time deleting stuff. My patch was forcing unload of tiles, so I fixed my memory issues the hardcore way. Stuart's way is more stylish, but there may be unknown problems with freeing memory when out of range (goes for the tile manager too, since it keeps way too many tiles in memory at once).
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Mon Feb 03, 2014 4:09 pm

As previously mentioned by bschack, the other problem is related to v2 scenery having massive amounts of unnecessary triangles: viewtopic.php?f=5&t=21498&p=195610&hilit=#p195610
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby Thorsten » Mon Feb 03, 2014 4:10 pm

I have (for unrelated reasons) made some experiments with memory occupation and scenery yesterday.

I can load 300 km visibility worth of CORINE scenery around the aircraft (bare scenery, i.e. no trees, buildings or objects), that took about 6 GB of memory. It completely jams the vertex shader though, I think I ended up with 7 fps.

Loading the entirety of Iceland from the new scenery into memory took 5.7 GB, again jamming the vertex shader down to 7 fps. That's theoretically with random buildings and trees, except on Iceland there are really close to none.

Loading 500 km worth of old scenery is comparatively unproblematic - my spaceflight shots over the Mojave with 500 - 600 km visibility had I think about 4 GB of memory consumption and had about 20 fps.

Given that even for these insane visibility ranges memory consumption remains manageable and that the number of terrain vertices goes with the square of the visibility range, I do not think that even CORINE terrain can be really doing so much harm. Say when almost no terrain is loaded on the runway, I have about a GB of memory occupied (that's OS and FG and whatnot). So if 300 km of terrain are worth 5 GB, then 30 km must be 1/100 of that. You can keep loads of that in memory before things go really bad.
Thorsten
 
Posts: 10587
Joined: Mon Nov 02, 2009 8:33 am

Re: fg3.0rc1 memory usage

Postby Hooray » Mon Feb 03, 2014 4:20 pm

At least the random buildings code got some logging statements (SG_TERRAIN) to tell us what's going on there - so if it should have anything to do with this, the logs should be helpful. Otherwise, ref_ptr would need to be overloaded to get runtime stats ...

For the time being, we do not even know what could be the reason here - random buildings and the recent PagedLOD work were mentioned, because those would seem like obvious candidates (having to do with RAM management, and being instantiated very often) - but that doesn't have to mean anything.

So first of all, someone should check if the problem seems in affected in any way by completely disabling certain features, such as vegetation, random objects or random buildings.
If so, please report back here so that we can take a look - or directly use the issue tracker.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby KL-666 » Mon Feb 03, 2014 10:08 pm

Hooray wrote:

So first of all, someone should check if the problem seems in affected in any way by completely disabling certain features, such as vegetation, random objects or random buildings.
If so, please report back here so that we can take a look - or directly use the issue tracker.


Maybe you want to read the OP. There is amongst other combinations flown with everything off: no vegetation, random objects or random buildings and and no advanced weather.
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Mon Feb 03, 2014 11:41 pm

well, let's try to keep this constructive, ok ?
I am responding to dozens of postings here each week, and there's a reason why I ask for people to use the issue tracker and post their exact details there.
As long as you refuse to comply with our requests, there's not a good chance to see this fixed anytime soon.

Obviously, quite a few developers are not seeing these problems - and I have been talking with two more developers about this to ask for feedback, and neither of them saw this - so there MUST be something going here, and as long as you refuse to share a reproducible setup, we cannot help you.

Yes, there are quite a few issues related to memory management (lack thereof) - but you guys need to be our eyes and ears here, as we cannot see what you're doing.

Now, try this for a change and report back if the issue remains or not: http://wiki.flightgear.org/Howto:Debugg ... l_settings
If it doesn't, re-enable features one by one and always check back if the issue shows up again or not - once it does, tell us exactly which settings contributed to the problem.

Thank you!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby Hooray » Tue Feb 04, 2014 2:20 am

I have asked a few more people if they are also seeing such an excessive memory occupation, and nobody could confirm this so far - so we really need to gather some more details here, i.e. platform/OS, settiings (startup & runtime) to see if there really is an urgent problem, if we're dealing with system-specific settings or with people mis-interpreting memory stats ... I am not saying there's no problem, but please be as specific as you can be, so that others can reproduce your settings. There's only two weeks remaining until the 3.0 release ...

Also, here are some troubleshooting tips kindly provided by poweroftwo - one of our windows-based developers familiar with the scenery/OSG side of things in FG:


poweroftwo wrote:
    * what OS?
    * If running on Windows, is fgfs 32 or 64 bit? If 32 bit, Windows limits application memory space to 2GB max! The only way to get to 3GB that I am aware of is to turn on the large address awareness (see http://msdn.microsoft.com/en-us/library ... 73(v=vs.85).aspx). Once the 2GB (private app bytes) are exceeded or 3GB (w/large address aware), OpenGL context will go bad for certain), but user won't necessarily see an exception just oddities as memory thrashing continues.
    * For windows actual memory monitoring (32/64) I highly suggest running MS' SysInternals Process Explorer available here:
    http://technet.microsoft.com/en-us/sysi ... 96653.aspx . It can run as a task manager replacement and is more functional. Run fgfs and open details to see actual private bytes being consumed.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11312
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby dany93 » Tue Feb 04, 2014 10:46 am

I have installed FG 3.0.0 RC2.

Impossible to fly with Terrasync downloaded sceneries or France850 sceneries. I've tried in France (near Paris, or Britanny). RAM occupation increases, I have long freezes, 2 - 3 mn (swaps ?? Not sure, or not every time. The disk can be quiet at 1 - 2%, but the 4 CPU cores work at 50%). Like this, up to the crash after less than 5 - 10 mn.
I can only fly in KSFO region.

My observation is that it occurs every time after the RAM occupation reaches 90 - 95% of my available 3GB.
This happens even with UFO, and defaults settings in FG.

Windows takes 35 - 40% (1 - 1.2 Go) RAM, et I've seen 1.55 - 1.77 Go for fgfs.exe.
________________________________________________________________________

My system:
Dell XPS 420 with Windows Vista 6.0 (No 6002 ; Service Pack 2) 32 bits
Graphics card: NVIDIA GEFORCE GTX 650 1GB (driver 331.65)

CPU : Intel Core2 Quad CPU Q 6600 @ 2.40 GHz, 2394 MHz, FSB 1066 MHz
RAM 3 Go (Windows 32 bits).
Physical installed RAM 4x1024 MB Dual Channel DDR2 800 MHz
Cache memory 6.19 Go (never saturated).
_________________________________________________________________________

Command line (copy from FGRun):

C:\FG\FG300_RC2\bin\Win32\fgfs.exe
--fg-root=C:\FG\FG300_RC2\data
--fg-scenery=C:\FG\FG300_RC2\data\Scenery;C:\FG\FG300_RC2\scenery;C:/Users/Daniel_2/scenery-france-850;C:\FG\Terrasync300
--terrasync-dir=C:\FG\Terrasync300
--fg-aircraft=C:/FG/Aircraft-Dan
--airport=LFPG
--aircraft=ufo
--control=joystick
--language=en
--disable-random-objects
--prop:/sim/rendering/random-vegetation=false
--disable-ai-models
--disable-ai-traffic
--disable-real-weather-fetch
--geometry=1024x768
--bpp=32
--timeofday=noon
--disable-terrasync
--disable-fgcom

No error messages in the usual console (having it by --console), even at FG crash with "Send error report"
Last edited by dany93 on Tue Feb 04, 2014 5:44 pm, edited 3 times in total.
dany93
 
Posts: 764
Joined: Mon Sep 07, 2009 3:43 pm
Location: France (Paris region)
Version: 2018.4.0
OS: Linux Mint 18 (64 b)

Next

Return to 3.0

Who is online

Users browsing this forum: No registered users and 0 guests