Board index FlightGear Release candidates

RC 3.4 memory consumption

Release candidate testers are encouraged to post their feedback here. Please read the introduction topic for details.
Forum rules
Please read the introduction topic for details.

Re: RC 3.4

Postby Thorsten » Sat Feb 07, 2015 9:27 am

No, we're not going to swap stories all day, one thing became very clear if i continue in the FG, my next computer will have 32 gigas of memory to run " clean", "soft", with '"only" 28 or 24 gigas.


I think this just turned a bit silly. Let me know when you're ready to return to a serious discussion.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4

Postby Müller » Sat Feb 07, 2015 9:41 am

I'm going to sleep, I've wasted too much time on this topic, I came to help, but it seems that everything is resolved, this citizen already solved everything. that is, no bug, since you have 16 gigas of memory ram and will run the FG, clean, soft with 9 or 7 gigas.
One more thing you don't know me, I be mad? out of control, but not really, but I've seen you here in discussions with punkepanda lose the course of conversation.
Close by here, bye bye Mr. Thorsten
I7-8700k - MSi Z370 Plus Sli - Galax RTX 2070 - G.Skill 16gb 3000mhz - SSD MX 500 -500gb, Hd 1tb Seagate, HD 2tb Wd - WC Corsair H100iv2 - Corsair RM 650 - Monitor 24'' Asus - Gab.Aerocool 800
Müller
 
Posts: 411
Joined: Mon Feb 20, 2012 1:37 am

Re: RC 3.4

Postby Thorsten » Sat Feb 07, 2015 10:50 am

One more thing you don't know me, I be mad?


I can't recall claiming that.

But:

You give us an actually measured number:

fgfs.exe: memory consumption, 3 gigas, 680 megas 132 kb,


and then you proceed with

one thing became very clear if i continue in the FG, my next computer will have 32 gigas of memory to run " clean", "soft", with '"only" 28 or 24 gigas.

that is, no bug, since you have 16 gigas of memory ram


Numbers don't mean anything to you apparently. I say I have 8 GB, yet you quote me with having 16. Okay - it's only a factor two, so why waste time on such trivial issues? You report you need 4 GB, and then you claim you're going to need 32 in the future - well, that's a factor eight appearing out of nowhere.

I don't know about you, but for me numbers mean something, and if we're going to find out whether something is odd, it matters whether it's 4 or 32. You can't randomly change the values.

I can take one more example of you steering this discussion away from being productive:

I'm telling you that visibility is the single most decisive issue in memory consumption because it goes with visibility squared. Next thing you do is show a screenshot which has no info on visibility settings and none on memory occupancy. You report memory consumption for a single flight - yet if on that flight visibility was initially bad and then increased a lot, you can observe memory to double. Conversely, if visibility is initially high and then gets bad, you can see that it goes down.

Memory consumption is a complicated problem, and it depends on a number of variables. Which is what I try to explain to you so that you assemble information which actually means something. But apparently you want it to be simple, you already 'know' that something 'must be wrong' and you want me to confirm that. But I can't do that, because I understand that the problem is complicated. So you blame me for this.

Ask yourself how reasonable that is.

There's been lots of discussion on memory usage on the mailing list, Rebecca has done a number of benchmark tests, I have done my own set, there's been a number of memory-saving features introduced - all of that means, we have fairly good data and a decent grasp where memory is going, and moreover, we know what questions to ask. You're probably unaware of all of that effort.

But it doesn't matter - since you already 'know' that something is not right. Isn't that the case? :-)

but I've seen you here in discussions with punkepanda lose the course of conversation.


Um... so your case that this is all somehow my fault is my interaction with a rather abusive user who has managed to be so disruptive that he was temporarily banned from the forum? Really? Think again.

Anyway, if you want this to be productive, try to understand why we need benchmark tests to understand anything and why your reported memory increase might mean anything since the confounding factors aren't under control.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4 memory consumption

Postby Hooray » Sat Feb 07, 2015 1:41 pm

I'd say "memory usage" (memory occupancy) and "memory consumption" (memory leaking) are two different issues - the former is indeed largely linked to scenery, but memory being "consumed" (leaking) can be seen in a number of subsystems completely unrelated to scenery/terrain - I think you can approach/troubleshoot this by proceeding step by step using the minimal startup profile, and adding/changing config options/features one by one.

Just as a reminder, we are having these discussions for a couple of years now - and some of the more senior contributors (including Thorsten and myself) have come up with various suggestions on fixing this, i.e. by either reverting scenery, changing visibility or using the minimal startup profile to better understand what's going on, all the while suggesting that the problem is primarily a configuration issue, not an "actual bug".

However, in the meantime, some of the more senior core developers (including Torsten and Rebecca) have actually identified bugs in the way memory is leaked, including the recently fixed listener issue that would gobble up hundreds of listeners - equally, there are similiar issues known about Nasal timers and listeners. So it really isn't unlikely that there's more going on beyond just a simple misconfiguration issue. Curt recently mentioned this on the devel list: stock FlightGear no longer scales well for lower-end hardware, usually because FlightGear is eating up crazy amounts of RAM - without that necessarily having to be scenery specific; to see for yourself: go to a location without any scenery (e.g. using the minimal startup profile) and pick an arbitrary aircraft while keeping FlightGear running over night, and you'll see valgrind/AddressSanitizer reporting memory leaks in quite a few places, without the scenery/tile manager even doing anything at all.

In retrospective, our understanding of the issue has been pretty incomplete admittedly given some of the recent discoveries - equally, we should be willing to accept that our current understanding may still not be entirely complete. Especially heavy aircraft cause FlightGear to leak memory in a number of subsystems completely unrelated to terrain/scenery, without any of that memory being managed by osg::ref_ptr<> either.

Given the recent work done by the TerraGear team (i.e. psadro_gm's LOD efforts) and the recent PagedLOD work done at the FG level, it is likely that scenery/terrain related issues are going to be addressed over time-equally, Rebecca Palmer recently mentioned that there are plans to explore using actual live RAM statistics to control PagedLOD behavior - but that doesn't magically solve any issues unrelated to scenery/terrain (think aircraft, Nasal).

Like Thorsten mentioned, the right approach would be coming up with a standardized test case ("benchmark") and repeating this using different startup settings run-time options, to draw conclusions from each test. This can be automated to a certain degree using pre-recorded flights, the replay system and/or flight plans (via the route manager). Currently, none of this can be fully automated, but with some shell scripting, it is do-able to run fgfs within a gdb/valgrind or AddressSanitizier session and log all data to a file for plotting. Like wlbragg suggested, using the "minimal startup profile" should provide a good baseline to start experimenting with. Equally, Philosopher's test-case can be used for running fgfs unattended while logging RAM usage stats to a file. There also is a built-in profiler using google perftools which can be used to do leak checking, too.

Overall, it is thanks to such tests and feedback that FlightGear releases 5 years from now will hopefully run more stable, while being overall better performing and more resource-friendly than FlightGear 3.xx :D
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: RC 3.4 memory consumption

Postby Thorsten » Sat Feb 07, 2015 2:03 pm

I'd say "memory usage" (memory occupancy) and "memory consumption" (memory leaking) are two different issues - the former is indeed largely linked to scenery, but memory being "consumed" (leaking) can be seen in a number of subsystems completely unrelated to scenery/terrain


Quite so, but we don't know with what we're dealing with here. The mere fact that you observe a memory increase from takeoff to landing can be caused by any number of factors - visibility increasing through the flight, starting from the coast and flying into the continent or starting in a sparsely populated region into city terrain and a well-modeled airport will all do it. So will a memory leak in aircraft systems, increased AI traffic and a core leak.

Which is why basically everyone agrees... benchmark testing.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4 memory consumption

Postby Hooray » Sat Feb 07, 2015 3:00 pm

Müller wrote in Sat Feb 07, 2015 8:52 am:No, we're not going to swap stories all day, one thing became very clear if i continue in the FG, my next computer will have 32 gigas of memory to run " clean", "soft", with '"only" 28 or 24 gigas.


This is one of the more common concerns expressed even by long-term contributors - in general, core development is hard to "affect" without being very involved unfortunately. If you are feeling fed up with the way core development has been happening recently and if you have the corresponding bandwidth, I'd suggest that you check out the osgEarth build/branch poweroftwo is maintaining, most scenery/terain related issues commonly known to affect FG performance, are basically gone there because a completely different terrain engine is used - obviously, things are currently not yet very well integrated - and there'll be many regressions, but overall performance definitely is very compelling for people on a cable/DSL connection.

However, if you should consider leaving FlightGear behind (as your posting above might suggest) because things no longer work too well for you, you are definitely not alone - and one of the best recommendations to be given in that case is quoted below, which is a posting from 2010 made by another long-term contributor in response to this farewell posting from Ian Cunningham:

https://www.mail-archive.com/flightgear ... 27889.html
Alex Perry wrote: I hope (and recommend) that you will irregularly take
another look, for example after every major release, to see whether
the technology implementation has changed and become suitable for you
to want to be involved with FGFS again. Over the years, there have
been several occasions where the developer team made changes that
precluded my involvement with the project. A year or two later, other
technology updates occurred that enabled me to rejoin the project
again. Given the number of times that's happened to me, and I know of
others it also happened to, it seems likely that you can follow the
same pattern.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: RC 3.4 memory consumption

Postby KL-666 » Sat Feb 07, 2015 5:38 pm

There is no reason to run away from fg immediately if there is a problem with a version. Once i just skipped a version because it had consistent crashes at places i wanted to fly. The version after that it was ok again.

Concerning memory issues:

I have to recalibrate my settings with every version since 3.0. Am on 3.2 now. The setting that influences memory absolutely most for me is LOD Bare. So I take a resource heavy plane like 777 and fly over the alps at my personal max cruising alt fl 400. All the time i watch total os memory consumption that it stays below 90%. I use total os memory because i also run Firefox for mpmap. That should be included in the total. This calibrating may take several overflights over the alps until i get it right. Too much memory consumption is obviously wrong. Too little gives me an unnecessary short LOD Bare. After such calibration i am fine anywhere with the new version.

BTW. Going from 3.0 to 3.2 the result was that i could use 20% less LOD Bare.

Please try this, before concluding a version is unworkable.

Kind regards, Vincent.
KL-666
 
Posts: 781
Joined: Sat Jan 19, 2013 2:32 pm

Re: RC 3.4 memory consumption

Postby Thorsten » Sat Feb 07, 2015 7:25 pm

No, we're not going to swap stories all day, one thing became very clear if i continue in the FG, my next computer will have 32 gigas of memory to run " clean", "soft", with '"only" 28 or 24 gigas.



This is one of the more common concerns expressed even by long-term contributors


Well, call me old-fashioned, but I don't care so much whether a concern is common, but whether it is justified.

The truth of the matter is that there's been lots of development on the memory front at the core since 2.12. The page LOD for objects, trees and buildings has reduced their footprint significantly whenever they come out of view. The modified tile cache behavior has reduced the footprint for long flights by a factor two or so, since tiles are freed from memory almost as soon as they're no longer visible, not kept to twice the visibility as before. Now we're getting another patch which drops the meta-information used to assemble object groups as soon as the tile is loaded, leading to another factor of two.

If you'd do a fair benchmark, you'd come to the conclusion that 3.4 is much lighter on memory than 2.12. Oops.

But, of course there's been lots of work on planes - the 777 in particular got quite a bit more detailed in the last year, so did the F-14b. The 777 from 2.12 was quite a bit lighter than the 777 from 3.4. Modelers have been busy populating the scenery with more static objects. We have some modest increase in texture quality. And expectations are moving upward - people want to use the goodies coming with the new version.

I understand that the average user can't appreciate these fine points - he sees his favourite plane, and the defaults, and makes a conclusion on memory. I have an opinion on how the defaults are set, how they should be set, and I've said my piece on the devel list - more than once. Anyone who likes can read it up there.

What I don't understand is that long-term contributors keep spreading the myth that 'FG development is on a wrong track, because it's getting such a resource hog and nobody tries to optimize anything because developers all have 16 GB and don't care'. Whereas in fact the opposite is happening. Is it a comfortable excuse somehow? Is it asked too much for a long-term contributor to understand memory management?

Anyway - please bury the myth. Truth isn't established by repeating something that's not true. It's also not established by many people repeating the same thing. It's established by evidence. Do a fair benchmark test, and you'll be surprised.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4 memory consumption

Postby Müller » Sat Feb 07, 2015 7:58 pm

wlbragg, thank you, for your suggestions and also for the willingness to make tests.
Hooray, my thanks also for your way to explain the topic, an intelligently, of easy understanding and also in a polite way.
KL-666 thanks for your post.
I haven't spent a whole night in FG watching that citizen wrote, I spent an evening doing tests, tweaks, to try to improve, lessen the increasing consumption of memory.
And one more thing, running only the FG sucking 3 gigas 680megas (nearly 700) and increased gradually, in a 32 bits would be something unusual, magical, in making the computer work without making a blue screen in front of you.
Well, I did the tests, but will not post results, I will not be the only one to get beaten here, has more than 5 thousand members in the Forum watching and so that others also do tests and post results. I'll cross my arms and stare as virtually everyone is doing.
I7-8700k - MSi Z370 Plus Sli - Galax RTX 2070 - G.Skill 16gb 3000mhz - SSD MX 500 -500gb, Hd 1tb Seagate, HD 2tb Wd - WC Corsair H100iv2 - Corsair RM 650 - Monitor 24'' Asus - Gab.Aerocool 800
Müller
 
Posts: 411
Joined: Mon Feb 20, 2012 1:37 am

Re: RC 3.4 memory consumption

Postby Thorsten » Sun Feb 08, 2015 7:25 am

Well, I did the tests, but will not post results,


Isn't that a bit childish?

And one more thing, running only the FG sucking 3 gigas 680megas (nearly 700) and increased gradually, in a 32 bits would be something unusual, magical, in making the computer work without making a blue screen in front of you.


Actually no, it wouldn't - if you'd run WS 1.0, it'd be quite fine on a 32bit system. It's the scenery data that takes the memory, not so much the core.

If you would have spent 10 minutes reading what I wrote before doing your tests, you'd have a better understanding of this because I explained it. What you're doing is re-inventing the wheel. But being a re-inventor of wheels myself, I can't really see that as a flaw - doing your own experiments always teaches you something.

I'll cross my arms and stare as virtually everyone is doing.


Have it you way - but you forfeit the moral right to complain about anything that way.

I've tried constructively to explain what we need in order to understand whether what you see is normal and to give you some background on how memory is spent, you have chosen to take my questions for the relevant info as an insult for some unfathomable reason. If you read back the conversation, you won't find any place where you have been 'beaten'. The claim is just as wrong as your huge numbers of 24 GB pulled out of somewhere.

It's okay if you decide not to be constructive, but please let's stay with the truth here - it's not okay to tell lies :-)
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4 memory consumption

Postby Müller » Sun Feb 08, 2015 8:18 pm

I am not childish, I'm here for fun and hobby, I try to learn something that makes me useful but what I enjoy doing and testing, benckmarks is not in any way a leisure to me. .. This isn't my workplace, understand it or not.
And again I say, there is this running one program on a 32-bit with this memory consumption, you will see the blue screen of death in front of you, if you want to distort things feel free, but don't put here untruths.
I7-8700k - MSi Z370 Plus Sli - Galax RTX 2070 - G.Skill 16gb 3000mhz - SSD MX 500 -500gb, Hd 1tb Seagate, HD 2tb Wd - WC Corsair H100iv2 - Corsair RM 650 - Monitor 24'' Asus - Gab.Aerocool 800
Müller
 
Posts: 411
Joined: Mon Feb 20, 2012 1:37 am

Re: RC 3.4 memory consumption

Postby Johan G » Sun Feb 08, 2015 10:14 pm

@ Müller:

I also run FlightGear 3.0. On my computer I get somewhere between 10 and 20 fps (sometimes even up to 30 fps) depending on location and nearby aircraft (though I avoid certain places, in particular San Fransisco, Paris and Berlin).

However I run FlightGear on a much, much weaker computer than yours. About half the CPU frequency and a fourth of the memory (click on my name for my profile and the specs). Yes, memory consumption usually rises to some 70-90 % on my computer.

There have only been one way to get there though, and that is reducing the settings. Thorsten in particular have tried to give you pointers on what settings would be worth adjusting. :roll:

Some of the settings I have reduced are:
  1. I have not even tried using Rembrandt (if I am not mistaken it has to render each frame four times)
  2. Visual range reduced to 10 km (reducing this has the biggest impact by far, well except Rembrandt I would guess)
  3. No random buildings
  4. Disabling complex aircraft, usually airliners, by moving them away from the aircraft directory
  5. No TerraSync as it will cause annoying delays while flying
  6. The shader effects slider set to the Performance end (in essence most of them will be disabled)
  7. No random objects
  8. Only using random vegetation when in otherwise "sparse" scenery (and at a low density as well)
  9. Rarely use 3D clouds
  10. Sometimes even turn off ALS

If you can accept less than perfect graphics, and have the patience to read in on what would work an try things out, you really should be able to get much, much better performance than me. :wink:
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Some YouTube videos
Johan G
Moderator
 
Posts: 6629
Joined: Fri Aug 06, 2010 6:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 2020.3.4
OS: Windows 10, 64 bit

Re: RC 3.4 memory consumption

Postby Hooray » Mon Feb 09, 2015 2:30 am

if more system level metrics (available RAM/VRAM, platform/architecture) were exposed as properties, we could either bail out during startup or at least disable certain options at run-time and show a dialog. People wanting to run Rembrandt or ALS on Intel GMA hardware is just plain ridiculous and should probably be discouraged - once such things are in the property tree, it only takes a handful lines of Nasal code to make certain settings unavailable (via the GUI that is)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: RC 3.4 memory consumption

Postby Thorsten » Mon Feb 09, 2015 7:41 am

I'm here for fun and hobby, I try to learn something that makes me useful but what I enjoy doing and testing, benckmarks is not in any way a leisure to me. .. This isn't my workplace, understand it or not.


Oops. Same here.

This isn't my workplace, it's supposed to be my hobby. I ain't get paid to fix bugs, I ain't get paid to explain things to users, I ain't get paid to do benchmarks either. And I don't have fun doing that either. So why do you think anyone here should bother with your problems?

But somebody has to do the dirty work, or we all end up with nothing. So since the world is more fair if we all pick our share of the less pleasant things, I do mine.

Yet you, my friend, feel that somebody else has to put up with your memory issues, work to reduce performance footprint, explain things, because for you it's a hobby, and that means we should do the work and you get to complain.

And then you go and tell us I've done tests, but I won't tell.

In doing so making really sure we'll never find out if what you see is a genuine problem or not - which incidentially ensures that it won't be fixed if it isn't real - which ensures that you can also complain in the future of how incompetent the developers are.

Tell me with a straight face that this isn't childish...



And again I say, there is this running one program on a 32-bit with this memory consumption, you will see the blue screen of death in front of you, if you want to distort things feel free, but don't put here untruths.


That's not what I said, I said the memory consumption depends on visibility and scenery, and you can run lots of visibility on 32bit systems with the 1.0 World Scenery without memory problems (I have tried 250 km once...). Of course with a 4+ GB consumption FG will crash - so configure it to use less, takes two minutes.

Needless to say, I can also configure FG to crash my system. How'd that prove that the software is bad?
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: RC 3.4 memory consumption

Postby Thorsten » Mon Feb 09, 2015 7:52 am

if more system level metrics (available RAM/VRAM, platform/architecture) were exposed as properties, we could either bail out during startup or at least disable certain options at run-time and show a dialog. People wanting to run Rembrandt or ALS on Intel GMA hardware is just plain ridiculous and should probably be discouraged - once such things are in the property tree, it only takes a handful lines of Nasal code to make certain settings unavailable (via the GUI that is)


The problem is that it's often either/or decisions.

I mean, for instance I can run a high visibility and LOD bare with sparse tree cover, or lower visibility and dense tree cover. For airlining I'd like to do the first, for glider or helicopter flight the second. I can't do both, as this would blow my memory.

I'd hope that if users get 2 fps with ALS on an Intel GMA, they'd realize that they should switch it off again. Of course, a fair fraction will opt to complain instead. But feedback really goes both ways - we should expose more options, even if they can crash FG if misused, and we should expose less because it's so complicated. Which probably means we're not on a bad path if the current state is equal misery for both camps.

Also, if we'd do that, we can look forward to 'Why is this option always disabled for me?' in the forum. I guess ultimately it's a bit against the idea of 'Fly free' and 'free software' which means you're given the thing, no strings attached, even if your use case is plain ridiculous, you're free to do that.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

PreviousNext

Return to Release candidates

Who is online

Users browsing this forum: No registered users and 3 guests