Board index FlightGear Release candidates 3.0

fg3.0rc1 memory usage

This is the archive of topics about the 3.0 release candidates.

Re: fg3.0rc1 memory usage

Postby KL-666 » Tue Feb 04, 2014 12:25 pm

@hooray

To reproduce you need to spend time flying, at least half hour to an hour. Fastest method is to fly high. Go to fl 400. The lower you fly the longer it takes. And fly over land, not sea, i flew over Europe. If all extra features in fg are off it also takes longer than with buildings,trees and advanced weather on.

If you are patient and fly long enough, then memory of the fg process will inevitable go to max AVAILABLE memory. If i have not much else running it goes over 10 out of 12 Gb. If i have another large process it goes to 7 Gb, if that is what is available. There it stays and you can continue to fly (at total mem use of 90-95%). So something in fg is managing memory. The parts of fg that have memory managed must be the ones that grow so excessively.

Someone told me he had 4 Gb filled up. So he bought another 4 Gb, and now the 8 Gb is filled up. And my 12 Gb gets filled up.

My system: Asus G75VX, gtx670mx, i7-3630qm, 12Gb mem, win8.1 64bit, fg3.0rc2
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby KL-666 » Tue Feb 04, 2014 1:44 pm

In the log i can find many messages about loading tiles:

Code: Select all
terrain:3:..\..\simgear\simgear\scene\tgdb\ReaderWriterSTG.cxx:488:Loading tile 3121873.stg
terrain:3:..\..\simgear\simgear\scene\tgdb\ReaderWriterSTG.cxx:256:Loading stg file e:/flightgear/terrasync/Objects/e010n40/e010n49/3121873.stg
terrain:3:..\..\simgear\simgear\scene\tgdb\ReaderWriterSTG.cxx:256:Loading stg file e:/flightgear/terrasync/Terrain/e010n40/e010n49/3121873.stg


But no messages about unloading tiles. Does anyone know what an unload log line would look like? Are unloads logged at all?

Another thing is that there are many messages about failing to remove .terrasync_cache files due to permission. But i can remove the files manually as the same user i start fg with (there is just one user, on my win8.1 laptop). Does fg use some system user secretly? Here an example message:

Code: Select all
io:4:..\..\simgear\simgear\misc\sg_path.cxx:585:file remove failed: (e:/flightgear/terrasync/Airports/B/G/.terrasync_cache)  reason: Permission denied
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Tue Feb 04, 2014 4:08 pm

tile loading/unloading is done via the tile manager, its logging class is SG_TERRAIN IIRC - so you can specifically look for such events.
the tile_mgr code not unloading tiles would fit earlier assumptions regarding the recent PagedLOD work potentially having an impact here.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11598
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby KL-666 » Tue Feb 04, 2014 5:04 pm

So that logging class write the lines starting with "terrain:" as in the first excerpt in my previous message? If that is the case, there are definitely no messages suggesting unloading in a logfile covering hundreds of miles flying.
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Tue Feb 04, 2014 5:18 pm

basically right - unless there's no such thing as an "unloading tile/terrain" message ...
Which needs to be checked first.

I think you mentioned being able to read code, right ?
If that's the case, you could look at the code to see what's going - and which messages you should see:

In general, the FG side of things is in $FG_SRC/Scenery: https://gitorious.org/fg/flightgear/sou ... rc/Scenery
Here, you will also find the tile manager subsystem: https://gitorious.org/fg/flightgear/sou ... ilemgr.cxx

In SimGear, you find the paging stuff (terrain, buildings, objects): https://gitorious.org/fg/simgear/source ... scene/tgdb

The main file here is obj.cxx: https://gitorious.org/fg/simgear/source ... db/obj.cxx
buildings and other features are implemented in separate files, see for example: https://gitorious.org/fg/simgear/source ... ingBin.cxx

Regarding log messages, you will want to look for SG_LOG uses - especially, SG_LOG(SG_TERRAIN).

that should take only a few minutes and give you a good idea regarding logging.

If you can build from source, we can provide some patches to provide additional diagnostics/logging.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11598
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby KL-666 » Tue Feb 04, 2014 10:21 pm

Hello Hooray,

I cant remember telling anyone i'm a c programmer, because that would be a lie. But i took a look, and what i make of it is that none of the tile manager / tile cache code that has to do with removal is ever executed (there is some logging in there, that is never executed). Better have a c programmer or even better the programmer of this code look at it, than me.

Kind regards, Vincent
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Tue Feb 04, 2014 10:24 pm

Did you raise your log-level accordingly ? Otherwise, those messages won't shop up.
if you're able to build from source, I would suggest to add some custom SG_LOG() statements to see if these are executed or not.
You can also cout or printf() instead.

You can also simply add those logging statements to the destructor of the object - but then you MUST use SG_LOG(), because the stuff is running in another thread - i.e. needs locking ... which is done automatically in the case of SGLOG

Thanks for looking into it!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11598
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby KL-666 » Tue Feb 04, 2014 10:53 pm

Ofcourse i set the loglevel where i need it, i am a programmer of profession, but not a c programmer. Therefore i have nothing set up to build from source in c. Finding out now how to do that on a windows box would be quite a waste of time. There must be people that have everything ready and can run tests in a whim. On top of that i said that i can not do much more in terms of logging, because where i thought i needed logging, there already were logging statements. But as a non c programmer i might have looked at some wrong methods.
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: fg3.0rc1 memory usage

Postby Johan G » Wed Feb 05, 2014 4:39 pm

Hooray wrote in Mon Feb 03, 2014 11:41 pm:...there's a reason why I ask for people to use the issue tracker and post their exact details there.
...
Now, try this for a change and report back if the issue remains or not: http://wiki.flightgear.org/Howto:Debugg ... l_settings
If it doesn't, re-enable features one by one and always check back if the issue shows up again or not - once it does, tell us exactly which settings contributed to the problem.

I have been trying this today, but ended up creating a few bug reports:
  • Issue 1355: "fgfs.exe -h" and "fgfs.exe -h" requires additional parameters to show usage information
  • Issue 1356: Disabling "Sky Blend" in FGRun andvanced dialouge causes FlightGear to not start
  • Issue 1357: Disabling clouds, textures and fog in FGRun has no effect
I think there might be a chance that some of these bugs are already fixed in 3.0.0-rc3. :wink:

I also noted that using an fgfsrc file is difficult from the console and impossible from FGRun, since there will be multiple options which seems to be another thing that makes FlightGear not run. :(
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 5634
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 3.0.0
OS: Windows 7, 32 bit

Re: fg3.0rc1 memory usage

Postby jomo » Thu Feb 06, 2014 8:00 am

I am not sure my findings fit 100% this header (I would have called it "TerraSync and Memory/Loading") - but I guess that are the same symptoms! So here my dime:

As ATC at EDDF I get a lot of visitors - and noticed that more and more people crash when getting close to EDDF - so I tried to find a reason for it:
So I gathered as much data as possible and placed those onto my homepage - You may download http://www.emmerich-j.de/TerSyn-Analysis.zip - I will keep that up there this month (and then delete):
- see especially my PC-data in the Sub "SysInfos"
- and the Start-File "Biggy.sh" ( I do not like the FGrun - it's startup is always very time-consuming!)
- for my Scenery Status see:
"--fg-scenery=/home/emmerich/FGFS/Scenery/eddf-ellx:/home/emmerich/FGFS/Scenery/TerraSync:/media/emmerich/Central/FGFS_S \"
    -- eddf-ellx is a AddOn to EDDF
    -- TerraSync was build up new from Scratch when new scenery became available by TerraSync
    -- FGFS_S is the "whole world" in the old style!
- my INTERNET-connection is an 16.000 unlimited from 1&1 via a Fritz 7390, PC is connected via LAN.
I guess in general that is "a medium equipment for FGFS" (slightly at the upper edge).

1) I flew some patterns around EDDF (EDDF -> RID -> CHA -> EDDF) and noticed each time a "Segmentation fault" near VOR CHA, just when crossing the 8°30' east. Looking into the attached fgfs_0.logBU I noticed constantly repeating errors for port 5505 and 16661 (was needed before 3.0 for ATLAS+TerraSync and FGCom). So I removed those - then I could complete my Pattern - and did not ever get that Problem back - even when adding those two definitions again. So:
--> seems that these "port errors" prevent TerraSync from loading new stg-files
--> is 5505 still needed for ATLAS ??
--> will 16661 be needed again in future for the final FGCom ??
--> wouldn't it be enough to report that "errors" once in the header - and not just mess up the whole LOG?

2) Then I tried a new route "EBBR to EDDF" (where I had no data yet in TerraSync):
a) TearrySync1: As in "Biggy.sh" - without the 2 ports:
--> Seg-Fault while loading stg-files
  • after several stg's were already loaded without problems
  • (no idea why it did not find the "..Central/FGFS_S" with its worldwide "Terrain" + "Objects" - but obviously that does not have any influence onto the problem - so I do not care (for now))
  • see the "Memory" slowly raising while "Receiving" is active
  • the "CPU1" each time gets limited at 100% while loading

b) TearrySync2: As in "Biggy.sh" - I found in http://wiki.flightgear.org/TerraSync that I may have to use the extra "--terrasync-dir=.." - so I did:
--> "Seg-Fault while loading stg-files
  • But interesting: Comparing the fgfs.log from TerraSync1 with the one in TerraSync2 you notice that this time the problem occurred later - i.e. it is not a fault in the scenery-files - but terrasync obviously can always only handle a certain amount of NEW stg's !

c) TearrySync3: As in "Biggy.sh" - but just to make sure there is no problem when using old/new scenery I removed all other scenery-dir's except the "TerraSync" (containing only new scenery!):
--> same result
  • so it seems definitely that TerraSync is very much limited in the amount of downloading new scenery (at a certain speed/time !?! - BUT 380KN SEEMS NOT THAT FAST!!)

My conclusion:
  • I) Is it possible to make that TerraSync Fail-Safe?
      - at least report a real cause prior to crash ??
      - even better: If you cannot download in time just stop FGFS (like PAUSE) and continue when loaded - not nice for the user - but if you maybe display the reason on screen the user may forgive you!
  • II) And see the constant upload into memory!
      Should there also be some download again after some time? Memory cannot raise unlimited during long flights in new terrain!

pls tell me if you need more data - I will be glad to help
thx for your efforts (and the new scenery looks a lot better)
jomo / ATCjomo + EDDFjo1 + EDDFjo2
ATC at EDDF Fr,Sa,Su,We from 20:00 to 24:00 CET/MEZ., see http://www.emmerich-j.de
User avatar
jomo
 
Posts: 936
Joined: Thu Feb 12, 2009 6:46 pm
Location: Mainz, Germany
Callsign: jomo jomoATC
OS: UBUNTU 18.4

Re: fg3.0rc1 memory usage

Postby adrian » Thu Feb 06, 2014 9:18 am

Guys, I hate to repeat myself, but the tile unloading stuff is deferred to the OSGdb pager thread. The problem with memory is related to the fact that the tile manager stores too many tiles at once. If I'm not mistaken, I even posted a screenshot with a modified map showing all tiles which are scheduled by the scenery manager. That's why I resorted to using a tile ring instead of a long tail buffer. (actually there were two rings, but since terrain LOD is absent and PagedLod is in effect now that's obsolete)
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Thu Feb 06, 2014 10:01 am

I remember seeing that screen shot - showing a modified version of the map dialog, right ?
Something like this could be useful to look behind the scenes.
I am not sure if the problem is as simple as that though - we have several factors involved here.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11598
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby adrian » Thu Feb 06, 2014 10:16 am

Her eit is again:
Image

I'm actually sure the problem is more complex, but what I said should be a good starting point.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: fg3.0rc1 memory usage

Postby Hooray » Thu Feb 06, 2014 10:20 am

this would seem like another good candidate for review by Zakalawe and/or Stuart as it could be a great debugging aid when dealing with the tile manager
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11598
Joined: Tue Mar 25, 2008 8:40 am

Re: fg3.0rc1 memory usage

Postby adrian » Thu Feb 06, 2014 10:41 am

Notice how there's no tail of terrain behind the AC, terrain always stays constant outside the viewer. I'm sorry I don't seem to find the other screenshot showing the default behaviour of the tile manager, without my patches (long tail behind the aircraft, double the number of tiles which are not viewable behind the aircraft).
The behaviour was due to keeping lots and lots of tiles to have the tower view and prevent terrain reloading when switching views. My idea was to disable the tower view when further than a couple of clicks from the AC, since the AC would become a dot anyway. Didn't make it to review.
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

PreviousNext

Return to 3.0

Who is online

Users browsing this forum: No registered users and 1 guest