Board index FlightGear Support

How to clean up TerraSync data

All general support: help on flying, installation, hardware, getting online etc. There are lots of users and developers to help you out.
Forum rules
In order to help you, we need to know a lot of information. Make sure to include answers to at least the following questions in your initial post.

- what OS (Windows Xp/Vista, Mac etc.) are you running?
- what FlightGear version do you use?
- what graphics card do you have?
- does the problem occur with any aircraft, at any airport?
- where did you download your aircraft/scenery from?
- is there any output printed to the console (black window)?
- copy&paste your commandline (tick the "Show commandline box on the last page of FGRun or the "Others" section on the Mac launcher).

Please report any bugs not specific to an aircraft on the issue tracker.
To run FlightGear on old computers with bad OpenGL support, please take a look at this wiki article.

Note: If you did not get a reponse, even after 7 days, you may want to check out the FlightGear mailing lists to ask your question there.

How to clean up TerraSync data

Postby IceDiamond » Sat Mar 26, 2022 3:58 pm

My drive almost run out of storage, only 20GB left, then I found that there is 60GB data on user/FlightGear/Downloads/TerraSync.
I want to clean this up to get some storage, but how to clean up these data?
IceDiamond
 
Posts: 20
Joined: Sun Mar 08, 2020 6:39 am

Re: How to clean up TerraSync data

Postby Michat » Sat Mar 26, 2022 4:58 pm

Go to Terrasync folder and remove the content. Will give you back your 60 GB.

29.92
User avatar
Michat
 
Posts: 1226
Joined: Mon Jan 25, 2010 7:24 pm
Location: Spain
Version: 191b
OS: MX 21 Fluxbox oniMac

Re: How to clean up TerraSync data

Postby IceDiamond » Sat Mar 26, 2022 5:12 pm

Michat wrote in Sat Mar 26, 2022 4:58 pm:Go to Terrasync folder and remove the content. Will give you back your 60 GB.

29.92

Directly?
IceDiamond
 
Posts: 20
Joined: Sun Mar 08, 2020 6:39 am

Re: How to clean up TerraSync data

Postby TheEagle » Sat Mar 26, 2022 6:23 pm

Yes - note though that you'll have to redownload the scenery for any area you are going to be flying in after that. I recommend not to delete the Models/ folder - it will be downloaded again as a whole anyways the next time you open FG - deleting that folder only brings you additional waiting time the next time you want to fly and puts extra strain on the TerraSync servers.
Cessna 210 (Wiki)
My other aircraft: my wiki profile !
Other: FGTools (GitHub)
World tour: View on SkyVector
Please consider donating $1 / €1 to help me finance a new camera !
User avatar
TheEagle
 
Posts: 3411
Joined: Sat May 01, 2021 3:27 pm
Location: France
Pronouns: You, he
Callsign: F-EAGLE
IRC name: none
Version: Git next
OS: Ubuntu Studio 22.04

Re: How to clean up TerraSync data

Postby wkitty42 » Sun Mar 27, 2022 1:23 pm

FWIW: it should be noted that the scenery for the entire world is over 200GB in total... i haven't updated my entire world directory in a long time so i don't have current figures for it all but what i do have on my main system tallies up with the following figures...

Code: Select all
Size: 246.7GiB (264930183688)
      2794735 files, 48149 sub-folders
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: How to clean up TerraSync data

Postby Alant » Sun Mar 27, 2022 2:42 pm

Here I see 155GB, 1,765,085 files, 50,638 folders.
Updated with this command
py D:\fg\flightgear\scripts\python\TerraSync\terrasync.py --target=D:\fg\WorldScenery --quick --remove-orphan --url=https://ukmirror.flightgear.org/fgscenery
Alant
 
Posts: 1219
Joined: Wed Jun 23, 2010 6:58 am
Location: Portugal
Callsign: Tarnish99
Version: latest Git
OS: Windows 10/11

Re: How to clean up TerraSync data

Postby wkitty42 » Sun Mar 27, 2022 7:25 pm

FWIW, alant... that's only the regular scenery stuff... that URL does not contain any of the osm2city stuff... getting that requires using another URL as well as a different command line so you don't lose what you already have... specifically, you don't want to use "--remove-orphan"...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: How to clean up TerraSync data

Postby benih » Tue Nov 28, 2023 1:57 pm

Can one selectively delete stuff (like terrain, osm data, etc) older than x, like this?
Code: Select all
TerraSync$ find . -type f -mtime -365 -delete


I got this structure (175GB in total):
Code: Select all
TerraSync$ du --si -s *
312M    Airports
47G     Buildings
23G     Details
531M    Models
718M    Objects
25G     Pylons
17k     RecheckCache
72G     Roads
7,8G    Terrain
4,1k    TerraSync-WARNING.txt
13k     Trees
User avatar
benih
 
Posts: 1689
Joined: Tue Aug 15, 2017 10:34 am
Callsign: D-EBHX
Version: next
OS: Debian Linux 64bit

Re: How to clean up TerraSync data

Postby polly » Tue Nov 28, 2023 2:03 pm

I think wkitty already answered in the negatory, besides, looking at my scenery folder there's a huge disparity in sizes, W080N40/Roads is a whopping 9Gby plus, ( trying to drive around here it makes sense ) but it means I'd be continually whacking and reloading the one folder to make any meningful change to the folder's level.
User avatar
polly
 
Posts: 956
Joined: Thu Nov 04, 2010 3:45 pm

Re: How to clean up TerraSync data

Postby benih » Tue Nov 28, 2023 3:33 pm

I'm unsure.

So I:
- cannot delete single files
- but i could remove for example ALL the roads, by removing the Roads subfolder?
User avatar
benih
 
Posts: 1689
Joined: Tue Aug 15, 2017 10:34 am
Callsign: D-EBHX
Version: next
OS: Debian Linux 64bit

Re: How to clean up TerraSync data

Postby polly » Tue Nov 28, 2023 3:49 pm

Sorry to be unclear, wkitty42 answered on another thread that it is not possible to delete parts of the scenery folder according to date. After looking at my scenery folder I concluded that many of the older dated files were much, much smaller than a few of the biggest folders, W080N40/Roads being the biggest by far in my scenery. So to get any meaningful reduction in the size of my scenery I'd have to delete the biggest, which would immediately get downloaded again. I ended up doing an LV resize.
User avatar
polly
 
Posts: 956
Joined: Thu Nov 04, 2010 3:45 pm

Re: How to clean up TerraSync data

Postby wkitty42 » Tue Nov 28, 2023 4:57 pm

benih wrote in Tue Nov 28, 2023 1:57 pm:Can one selectively delete stuff (like terrain, osm data, etc) older than x, like this?
Code: Select all
TerraSync$ find . -type f -mtime -365 -delete

that could work but depending on where you run it, there are some things you do not want to delete, though... like do not delete the three files in the base of TerraSync...

if i were to work up some external "remove terrasync tiles older than X days" script, i would make a list of files in specific terrasync directories (eg: do NOT dig into the Airports and Models directories) that meet the criteria, work out which tile directory they are in, and then remove all the files in that directory... the last step would be to remove any empty directories... note that there will be dotted files in each directory, too... .dirhash and .dirindex... if a directory is being wiped clean, these files need to go, too, so the directory is really empty and can be removed... i mention these specifically because some tools cannot see dotted files without specific instructions...

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

i had to laugh when i saw this pop up in today's run through the forum... i installed TerraMaster the other day because i was helping someone and i also wanted to see how my almost full load of terrasync looked and update it since it has been a couple of years... one of the first things i noticed was a band completely around the Earth at 17S latitude... "that's interesting," i thought... i kind of remembered seeing it some years back and thought it might have been a bug that's since been fixed and TS cleaned up...

to test this, i removed all of the ones at this latitude that were in the ocean because there can't be islands or objects all the way around the Earth in this one specific latitude, right? once they were removed, i exited TerraMaster, fired up FG with the UFO, and spawned in at some longitude and 16.5S latitude... then i proceeded to fly around the world at 90True or 270True... either one would keep me at 16.5S all the way around the Earth... when i completed the full circuit, i exited FG, fired up TerraMaster, and took another look... low and behold that entire line had returned! i was quite surprised...

now, i'm not sure how TM decides to draw a colored frame around the tiles... does the existence of the deepest tile directory cause it or must there also be an .stg? i don't know... i do still find some top level eXXXnXX directories with only .dirhash and .dirindex files in them... the .dirhash files shows the hash of the .dirindex file and the .dirindex file shows hashes and file sizes (i think) for txz files which do not exist... i'm thinking that TM isn't fully updated to the current scenery methodology... the last release of TM was back in 2021 during the time that the OSM stuff was being added and figured out how to store and update... i do remember that that's when the txz files were introduced...

in any case, while i was going through all of that, i decided to whip up a script to remove empty directories because i didn't want anything left of 17S that was not on actual terrain... the script is very similar to what is being asked here about removing files/tile directories that are older than X days...

the meat of my "remove empty files and directories" (bash) script follows...
Code: Select all
#work only inside these six terrasync directories
for foo in Buildings Details Objects Pylons Roads Terrain
do
  #find files first, then directories
  for bar in f d
  do
    #build command line for test mode
    CMD="find $HOME/myflightgear/TSScenery/$foo -empty -type $bar"
    #if this is NOT a test, add the -delete command with -print to log everything being removed
    if [[ "$MYTEST" = false ]] ; then
      CMD="$CMD -delete -print"
    fi
    printf "\n\nExecuting: $CMD\n"
    printf "################################################################################\n\n"
    $CMD
    printf "\n\n################################################################################\n"
  done
done


while writing this response, i had a thought that maybe detecting the -mtime of only the directories should work, too... if the directory hasn't been modified in X days, it doesn't matter about the files inside it... just delete them, remove the directory, and go to the next one... the question, then, is what does FG and/or TerraMaster do when they process the .dirhash and .dirindex files and see the referenced directories do not exist... do they recreate them with .dirhash and .dirindex files in them? if so, then i guess we could leave them alone but it doesn't really matter since they don't consume that much space, really... i guess one could test this pretty easily by creating a new testing terrasync directory for FG and TerraMaster to use and then flying in a limited area to see what directories and files are created/downloaded... i would hope only the ones needed are created and not a bunch of empty ones containing only .dirhash and .dirindex... and yes, i know this script cannot remove directories that do only contain .dirhash and .dirindex since the directories are, technically, not empty...
Last edited by wkitty42 on Tue Nov 28, 2023 5:01 pm, edited 2 times in total.
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: How to clean up TerraSync data

Postby wkitty42 » Tue Nov 28, 2023 4:58 pm

polly wrote in Tue Nov 28, 2023 2:03 pm:I think wkitty already answered in the negatory

i said that about FG doing it automatically... not about something doing it externally ;)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: How to clean up TerraSync data

Postby polly » Tue Nov 28, 2023 5:08 pm

Oh, so it's OK to remove files at will, there's no checksum or index or anything to screw up, right ?
User avatar
polly
 
Posts: 956
Joined: Thu Nov 04, 2010 3:45 pm

Re: How to clean up TerraSync data

Postby wkitty42 » Tue Nov 28, 2023 5:29 pm

there are checksums and indexes, yes... AFAIK, they won't be "screwed up" but they may cause the missing files to be downloaded again if one gets too close to their location... but that's going to happen anyway... remember, the server has the last word... if you are within range and files exist on the server, they will be downloaded for use... if you have files locally that do not exist on the server, then they will be removed... the key factor is how is "within range" determined...

this is also why i tried to point out to not just remove files in a directory that match the age criteria but to remove all of the files in that directory if even one of them fits the criteria... even the younger ones... why leave the younger ones if you are not going to fly there anyway? this is also why i thought about looking only at the directory's -mtime to make the decision...

i think the hardest part will be dealing with the two dotted files, .dirhash and .dirindex, when deciding to remove a directory for being "empty" after it has been emptied of files for meeting the age criteria...

EDIT: also, i just had a thought that may complicate this even more...
if you visited a tile 2 years ago, you downloaded all the files available for that tile at that time... let's say that you still fly there all the time but over time new stuff has been added and new files also downloaded... now you have 2 year old files and younger ones... deleting files and directories based on the oldest files in them is not going to work... to do this properly, we needs to build a list of all the files used for each tile from all six scenery directories... then if the youngest file in that tile fits the "too old" age criteria, remove all the files in all six directories for that one tile... but this needs more thought... a simple one-liner won't work...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Next

Return to Support

Who is online

Users browsing this forum: No registered users and 6 guests