Board index FlightGear The FlightGear project

Google code colse there doors

Questions about the FlightGear organisation, website, wiki etc.

Re: Google code colse there doors

Postby KL-666 » Sat Mar 14, 2015 2:30 am

Ok, as i understand it there is 5 TB data storage needed for now and the near future, and bandwidth is unknown? Bandwidth should be known after all those years i think. And necessary to know if some solution is suitable.

Kind regards, Vincent
KL-666
 
Posts: 781
Joined: Sat Jan 19, 2013 2:32 pm

Re: Google code colse there doors

Postby www2 » Sat Mar 14, 2015 2:40 am

Why my idea is we ask the Internet Archive for hosting.
The 5TB is is a ballpoint mark from me base on a post on the mailing list.
And believe that the total size is not more than 200 GB.
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: Google code colse there doors

Postby Bjoern » Sat Mar 14, 2015 3:42 pm

Anything that can be done on the technical side to decrease storage requirements for the world scenery, e.g. by compressing each tile and have FG decompress it during loading and/or runtime?
Bjoern
 
Posts: 484
Joined: Fri Jan 06, 2012 11:00 pm
Location: TXL (RIP)
Version: Next
OS: ArchLinux

Re: Google code colse there doors

Postby Torsten » Sat Mar 14, 2015 9:11 pm

Some hard numbers (Revision 54481)
Code: Select all
900MB      Objects
155MB      Airports
233MB      Models
88382MB    Terrain, 85363MB thereof in compressed files (btg.gz)
overall 1663728 Files in 49620 Directories

Terrasync needs SVN access via HTTP.

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 648
Joined: Fri Feb 01, 2008 10:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Google code colse there doors

Postby Jabberwocky » Sun Mar 15, 2015 2:47 am

TerraSync relies on SVN? Or is that the scenery development part and the productive data is extra?
Jabberwocky
Retired
 
Posts: 1316
Joined: Sat Mar 22, 2014 8:36 pm
Callsign: JWOCKY
Version: 3.0.0
OS: Ubuntu 14.04

Re: Google code colse there doors

Postby zlsa » Sun Mar 15, 2015 2:59 am

TerraSync uses SVN. I can't imagine that there's a way to 'patch in' a different VCS in an older version of FG.
zlsa
 
Posts: 145
Joined: Fri Feb 14, 2014 3:29 am
Callsign: N275A
Version: 3.5 git
OS: Linux

Re: Google code colse there doors

Postby Jabberwocky » Sun Mar 15, 2015 3:00 am

Okay, that's some helpful information
Jabberwocky
Retired
 
Posts: 1316
Joined: Sat Mar 22, 2014 8:36 pm
Callsign: JWOCKY
Version: 3.0.0
OS: Ubuntu 14.04

Re: Google code colse there doors

Postby Hooray » Sun Mar 15, 2015 7:24 am

it's not that difficult - a few years ago, it would simply be using rsync and terrasync would thus be a separate tool.
It's thanks to svn and F-JJTH that things are now much better integrated.
But otherwise, using a different synchronization method (including even just ssh/ftp) would be possible
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Google code colse there doors

Postby hamzaalloush » Mon Mar 16, 2015 12:25 am

yes, it uses SVN, this is why when you dont compile FG with the system-side/local svn or dont include svn libaries you wont get it working in game, found that the hardway.

@Hooray, as an alternative to svn(for whatever reason), we need to use either some protocol native to all platform or provide a new library for static builds.

this is thread jacking now, but Terrasync with svn used to work very well, untill i noticed the "blank tiles" everyone was complaining about some days ago(fg 3.5). i'm behind a national firewall but it used to work prior to this version. just an observation, no one needs to respond to this point so we can keep on thread.
hamzaalloush
 
Posts: 631
Joined: Sat Oct 26, 2013 10:31 am
OS: Windows 10

Re: Google code colse there doors

Postby KL-666 » Mon Mar 16, 2015 12:37 am

From Thorstens numbers i get there is some 100 gigs of data. Make that 200 for the near future. The type of service is not important (cvs), but the bandwith usage absolutely is. That is the sole point where the costs are. So how many people download how much of that 100 gigs dayly/weekly/monthly?

Kind regards, Vincent
KL-666
 
Posts: 781
Joined: Sat Jan 19, 2013 2:32 pm

Re: Google code colse there doors

Postby www2 » Mon Mar 16, 2015 1:37 am

a tile is max 200MB and on average is around 20-90MB say a flight from EHAM to RJBB (Amsterdam to Tokyo) is in the worst case scenario (90MB per tile and no sea or ocean) is 36720 MB and on a average tile size of 3-4 mb is not more than 1080 MB and i estimate 3000 MB for this flight.
As a disclaimer case scenario and average scenario is are calculate with the foaling formula: Distend_in_degree * Tile_size * 3 for a user that have a clean install of a terra sync database (no areas in flight path are download in early sessions.
www2
 
Posts: 319
Joined: Thu Apr 16, 2009 2:58 pm
OS: Ubuntu

Re: Google code colse there doors

Postby wkitty42 » Mon Mar 16, 2015 3:00 am

FWIW: i've seen (and have on my system) tiles retrieved from the google code repo that are both compressed and uncompressed files for the same piece... this may be due to my use of terramaster to try to prefill the areas but i have many directories full of files with the same base name and with both compressed and uncompressed extensions... there's a lot more on the google code repo than just only compressed tile files...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9146
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: Google code colse there doors

Postby IAHM-COL » Thu Mar 26, 2015 8:06 pm

Torsten wrote in Sat Mar 14, 2015 9:11 pm:Some hard numbers (Revision 54481)
Code: Select all
900MB      Objects
155MB      Airports
233MB      Models
88382MB    Terrain, 85363MB thereof in compressed files (btg.gz)
overall 1663728 Files in 49620 Directories

Terrasync needs SVN access via HTTP.

Torsten



Actually, Lesbof got me thinking a bit about this problem. I think a git-repository collection in github may actually solve the problem of google codes abandoning the scene.

I elaborated a proposal here:
viewtopic.php?f=18&t=25314&start=150#p236877

Which I think should be easy to structure, and the code will almost be able to get info already.
Noting that github does offer SVN via HTTP protocol to access the data, and thus terrasync should be able to interact nicely with it.

Best,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Google code colse there doors

Postby elgaton » Thu Mar 26, 2015 11:07 pm

I've read your proposal - it looks nice, but I'm a bit worried about keeping all generated scenery in a big collection of repositories: their number is huge and, since most scenery files are binary, it would be extremely difficult to perform merges in case there's a conflict.

In fact, when a user downloads some scenery from TerraSync, (s)he only needs to get the latest version - the SVN "overlay" just does the equivalent of a version check. (I'm skimming through the code right now to see if the SimGear implementation of SVN also guarantees that each tile is atomically updated). Version control is only needed to keep track of the OSM changesets/X-Plane airport IDs used to perform the build (both OSM and the Gateway allow retrieving old data versions given their ID); the only exception could be custom modifications to landcover/elevation data. This should reduce disk space requirements for the scenery repository.

Therefore, we could adopt an even simpler approach:
  1. improve the existing tools (and add new ones) to support regenerating a single tile without depending on the surrounding ones;
  2. expand the FlightGear Scenery Database webforms, if needed, to allow any FG user to submit improvements that can not be done via the Airport Gateway or OSM;
  3. periodically grab all improvements from OSM/the Airport Gateway via their APIs;
  4. regenerate the single tiles using a distributed system like BOINC (see this page on the wiki);
  5. publish the scenery via HTTP (from a single website or a network of mirrors).
Clients would thus only need to perform an HTTP request that includes an "If-Modified-Since" header to check whether the scenery has changed or not. (Any correctly configured HTTP server would thus serve the updated tile or issue a "Not modified" response). The only caveat is that, if we decide to use a network of mirrors rather than a single site, timestamps on files should be synchronized (otherwise, should the client decide to change the used mirror, it might be tricked into downloading the same scenery again when it's not necessary to do it).

Apart from the fact that some of these goals are clearly medium-to-long term, just replacing the SVN dependency with plain HTTP would remove the requirement for the project to rely on code hosting facilities - a plain, HTTP-only mirror should suffice. A number of University services offer ample disk space (see e.g. the GARR Mirror Network in Italy).
Last edited by Johan G on Fri Mar 27, 2015 12:31 am, edited 1 time in total.
Reason: Fixing broken tags
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1106
Joined: Tue Mar 19, 2013 5:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: Google code colse there doors

Postby IAHM-COL » Thu Mar 26, 2015 11:50 pm

yes. Elgaton
I had not finish reading but wanted to comment on the git size issue and repo download

http://blogs.atlassian.com/2014/05/handle-big-repositories-git/ wrote:Simple solution is a shallow clone

The first solution to a fast clone and to saving developers and systems time and disk space is to perform a shallow clone using git. A shallow clone allows you to clone a repository keeping only the latest n commits of history.

How do you do it? Just use the - -depth option, for example:

git clone --depth depth remote-url


Imagine you accumulated ten or more years of project history in your repository – for example for JIRA we migrated to git an 11 years old code base -, the time savings can add up and be very noticeable.


But more importantly too, the idea outlines the ability to use the "SVN" protocol to interact with the github repos.


____________

Now I read your whole response :D

Well yes. You approach is valid too.
It requires more messing with how the current code interacts with terrasync thou, but it is a valid alternative.

I understand that those scenery tiles are full of binaries, but a few tests I've done show me the situation is not so worrysome

Check this:
https://github.com/IAHM-COL/HKKI-fg-CustomScenery

And see the modification that Wlbragg did for OSM and PAPI light inclusion, on

https://github.com/IAHM-COL/HKKI-fg-Cus ... 7b571efe9b
and
https://github.com/IAHM-COL/HKKI-fg-Cus ... 10/e034s01

It seems to me that the repo handled the situation rather appropriately

Best,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

PreviousNext

Return to The FlightGear project

Who is online

Users browsing this forum: No registered users and 4 guests