Board index FlightGear The FlightGear project

Scenemodels and terrasync are going down

Questions about the FlightGear organisation, website, wiki etc.

Re: Scenemodels and terrasync are going down

Postby hamzaalloush » Tue Nov 17, 2015 6:25 pm

why not create distributed "nodes" all around for anyone and their grandma's old PC to turn into a server? (like repository package mirrors for example), just give us the interface how would you like that scenery provided, what protocols to use etc

edit: i guess we should contact martin to provide us with skeleton webpage/interface and software/distro requirements.
hamzaalloush
 
Posts: 632
Joined: Sat Oct 26, 2013 9:31 am
OS: Windows 10

Re: Scenemodels and terrasync are going down

Postby Hooray » Tue Nov 17, 2015 6:35 pm

funnily, terrasync used to be based on rsync - these days, it is based on svn, which is in part due to the hosting that was provided at no cost IIRC.
What you are talking about would be nice to have, but take tons of time and resources to develop.
Just look at the major disturbance caused by the gitorious shutdown.
Otherwise, some distributed setup - using e.g. torrents- would seem to make sense, but don't easily support versioning.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11349
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenemodels and terrasync are going down

Postby hamzaalloush » Tue Nov 17, 2015 7:44 pm

how do we make decentralized svn distribution? is it possible? for example i know that there are atleast a few people on the forum from where i am by location.

edit: if it were git, i would have an idea, for example as a script that re-bases upon master, or comparatively towards the latest updated server, etc.
hamzaalloush
 
Posts: 632
Joined: Sat Oct 26, 2013 9:31 am
OS: Windows 10

Re: Scenemodels and terrasync are going down

Postby elgaton » Tue Nov 17, 2015 7:56 pm

A sort of distribution is possible: you can create a dump of the whole repository/of some revisions (using "svnadmin dump") and import them on another machine (more or less, it's what git does when performing a push/pull operation).

However, this would require each server to host a complete copy of the World Scenery repository on disk at any time (which has sheer size requirements).
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1107
Joined: Tue Mar 19, 2013 4:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: Scenemodels and terrasync are going down

Postby Hooray » Tue Nov 17, 2015 8:01 pm

Given the mentioned timeframe (end of year, aka 6 weeks to go), it does not make sense to come up with a system from scratch, keep in mind that FG also needs to have client-side support integrated, and that would typically takes at least 2-3 full release cycles.
It is unfortunate enough that FG is stuck with some niche solutions like PUI, Nasal or the upcoming homegrown package manager - there are industry standards that are well-maintained, well-documented and well-supported,

For instance, for multi-platform torrent/p2p-like distributed file synchronization, something like SyncThing (open source) would be interesting: http://www.webupd8.org/2014/06/syncthin ... -sync.html
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11349
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenemodels and terrasync are going down

Postby bugman » Tue Nov 17, 2015 8:05 pm

Well, standard web server mirroring would be the way to go. The svn service would be temporarily deactivated, the svn repository backend files synced from the central server (maybe using rsync over http), and the svn service reactivated, all within a single cron script.

Regards,
Edward
bugman
Moderator
 
Posts: 1709
Joined: Thu Mar 19, 2015 9:01 am
Version: next

Re: Scenemodels and terrasync are going down

Postby jarl.arntzen » Tue Nov 17, 2015 9:23 pm

Hosting on Dropbox an option? http://www.dropboxwiki.com/tips-and-tri ... th-dropbox
This only applies to the scenery though, and will probably only work if TerraSync is based on something like rsync (?)
At least, €120 a year for 100 GB is quite within reach.
Frequent Flyer. Occasional Lander.
jarl.arntzen
 
Posts: 106
Joined: Thu Jan 03, 2013 9:43 pm
IRC name: Jarl Arntzen
Version: 2017.1.1
OS: Ubuntu 14.04

Re: Scenemodels and terrasync are going down

Postby www2 » Tue Nov 17, 2015 10:30 pm

I think we need to contact the Internet Archive [1] for hosting.

A fun note: Internet Archive HQ is near by KSFO and the golden gate bright. ;)

[1] http://archive.org
www2
 
Posts: 260
Joined: Thu Apr 16, 2009 1:58 pm

Re: Scenemodels and terrasync are going down

Postby Hooray » Tue Nov 17, 2015 10:44 pm

Whoevery you are going to contact about this, they will undoubtedly want to know more about the requirements/resources - unless of course they are so massively huge that they simply don't care (think google).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11349
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenemodels and terrasync are going down

Postby www2 » Tue Nov 17, 2015 11:07 pm

Hooray.
Storage is not the problem (21PBite), i don't know is he part about the database or server.
edit:
First 10min of this video show good how crazy (in a good sense) the Internet archive is:
www2
 
Posts: 260
Joined: Thu Apr 16, 2009 1:58 pm

Re: Scenemodels and terrasync are going down

Postby PINTO » Wed Nov 18, 2015 5:30 pm

RE: amount of storage currently in use by (just) the Wayback Machine on the Internet Archive.
From https://archive.org/about/faqs.php#9
As of July 1, 2015 the Internet Archive Wayback Machine contains 23 petabytes of data and is currently growing at a rate of 50-60 terabytes per week. This eclipses the amount of text contained in the world's largest libraries, including the Library of Congress.

From https://archive.org/about/faqs.php#10
A few highlights from the Petabox storage system:
As of December 1, 2014 -
Density: 1.4 petabytes / rack
Power consumption: 3 KW / PetaByte
No Air Conditioning, instead use excess heat to help heat the building.
Raw Numbers as of August 2014:
• 4 data centers, 550 nodes, 20,000 spinning disks
• Wayback Machine: 9.6 petabytes
• Books/Music/Video Collections: 9.8 petabytes
• Unique data: 20 petabytes
• Total used storage: 50 petabytes



Internet Archive policy on uploading content. Not sure if 100% applicable in this situation (but it probably is).
From https://archive.org/about/faqs.php#223 and below:

Is there a limit to what I can upload?
At this time there is no limit. Due to system architecture we recommend that item pages not exceed either 1000files or 25GB.


What are your fees?
At this time we have no fees for uploading and preserving materials. We estimate that permanent storage costs us approximately $2.00US per gigabyte. While there are no fees we always appreciate donations to offset these costs.


How many files and how much data can I upload to a single item?
Because items can "break" we typically recommend that you not exceed 1,000 files and/or 25GB per item page. You can upload very large files, unto 100GB. However, large files that require deriving may take a very long time or even timeout and fail.
Actively developing the MiG-21bis (github repo) (forum thread) (dev discord) (fg wiki)

http://opredflag.com is an active flightgear dogfighting community (using a system that isn’t bombable)
User avatar
PINTO
 
Posts: 945
Joined: Wed Oct 21, 2015 6:28 pm
Callsign: pinto
Version: 2016.3.0
OS: Win10

Re: Scenemodels and terrasync are going down

Postby wkitty42 » Wed Nov 18, 2015 5:36 pm

@alant: IIRC i said something similar quite a while back... IIRC the terrasync svn is updated each night which gives all the files a new svn commit number which means that they all get downloaded over and over again from day to day... i don't know who is managing the system but the repository really should only update those files that actually changed instead of the current method being used...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 5696
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Scenemodels and terrasync are going down

Postby f-ojac » Wed Nov 18, 2015 8:16 pm

Quickly created a short wiki page to explain a bit more the scenery infrastructure. May not be fully complete or true or precise, but still, better that nothing: http://wiki.flightgear.org/Scenery_Infrastructure
Concerning the terrasync update numbers, please see how it's done: http://scenery.flightgear.org/websvn/li ... 3d092e5c91 for instance and yes, there already are proxies to cache this, hopefully grabbing the changes too.
We don't need just storage space, we need much more specialized hosting to install our tools. Read the wiki for more. Don't think we'd be able to launch a terragear scenery construction on the Wayback machines...
--
If you want to support my Terrasync server, hosted on a private server, you can donate here: http://ns334561.ip-5-196-65.eu/WS2.0/WS ... 2.0.1.html
f-ojac
 
Posts: 1276
Joined: Fri Mar 07, 2008 9:50 am
Version: GIT
OS: GNU/Linux

Re: Scenemodels and terrasync are going down

Postby HaydenRead » Sun Dec 06, 2015 10:22 pm

Just wondering how this is going. Has new hosting been found? Is there anywhere that donations to the cost of hosting can be made?
HaydenRead
 
Posts: 1
Joined: Sun Dec 06, 2015 10:16 pm

Re: Scenemodels and terrasync are going down

Postby Hooray » Sun Dec 06, 2015 10:27 pm

HaydenRead wrote in Sun Dec 06, 2015 10:22 pm:Just wondering how this is going. Has new hosting been found? Is there anywhere that donations to the cost of hosting can be made?


You could try this, Curt is the only one likely to have access to the TLD, and the corresponding mail accounts:
Subject: FlightGear at LinuxTag and FSWeekend need your help
Torsten wrote:please consider donating a few Euros, Dollars, Pounds, Crowns, Yen, Francs, Afghanis, Dinars, Pesos or whatever your currency might be to

donations@flightgear.org using PayPal.

Donations are not tax deductable yet - this
might change at some time in the future.

Thanks for your support and for making FlightGear the best free flight simulator on earth.


Personally, I am still not convinced though ... it should be much more straightforward/promising to get in touch with a few hosting companies that are known to support OSS projects, without anybody having to raise $3k ...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11349
Joined: Tue Mar 25, 2008 8:40 am

PreviousNext

Return to The FlightGear project

Who is online

Users browsing this forum: No registered users and 1 guest