Scenemodels and terrasync are going down

[hamzaalloush]

why not create distributed "nodes" all around for anyone and their grandma's old PC to turn into a server? (like repository package mirrors for example), just give us the interface how would you like that scenery provided, what protocols to use etc

edit: i guess we should contact martin to provide us with skeleton webpage/interface and software/distro requirements.

[Hooray]

funnily, terrasync used to be based on rsync - these days, it is based on svn, which is in part due to the hosting that was provided at no cost IIRC.
What you are talking about would be nice to have, but take tons of time and resources to develop.
Just look at the major disturbance caused by the gitorious shutdown.
Otherwise, some distributed setup - using e.g. torrents- would seem to make sense, but don't easily support versioning.

[hamzaalloush]

how do we make decentralized svn distribution? is it possible? for example i know that there are atleast a few people on the forum from where i am by location.

edit: if it were git, i would have an idea, for example as a script that re-bases upon master, or comparatively towards the latest updated server, etc.

[elgaton]

A sort of distribution is possible: you can create a dump of the whole repository/of some revisions (using "svnadmin dump") and import them on another machine (more or less, it's what git does when performing a push/pull operation).

However, this would require each server to host a complete copy of the World Scenery repository on disk at any time (which has sheer size requirements).

[Hooray]

Given the mentioned timeframe (end of year, aka 6 weeks to go), it does not make sense to come up with a system from scratch, keep in mind that FG also needs to have client-side support integrated, and that would typically takes at least 2-3 full release cycles.
It is unfortunate enough that FG is stuck with some niche solutions like PUI, Nasal or the upcoming homegrown package manager - there are industry standards that are well-maintained, well-documented and well-supported,

For instance, for multi-platform torrent/p2p-like distributed file synchronization, something like SyncThing (open source) would be interesting: http://www.webupd8.org/2014/06/syncthin ... -sync.html

[bugman]

Well, standard web server mirroring would be the way to go. The svn service would be temporarily deactivated, the svn repository backend files synced from the central server (maybe using rsync over http), and the svn service reactivated, all within a single cron script.

Regards,
Edward

[jarl.arntzen]

Hosting on Dropbox an option? http://www.dropboxwiki.com/tips-and-tri ... th-dropbox
This only applies to the scenery though, and will probably only work if TerraSync is based on something like rsync (?)
At least, €120 a year for 100 GB is quite within reach.

[www2]

I think we need to contact the Internet Archive [1] for hosting.

A fun note: Internet Archive HQ is near by KSFO and the golden gate bright.

[1] http://archive.org

[Hooray]

Whoevery you are going to contact about this, they will undoubtedly want to know more about the requirements/resources - unless of course they are so massively huge that they simply don't care (think google).

[www2]

Hooray.
Storage is not the problem (21PBite), i don't know is he part about the database or server.
edit:
First 10min of this video show good how crazy (in a good sense) the Internet archive is:

[PINTO]

RE: amount of storage currently in use by (just) the Wayback Machine on the Internet Archive.
From https://archive.org/about/faqs.php#9

As of July 1, 2015 the Internet Archive Wayback Machine contains 23 petabytes of data and is currently growing at a rate of 50-60 terabytes per week. This eclipses the amount of text contained in the world's largest libraries, including the Library of Congress.

From https://archive.org/about/faqs.php#10

A few highlights from the Petabox storage system:
As of December 1, 2014 -
Density: 1.4 petabytes / rack
Power consumption: 3 KW / PetaByte
No Air Conditioning, instead use excess heat to help heat the building.
Raw Numbers as of August 2014:
• 4 data centers, 550 nodes, 20,000 spinning disks
• Wayback Machine: 9.6 petabytes
• Books/Music/Video Collections: 9.8 petabytes
• Unique data: 20 petabytes
• Total used storage: 50 petabytes

Internet Archive policy on uploading content. Not sure if 100% applicable in this situation (but it probably is).
From https://archive.org/about/faqs.php#223 and below:

Is there a limit to what I can upload?
At this time there is no limit. Due to system architecture we recommend that item pages not exceed either 1000files or 25GB.

What are your fees?
At this time we have no fees for uploading and preserving materials. We estimate that permanent storage costs us approximately $2.00US per gigabyte. While there are no fees we always appreciate donations to offset these costs.

How many files and how much data can I upload to a single item?
Because items can "break" we typically recommend that you not exceed 1,000 files and/or 25GB per item page. You can upload very large files, unto 100GB. However, large files that require deriving may take a very long time or even timeout and fail.

[wkitty42]

@alant: IIRC i said something similar quite a while back... IIRC the terrasync svn is updated each night which gives all the files a new svn commit number which means that they all get downloaded over and over again from day to day... i don't know who is managing the system but the repository really should only update those files that actually changed instead of the current method being used...

[f-ojac]

Quickly created a short wiki page to explain a bit more the scenery infrastructure. May not be fully complete or true or precise, but still, better that nothing: http://wiki.flightgear.org/Scenery_Infrastructure
Concerning the terrasync update numbers, please see how it's done: http://scenery.flightgear.org/websvn/li ... 3d092e5c91 for instance and yes, there already are proxies to cache this, hopefully grabbing the changes too.
We don't need just storage space, we need much more specialized hosting to install our tools. Read the wiki for more. Don't think we'd be able to launch a terragear scenery construction on the Wayback machines...

[HaydenRead]

Just wondering how this is going. Has new hosting been found? Is there anywhere that donations to the cost of hosting can be made?

[Hooray]

Just wondering how this is going. Has new hosting been found? Is there anywhere that donations to the cost of hosting can be made?

You could try this, Curt is the only one likely to have access to the TLD, and the corresponding mail accounts:
Subject: FlightGear at LinuxTag and FSWeekend need your help

please consider donating a few Euros, Dollars, Pounds, Crowns, Yen, Francs, Afghanis, Dinars, Pesos or whatever your currency might be to

donations@flightgear.org using PayPal.

Donations are not tax deductable yet - this
might change at some time in the future.

Thanks for your support and for making FlightGear the best free flight simulator on earth.

Personally, I am still not convinced though ... it should be much more straightforward/promising to get in touch with a few hosting companies that are known to support OSS projects, without anybody having to raise $3k ...