Board index FlightGear The FlightGear project

Scenemodels and terrasync are going down

Questions about the FlightGear organisation, website, wiki etc.

Scenemodels and terrasync are going down

Postby Torsten » Mon Nov 09, 2015 5:13 pm

Hi!

By the end of this year (2015), our current hosting of
- the scenemodels database
- the scenemodels website
- the terrasync repository
will no longer be available for use by the FlightGear project.

As a result, we need a new home for those services.
The technical requirements are roughly
- a postgresql database (with postgis extension) of approx. 40GB and growing (scenemodels db)
- apache webserver, php, access to the database and the usual toolkit for hosting the scenemodels website
- a svn server with 140GB+ for the terrasync scenery
- enough bandwith to serve the data
- One ore more reverse proxy serving terrasync requests (not much storage required, just bandwidth)

Is anybody out there able to provide a drop-in replacement?
Any serious idea or offer to help is welcome.

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 637
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Scenemodels and terrasync are going down

Postby legoboyvdlp » Mon Nov 09, 2015 7:32 pm

Dear Torsten.
I am not sure how these proposals will be considered.
However, Israel Hernandez is offereing his suggestion.
I will not support him being 'in charge' of it; however, this idea seems quite good.... if done properly.
Consider it.
Don't reject it JUST based on dislike of the author.
IAHM-COL wrote:Do you mind relaying me again
here: viewtopic.php?f=42&t=27948

Subject: Good Bye Terrasync?!

IAHM-COL wrote:Dear Torsten

I am a bit on shock with the news of terrasync reaching end of life cycle

Thanks for letting us know thou.


I wanted to communicate to you that a few months ago I have been suggesting the possibility of changing the way that scenery is fetched by FG. Specifically, I see a need of making some changes in the Terrain fetching to allow a modular approach.

The idea takes several fronts, which can be summarized as

1) Scenery becomes modular
2) change SVN to a git method
3) host the modules as 10x10 degrees quads on gitlab or github
4) This also allows making the scenery distributed (as opposed to centralized) making it viable to people to submit patches with newly installed buildings, or any other configurable xml (think of groundnets, jetways, others) via pull requests.
5) if a scenery construction server is established with standardized landclasses and compiled altitudes are used (gdalchop and ogrdecode already pre-performed), then, even we could speed installation of new layouts by local builds of small scenery regions. (as oppose to need for a long wait for new airport layouts and layout updates)

I would love that this idea can be given some serious consideration.

Best,
Israel Hernandez

(send privately because I can't post on the devel.list. Also, attempted to send at the devel list and posted to http://www.thejabberwocky.net forum)

User avatar
legoboyvdlp
 
Posts: 7090
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Scenemodels and terrasync are going down

Postby legoboyvdlp » Mon Nov 09, 2015 7:33 pm

Is this related to Google Code shutting down; i.e. can you not find a replacement?
User avatar
legoboyvdlp
 
Posts: 7090
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Scenemodels and terrasync are going down

Postby MIG29pilot » Mon Nov 09, 2015 7:40 pm

Would that plan be compatible with the current terrasynce setup?
User avatar
MIG29pilot
 
Posts: 1454
Joined: Tue May 19, 2015 4:03 pm
Location: 6 feet under Snow
Callsign: MIG29pilot
Version: 3.7nightly
OS: Windows 10

Re: Scenemodels and terrasync are going down

Postby elgaton » Mon Nov 09, 2015 8:15 pm

A core code change would be needed (TerraSync supports only SVN and rsync).
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1107
Joined: Tue Mar 19, 2013 4:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: Scenemodels and terrasync are going down

Postby Torsten » Mon Nov 09, 2015 8:19 pm

I can't see how Israels suggestion helps.
1) Scenery becomes modular

Our scenery is alread modular

2) change SVN to a git method

SVN is just the distribution method, no history accumulates on the client because terrasync uses a striped down svn client.
Does anybody seriously considers keeping more than 50.000 revisions of scenery locally?
Also, you are no longer backward compatible.

3) host the modules as 10x10 degrees quads on gitlab or github

I consider this an abuse of gitXXX. At least github asks to keep repositories under 1GB, creating hundrets or repositories for a single project as a data store does not make it better just because it is not explicitely forbidden.

4) This also allows making the scenery distributed (as opposed to centralized) making it viable to people to submit patches with newly installed buildings, or any other configurable xml (think of groundnets, jetways, others) via pull requests.

We have a way to use distributed scenery by providing reverse proxies. People can submit updates to scenery models to the scenemodel db using the web interface. Those updates get reviewed by a human and applied. Using git patches for the workflow might be possible but does not help in our situation.

5) if a scenery construction server is established with standardized landclasses and compiled altitudes are used (gdalchop and ogrdecode already pre-performed), then, even we could speed installation of new layouts by local builds of small scenery regions. (as oppose to need for a long wait for new airport layouts and layout updates)

As far as my understanding of the scenery building process goes, this is not as easy as it looks. Landcover triangles tend to cross 10x10deg areas unless you are on a small island. Those triangles need to be sliced along the boundary, the vertices marking the edge have to be on the exact same position after slicing.
While you can easily generate scenery for a single tile, it might look wrong at the edges.
Sure, a rolling-release of scenery is desirable. Unfortunately the tool chain is not stable enough to provide such a service.

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 637
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Scenemodels and terrasync are going down

Postby elgaton » Mon Nov 09, 2015 8:49 pm

Since FlightGear is used at some universities/research institutions, could one of those sponsor/host the main servers? Also, research networks (such as GARR here in Italy) provide free mirroring service for open source projects (they might support only HTTP, though).
NIATCA 2nd admin, regular ATC at LIPX and creator of the LIPX custom scenery
elgaton
 
Posts: 1107
Joined: Tue Mar 19, 2013 4:58 pm
Callsign: I-ELGA/LIPX_TW
Version: Git
OS: Windows + Arch Linux

Re: Scenemodels and terrasync are going down

Postby legoboyvdlp » Tue Nov 10, 2015 1:34 am

Torsten wrote in Mon Nov 09, 2015 8:19 pm:I can't see how Israels suggestion helps.
1) Scenery becomes modular

Our scenery is alread modular

2) change SVN to a git method

SVN is just the distribution method, no history accumulates on the client because terrasync uses a striped down svn client.
Does anybody seriously considers keeping more than 50.000 revisions of scenery locally?
Also, you are no longer backward compatible.

3) host the modules as 10x10 degrees quads on gitlab or github

I consider this an abuse of gitXXX. At least github asks to keep repositories under 1GB, creating hundrets or repositories for a single project as a data store does not make it better just because it is not explicitely forbidden.

4) This also allows making the scenery distributed (as opposed to centralized) making it viable to people to submit patches with newly installed buildings, or any other configurable xml (think of groundnets, jetways, others) via pull requests.

We have a way to use distributed scenery by providing reverse proxies. People can submit updates to scenery models to the scenemodel db using the web interface. Those updates get reviewed by a human and applied. Using git patches for the workflow might be possible but does not help in our situation.

5) if a scenery construction server is established with standardized landclasses and compiled altitudes are used (gdalchop and ogrdecode already pre-performed), then, even we could speed installation of new layouts by local builds of small scenery regions. (as oppose to need for a long wait for new airport layouts and layout updates)

As far as my understanding of the scenery building process goes, this is not as easy as it looks. Landcover triangles tend to cross 10x10deg areas unless you are on a small island. Those triangles need to be sliced along the boundary, the vertices marking the edge have to be on the exact same position after slicing.
While you can easily generate scenery for a single tile, it might look wrong at the edges.
Sure, a rolling-release of scenery is desirable. Unfortunately the tool chain is not stable enough to provide such a service.

Torsten

Your points are certainly valid.
I will be relaying this on.
Best,
Jonathan
User avatar
legoboyvdlp
 
Posts: 7090
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Scenemodels and terrasync are going down

Postby KL-666 » Tue Nov 10, 2015 2:12 am

What is exactly the "deal" with the current servers? Are they some sort of free google servers, or private paid servers? And what was the complaint of the server providers?

Kind regards, Vincent
KL-666
 
Posts: 784
Joined: Sat Jan 19, 2013 1:32 pm

Re: Scenemodels and terrasync are going down

Postby legoboyvdlp » Tue Nov 10, 2015 1:37 pm

IAHM-COL wrote:Hi Thorsten.

I can't see how Israels suggestion helps.


Thanks for your rebutal on viewtopic.php?f=42&t=27948#p263671

Hopefully that is a path for me to explain my points of view. And hopefully you can consider the situation with a bit openmind.

Our scenery is alread modular


Well. Yes and No. The scenery itself is not modular. You distribute parts of it and compile it in a non modular set.
This would requires lots of confusing paragraphs to explain. So I'll go with 1 image == 1000 words thing here

This is how FG scenery looks like, and the underlying structure coded to be understood by the core

Image

Now. When I mean modular, I mean that inside the Flightgear scenery directory, the content could be not the scenery directly, but a collection of Modules. Each directory inside could be a complete scenery package for a much much more small scenery package (a tile of 10x10 degrees, or a custom scenery, per example) Each of this directories are seen below with the letter M indicating they are actually scenery modules. Very similar as how the Aircraft directory actually contains subdirectories, each of them being one aircraft.

Image

So, although you say FG Scenery is modular, it is not as modular as it could be convenient in order to split it effectively in smaller more manageable chunks (with ease).


Does anybody seriously considers keeping more than 50.000 revisions of scenery locally


well. First of all, that is something to be decided individually. It is not to the core developers to decide for all users how much of the total revisions we can be willing or capable to have. But more importantly, 1) a new refactoring of scenery can start again from "initial commit" (revision 1 if you which). Past is past now. 2) secondly, if every Module as shown above where separated, each of them will have a completely different "history". Frankly, I doubt everyone of them will accumulate 50000 revisions. that will mean our world got REALLY REALLY complete. and thus at the end a reason to celebrate.
3) Git has options to have a shallow clone, which allow reducing the depth cloned at a given time to manage this situation. and 4) More importantly, if we were to use github, an option to fetch the git sceneries via SVN exist.


Also, you are no longer backward compatible.


Yes and no.

In fact, each individual module is (would be) backward compatible. We do it all the time when we create custom sceneries and point flightgear there as --fg-scenery.

I consider this an abuse of gitXXX. At least github asks to keep repositories under 1GB


Indeed I found this to be concerning before. Firstly, Gitlab had grown its individual repository size allowance to 10 G. And Github allows 1G.
As far as I am informed no single 10x10 degree tile in our whole world scenery approaches the 1G limit. Not even remotely.

A few months back I actually addressed github with the exact concern of having lets say 500 additional repos that exceeded 100 GB of content. As you are aware I have hosted the FG aircraft. a total of 700 repos counting for about 25 GB of data, and no concern by Github have been raised, but I worried about scenery being too large of a package. The answer by github was

Image

Which indicated to me that hosting such scenery, if no single repo exceeds 1GB was not a violation of their TOS, and they would be calm about it.

Keep in mind that in terms of hosting, it is better to have 600 repos of half a gig each than to have 1 repo of 200 G total. Modularity pays off a great deal with computing.

We have a way to use distributed scenery by providing reverse proxies. People can submit updates to scenery models to the scenemodel db using the web interface. Those updates get reviewed by a human and applied. Using git patches for the workflow might be possible but does not help in our situation.


I am aware of it. I used it. And it has been O.K.
But I am certain that the possibility of submitting pull requests with scenery improvements will be an step forward from this situation. It gives lots of much flexibility. It also allows for human revisions (git pull requests are not forced writes!).
Furthermore, it will allow us to submit pull request for changes like new groundnets, new tower.xml or jetways.xml configurations, in addition to just add new models. It gives us major power in customizing the scenery. It allows us also to submit scenery that is currently unsubmittable via the db, such as OSM2City outputs.

As far as my understanding of the scenery building process goes, this is not as easy as it looks


Granted it is not. But it is my opinion that enhancing the ability to patch scenery with new layouts "on the fly" is a clever way to go (objective to aim).
And I understand work is being done in this direction.
but, modularizing the scenery will be a major advantage is being able to modify smaller tile sizes as well. Maybe reconstructing a 1x1 degree as needed and submitting the commit via Pull Request, which would be modifying only a 10x10 degree tile scenery repo -- not a whole world repo.

Just food for thought.


Thanks a lot for your attention,

IH-COL


@Thorsten with an H, sorry for the misspelling; it is bot mine but his!
User avatar
legoboyvdlp
 
Posts: 7090
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Scenemodels and terrasync are going down

Postby f-ojac » Tue Nov 17, 2015 1:33 pm

I own a private server with enough space and unlimited bandwidth and I'm OK to share it for FG use. Only drawbacks : I don't have much time to manage the whole stuff and there's no backup on the site.
--
If you want to support my Terrasync server, hosted on a private server, you can donate here: http://ns334561.ip-5-196-65.eu/WS2.0/WS ... 2.0.1.html
f-ojac
 
Posts: 1276
Joined: Fri Mar 07, 2008 9:50 am
Version: GIT
OS: GNU/Linux

Re: Scenemodels and terrasync are going down

Postby Hooray » Tue Nov 17, 2015 2:23 pm

I didn't read everything, but I would suggest to create a wiki article detailing the technical requirements, so that we can separately get in touch with different hosting companies and see if we can work out a deal.

Apart from that, I do have access to a dedicated 16 core root server with 32 gb of RAM and 1tb of RAID5 space and 30 tb of monthly bandwidth, but I would also want to know up-front, if/how this is going to be affected by running fg related stuff on it - for example, fgms is said to be highly inefficient and eat up to 300 gb of bandwidth per day.

I am not using fgms (or terrasync) myself, but I guess that anybody considering to "donate" or "sponsor" resources to the FG project, would like to know what they are facing.

So please consider teaming up to create a corresponding wiki article that covers 1) which services are to be executed, 2) what degree of access/permissions is required, 3) what is realistically required in terms of resources/horsepower (CPU, RAM, bandwidth).

I think we would be much more likely to come up with a good offer/solution if we state the facts upfront.

My suggestion would be that a handful of us get in touch with the main companies listed at: https://en.wikipedia.org/wiki/Compariso ... facilities

And forward a link to the wiki article asking if they'd be willing to host the corresponding infrastructure.

And if so, a spokesperson should be nominated (Torsten?) so that they have someone to get in touch with, should they not be intimidated by the requirements :D

In the meantime, my suggestion would be to update the wiki/website accordingly, to make this much more prominent - and whoever was interviewed by sourceforge for the "project of the month" interview, should probably send out an eMail to them to also augment the interview accordingly and mention that FG needs new hosting by the end of the year (I finished updating the wiki accordingly).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11340
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenemodels and terrasync are going down

Postby legoboyvdlp » Tue Nov 17, 2015 3:46 pm

What is sad is that TerraSync is more reliable than ever.
Before, it kept going dead, and was aptly named TerraSuck by my friend, JWOCKY.
It went HTTP 0, which is quite cryptic....
But it has been fairly reliable these past few weeks, and it is sad that these servers are being closed.
User avatar
legoboyvdlp
 
Posts: 7090
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: Scenemodels and terrasync are going down

Postby Hooray » Tue Nov 17, 2015 4:45 pm

while it may be "sad" to those using it, it may also be understandable once we learn more about the resouce requirements (bandwidth, space, cpu, ram) - which is why I am hoping that someone in the know will chime in and tell us a little more about that
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11340
Joined: Tue Mar 25, 2008 8:40 am

Re: Scenemodels and terrasync are going down

Postby Alant » Tue Nov 17, 2015 5:41 pm

To reduce uneccesary Terrrasync downloads could this be investigated?
Each day, the first time that I start FG there is a long pause whilst "scenery is being downloaded". This delay does not happen again until the next day.
My suspicion is that yesterday´s Terrasync is considered old, even if there has been no change.

Alan
Alant
 
Posts: 913
Joined: Wed Jun 23, 2010 5:58 am
Location: Portugal
Callsign: Tarnish99
Version: from Git
OS: Windows 10

Next

Return to The FlightGear project

Who is online

Users browsing this forum: No registered users and 1 guest