Board index FlightGear Support Flying

Poll: Terrasync (dynamic loading of terrain) works?

Controlling your aircraft, using the autopilot etc.

See the post for the different issues

Issue 1
10
13%
Issue 2
1
1%
Issue 3
1
1%
Issue 4
2
3%
Issue 5
3
4%
Issue 6
8
10%
Issue 7
8
10%
Issue 8
0
No votes
Conclusion 1
10
13%
Conclusion 2
5
6%
Conclusion 3
4
5%
Tool 1
17
22%
Tool 2
3
4%
Tool 3
4
5%
Tool 4
3
4%
 
Total votes : 79

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Thorsten » Fri May 19, 2017 3:15 pm

- If two computers on the same network load/use scenery at different speeds consistently, and both have gigabit ethernet to the router, what explains the difference?


If that's true and thoroughly tested, it begs an explanation what's wrong with this particular system (note that I can't reproduce the claim - I get consistently the same speed for Terrasync and Terramaster on three different computers, neither can any of the people who develop TS apparently, so it's unlikely to be the fault of the FG binary and far more likely a local config issue).

Note also that we've had for instance a rock-solid claim in the forum that dds textures double framerate compared with png - claimed to be consistend and thoroughly tested. When I investigated and suggested benchmark tests the whole claim dissolved and the framerates were the same. So I tend to take spectacular reports with a grain of salt.

Note also that different people report different problems, and while the explanation for some may be exaggerated expectations what the internet connection can do, that's unlikely to account for everything.
Thorsten
 
Posts: 7933
Joined: Mon Nov 02, 2009 8:33 am

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby wkitty42 » Fri May 19, 2017 8:01 pm

Lydiot wrote in Fri May 19, 2017 2:50 pm:- If two computers on the same network load/use scenery at different speeds consistently, and both have gigabit ethernet to the router, what explains the difference?

OS load, different configurations, unwanted software installed, etc... the speed of your internal (gigabit to the router) is one tiny part of the equation... one of those machines may only have half-duplex to the router... there may be a bad port on the router... the machine's interrupts used to signal to the NIC may be shared and overused with other devices in the machine... the cable may be running too near and parallel to electrical cables... on wifi? yeah, i can easily block that just by turning on my microwave oven or doing some welding in the shop...

Lydiot wrote in Fri May 19, 2017 2:50 pm:- Does terramaster pull scenery from the same location as terrasync? If so, why is there a difference?

technically, yes... but there is not just one machine serving the scenery... there are several... which one you get to use at any point in time is determined by the initial DNS query to find one for you to use... then there's the server itself... some may be behind a load balancer pulling the information from numerous other machines behind them... some servers use bandwidth limiters and will restrict what you can pull as their load increases or possibly even based on how much you pull from them...

there is no "one size fits all" in today's internet and there won't ever be... especially if net neutrality goes away and providers start limiting traffic from certain services because they want to... ""you guys"" are acting like it is one highway and everyone has the same vehicle with the same capabilities but you're forgetting all the other factors and what happens when that highway gets full... one need only look at cable feeds and watch how they slow down when all the kids get home from school and overload those cable hubs that feed each neighborhood... has anyone ever heard of the term "oversold"? many ISPs do exactly that and add more infrastructure at some later date (if you're lucky)...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 2834
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.4

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby wkitty42 » Fri May 19, 2017 8:14 pm

Necolatis wrote in Wed May 17, 2017 10:05 pm:Issue 6:
When starting flightgear a place I already have download terrain for, terrasync still loads alot (/take a long time).

FWIW: terrasync still has to validate the files already downloaded on your system so it knows if it needs to download updates or not... there can easily be a huge lot of files to validate and calculate checksums for... at least Airports and Models are checked at startup every time along with the particular tile that you are loading on...

    Airports contains 95.0MiB (99617904 bytes) consisting of 10592 folders and 56134 files.
    Models contains 341.4Mib (357992615 bytes) consisting of 25 folders and 4113 files.

checking ~60000 files takes time no matter how fast your machine and storage is...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 2834
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.4

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Lydiot » Sun May 21, 2017 2:54 pm

wkitty42 wrote in Fri May 19, 2017 8:01 pm:
Lydiot wrote in Fri May 19, 2017 2:50 pm:- If two computers on the same network load/use scenery at different speeds consistently, and both have gigabit ethernet to the router, what explains the difference?

OS load, different configurations, unwanted software installed, etc... the speed of your internal (gigabit to the router) is one tiny part of the equation... one of those machines may only have half-duplex to the router... there may be a bad port on the router... the machine's interrupts used to signal to the NIC may be shared and overused with other devices in the machine... the cable may be running too near and parallel to electrical cables... on wifi? yeah, i can easily block that just by turning on my microwave oven or doing some welding in the shop...


Right. So not just "how the internet works" then.
Lydiot
 
Posts: 888
Joined: Tue Oct 22, 2013 10:50 pm

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Thorsten » Sun May 21, 2017 4:38 pm

Right. So not just "how the internet works" then.


Not sure what your native language is, but:

I'm frankly not sure everyone knows how the internet works. implies that the majority of people addressed knows how it works, but there are _probably_ exceptions.

What you apparently understood would have been expressed as 'Seems like nobody understands how the internet works because that explains everything.' But... nobody said that. Semantics is tricky...

(Also note the clarification of what I meant Note also that different people report different problems, and while the explanation for some may be exaggerated expectations what the internet connection can do, that's unlikely to account for everything.)

If you insist in just not reading what others write and are offended by explanations not even directed at you, don't post in threads discussing problems.
Thorsten
 
Posts: 7933
Joined: Mon Nov 02, 2009 8:33 am

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Lydiot » Sun May 21, 2017 5:59 pm

Hey buddy, why don't you leave the moderation to moderators, ok.... And why don't you start including the name of the person you're quoting to make it easier for people to read. Oh, right, you don't care about making it easier for others.

Speaking of, English is your native language?
Lydiot
 
Posts: 888
Joined: Tue Oct 22, 2013 10:50 pm

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Lydiot » Sun May 21, 2017 6:01 pm

"I'm frankly not sure what to make of your attitude."

I'm getting more certain by the post.
Lydiot
 
Posts: 888
Joined: Tue Oct 22, 2013 10:50 pm

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Thorsten » Sun May 21, 2017 6:09 pm

I don't know what your problem is, but please solve it elsewhere. This thread is about terrasync not your attitude problems.
Thorsten
 
Posts: 7933
Joined: Mon Nov 02, 2009 8:33 am

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby wkitty42 » Sun May 21, 2017 6:40 pm

Lydiot wrote in Sun May 21, 2017 2:54 pm:
wkitty42 wrote in Fri May 19, 2017 8:01 pm:OS load, different configurations, unwanted software installed, etc... the speed of your internal (gigabit to the router) is one tiny part of the equation... one of those machines may only have half-duplex to the router... there may be a bad port on the router... the machine's interrupts used to signal to the NIC may be shared and overused with other devices in the machine... the cable may be running too near and parallel to electrical cables... on wifi? yeah, i can easily block that just by turning on my microwave oven or doing some welding in the shop...


Right. So not just "how the internet works" then.

actually, yes, it is... everything i wrote can be applied to the routers used on the internet...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 2834
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.4

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby wkitty42 » Sun May 21, 2017 6:45 pm

Necolatis wrote in Wed May 17, 2017 10:05 pm:Tool 1:
I use Terrasync.

Tool 2:
I use TerraMaster.

Tool 3:
I use TerraGit.

Tool 4:
Offline: I download/buy my scenery or only fly around default airport.
Or use custom scenery.

you are missing at least two other tools... terrasync.py and terrafs... the first one is similar to terramaster but from the command line... the second is actually mounting the terrasync server as a remote drive... if your internet is fast enough, you can use it directly without downloading anything... otherwise, just copy the files using standard copy techniques...

i use terrasync, terramaster and terrasync.py all together...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 2834
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.4

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Lydiot » Sun May 21, 2017 11:36 pm

wkitty42 wrote in Sun May 21, 2017 6:40 pm:
Lydiot wrote in Sun May 21, 2017 2:54 pm:Right. So not just "how the internet works" then.

actually, yes, it is... everything i wrote can be applied to the routers used on the internet...


Ah, so all problems experienced with Terrasync are due to the internet, nothing local. I have no idea how one can be that sure about it, but, 'got it'.
Lydiot
 
Posts: 888
Joined: Tue Oct 22, 2013 10:50 pm

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Thorsten » Mon May 22, 2017 5:11 am

Ah, so all problems experienced with Terrasync are due to the internet, nothing local. I have no idea how one can be that sure about it, but, 'got it'.


Are you doing this deliberately, or do you really have trouble understanding English?
Thorsten
 
Posts: 7933
Joined: Mon Nov 02, 2009 8:33 am

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Torsten » Mon May 22, 2017 8:58 pm

Let me add some facts about the terrasync infrastructure:
We currently have three "active" terrasync mirrors used by clients and one "master" server as a synchronization source.
The master is hosted at SourceForge, two of the active mirrors are hosted at dreamhost and one mirror is on a privately owned server with a fast uplink.

We try to keep SourceForge clear of client requests because we already use their systems quite heavily and we try not to abuse their excellent free services. Also, they throttle bandwidth for files bigger than a certain size.

Dreamhost provides "unlimited server space and traffic" under a paid plan but reserve the right to throttle bandwidth when they detect heavy load. Note: I have never exactly noticed a service limitation but that does not mean it has not happened yet.
I believe the privately donated server space is free of limitations.

That being said, limitations due to bandwidth throttling may easily take place at any time. Anybody willing to provide a fast webspace, please get in touch with me (but make sure to read to the end of this post).

Most of the times, the http client does not have a chance to fully leverage your uplink speed. Scenery consist of many relatively small files of a few hundret to a few kilobytes. Transferring those over http is expensive and takes time. We already use http pipelining and parallel connections to somewhat compensate for that but it's not perfect.

I also have the impression that the issue is more prevailing on Windows than it is on OSX or Linux. I'd really love to have someone on Windows doing some Wireshark monitoring to check if pipelining over multiple connections actually work with libcurl on Windows.

To get rid of that "small file" problem, we are thinking about packing folders but that requires serious refactoring of the client's code. Nothing to be expected anytime soon.

And for those who like numbers, here are some for one of the three active mirrors in April 2017:
182,322,022 requests served
130TB of data transferred
The busiest hour saw 9,581,665 requests within the period of 60 minutes

Our TS client distributes the load evenly across all the three mirrors. It should be safe to tripple the those numbers to get an idea of how much data we distribute to our valued clients.

I hope this clarifies a bit why it is anything but easy to track down speed issues with Terrasync.

Torsten
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 561
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Torsten » Mon May 22, 2017 9:47 pm

And for the vote: I checked "terrasync works perfectly for me" but my environment might be a little bit non-standard.
I access the master's master-copy of terrasync via an otherwise mostly unused Apache web server on the local gigabit LAN without any routers between my FlightGear instance and the web server. That's one of the few benefits of hosting and running the scenery database :-)
flightgear.org - where development happens.
User avatar
Torsten
 
Posts: 561
Joined: Fri Feb 01, 2008 9:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: Poll: Terrasync (dynamic loading of terrain) works?

Postby Necolatis » Mon May 22, 2017 10:12 pm

Thank you for that insight Torsten.

I have not used Wireshark before, but I will try it now to test out Terrasync.
Anything specific I should be looking for? or just that terrasync connects over multiple connections?

I can also do some logging of it, if I get some instructions on how to log terrasync. (I have FG build environment setup, if custom FG needed)
"Airplane travel is nature's way of making you look like your passport photo."
— Al Gore


Hangar: https://sites.google.com/site/fghangar/
User avatar
Necolatis
 
Posts: 1631
Joined: Mon Oct 29, 2012 12:40 am
Location: EKOD
Callsign: Leto
IRC name: Neco
Version: 2016.4.3
OS: Windows 10

PreviousNext

Return to Flying

Who is online

Users browsing this forum: No registered users and 2 guests