Board index FlightGear Support Installation Linux

FGdata downloading error  Topic is solved

Installing FlightGear, scenery, aircraft etc. on Linux systems like Ubuntu.

FGdata downloading error

Postby rleibner » Thu Jan 11, 2018 3:31 pm

Hi all,

I'm trying to update SimGear, FlightGear and FGData from 'next' using
Code: Select all
$ ../download_and_compile.sh -p n -j 4
the compilation seems ok (I didn't saw any error msg) but when downloading FGFS DATA (arriving next to 30%) the process stops with
Code: Select all
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
Yesterday I tried it at least 5 times, always with a similar (and frustrating) result. I blame my poor internet connexion.

Is there any way to download DATA in batches (so I would not lose the downloaded KBs in each attempt)?
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby rominet » Thu Jan 11, 2018 4:05 pm

Hmm, not cool. If you run the command:
Code: Select all
git clone https://git.code.sf.net/p/flightgear/fgdata

several times in the same directory, does it restart from scratch every time?

Edit: this probably won't work, sorry (according to this stackoverflow page).

Otherwise, Parnikkapore used to seed Torrents for FG. If they contained even a non-up-to-date FGData with all the Git metadata (= the .git folder), you could probably bring it up to date (the rest to download would be small).
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rleibner » Thu Jan 11, 2018 4:58 pm

Not cool at all :cry:
A torrent would be great, but I found only the 2017.3.1 version (the version that I have yet) and need to test some new changes availables at 'next'.
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby rominet » Thu Jan 11, 2018 6:18 pm

I can put a tarball for download over http (no https, sorry, but I can give checksums, sign them, and you can check commit ids :wink: ), but I don't know if the upload rate will be high enough for you. I downloaded a bare clone (we'll see later in case you get it), it is 2.3 GB, compressed it as .tar.xz but that didn't make it shrink. Trying 'git clone --depth=1', which should be smaller and apparently can be expanded later if you want access to older commits.

Note: the “download snapshot” feature of SourceForge is not very useful here, as it doesn't have the .git directory.
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rleibner » Thu Jan 11, 2018 6:38 pm

Thanks, rominet.
Let us try that.
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby Hooray » Thu Jan 11, 2018 6:51 pm

Is the problem specific to sourceforge only, or are you seeing other connection issues ?
What about non SSL connections ?

In other words, don't you already have a working fgdata clone, i.e. wouldn't you only need the most recent changes ?
I suppose there are a number of recent "clones" that you could pull from, including some in your country or somewhere close to you, if that'd be of any help ?

Are you familiar with trace route to see if/where and how the connection is "broken" ?
Depending on the issue, you may be better of with a clone on a USB stick - I do remember, that we had this situation once, and a number of folks volunteered to send a flash disk by mail ... So that'd be another option.

Anyway, if the issue is fgdata-only, we could also post the corresponding patches and/or new files.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11100
Joined: Tue Mar 25, 2008 8:40 am

Re: FGdata downloading error

Postby rleibner » Thu Jan 11, 2018 7:18 pm

Usually my download speed is near 200 kbps, but from SF it barely reaches 70-90kbps.
Hooray wrote in Thu Jan 11, 2018 6:51 pm:In other words, don't you already have a working fgdata clone, i.e. wouldn't you only need the most recent changes ?

Florent has fixed the issue " FGaddons and Canvas Image".
I quote him:
Your oscilloscope is gorgeous! Make sure you have SimGear, FlightGear
and FGData up-to-date from 'next', then use something like:

var addon = addons.getAddon("org.flightgear.addons.Oscilloscope");

[...]

var skin = root.createChild("image", 'skin').set("src",
addon.resourcePath("osc2.png"));

Alternatively, you can obtain the addons.Addon ghost (here, the value of
the 'addon' variable) from the add-on main() function: since about 24
hours (FGData commit a57aed195e4), this is the only parameter passed to
main().

Thanks to the resourcePath() method of addons.Addon objects, you don't
have to use the special '[addon=ADDON_ID]relative/path' syntax when you
need a resource from your add-on, and you don't get to hardcode the
add-on identifier either in all such paths (still, the lookup will only
find files inside *your* add-on).
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby rominet » Thu Jan 11, 2018 11:09 pm

The minimum size I managed to get is 1.3 GB, that was with:
Code: Select all
git clone --bare --depth=1 https://git.code.sf.net/p/flightgear/fgdata fgdata-bare-depth1

However, trying to work with the resulting repository ('git fetch --depth=3', etc.) quickly wanted to download a lot of data, so finally I decided for a bare clone with full history:
Code: Select all
git clone --bare https://git.code.sf.net/p/flightgear/fgdata fgdata.git-bare

This one takes 2.3 GB, not that bad considering that it will become 4.2 GB when made non-bare (bare = the normal files of the repo are not visible, but can be recreated from the Git internal data inside the repo).

So, the main file you need to get is fgdata-bare-b45ae04c153.git.tar.gz. It is currently being uploaded, which should be finished in a few hours. There are signature files in the same directory.

In the following, I assume that you've downloaded all files from that directory, and that it is now your current directory.

If you know how to use gpg[1], you can verify my OpenPGP signature on the *.SHA256SUM and *.SHA512SUM files, using the corresponding *.sign files:
Code: Select all
% gpg2 --verify fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM.sign fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM
gpg: Signature made Thu 11 Jan 2018 11:23:04 PM CET
gpg:                using RSA key 125B5A0FDB788FDD0EF41A9DC785B90B5053A3A2
gpg: Good signature from "Florent Rougon <...>" [ultimate]
gpg:                 aka "Florent Rougon <...>" [ultimate]
% gpg2 --verify fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM.sign fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM
gpg: Signature made Thu 11 Jan 2018 11:23:13 PM CET
gpg:                using RSA key 125B5A0FDB788FDD0EF41A9DC785B90B5053A3A2
gpg: Good signature from "Florent Rougon <...>" [ultimate]
gpg:                 aka "Florent Rougon <...>" [ultimate]
%

Once you trust the *.SHA256SUM and *.SHA512SUM files, you can use them to check the integrity of your downloaded fgdata-bare-b45ae04c153.git.tar.gz:
Code: Select all
% sha256sum -c fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM
fgdata-bare-b45ae04c153.git.tar.gz: OK
% sha512sum -c fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM
fgdata-bare-b45ae04c153.git.tar.gz: OK
%

Finally, let's unpack the tarball and set up this repository (i.e., make it non-bare and configure the 'origin' remote to fetch updates from the official FGData repo):
Code: Select all
tar -xzf fgdata-bare-b45ae04c153.git.tar.gz
git clone fgdata.git-bare fgdata
cd fgdata
git remote set-url origin https://git.code.sf.net/p/flightgear/fgdata
git remote set-head origin next

You should now be able to update it with a simple 'git pull'. You can also remove the fgdata.git-bare directory in order to save space.

Good luck. :wink:

Edit: I simplified the instructions by removing output, and kept the default remote name 'origin' so as to avoid future confusion.

Edit 2: for better security in case the gpg verification step is not practicable (for you or anyone else), here are the contents of fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM:
Code: Select all
1b88f6ea5840f0618c74718ae7894006ee598285f55845006c10763ec42ec02d  fgdata-bare-b45ae04c153.git.tar.gz

and of fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM:
Code: Select all
0ff5947527d3361df2d8a182a4c32128342cc418c36b594f7e06bd1653da2893c00c6370ee517c46caf184170f40502f77e59c57262c175dca1dc06301e5557f  fgdata-bare-b45ae04c153.git.tar.gz


[1] This requires my OpenPGP key, instructions to get it can be found here.
Last edited by rominet on Sat Jan 13, 2018 8:15 pm, edited 2 times in total.
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rominet » Fri Jan 12, 2018 6:20 am

I had a problem with the upload, starting it again...

Edit: the upload is now completed.
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rominet » Fri Jan 12, 2018 9:42 pm

By the way, I didn't split the tarball, but since it's a plain HTTP download, you can resume it if interrupted, for instance with wget's -c option:
Code: Select all
wget -c http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
--2018-01-12 22:37:31--  http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
Resolving frougon.net (frougon.net)... 217.70.180.136, 2001:4b98:dc0:950::136
Connecting to frougon.net (frougon.net)|217.70.180.136|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 2427139443 (2.3G), 2368543717 (2.2G) remaining [application/x-gzip]
Saving to: ‘fgdata-bare-b45ae04c153.git.tar.gz’

fgdata-bare-b45ae04   2%[                    ]  60.69M  2.40MB/s

(this was run after interrupting the first download with Ctrl-C)
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rleibner » Sat Jan 13, 2018 5:08 pm

rominet, I am very grateful for your help. I am already downloading the tar file.

It seems that this new 2018 is testing my patience. Yesterday an intermittence in the power network spoiled my hdd.
I've reinstalled the OS and now downloading your tar file.
But it will take me a few days to install all my tools and get my work back.

I'll be in touch with you when ready.
Regards
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby rominet » Sat Jan 13, 2018 8:11 pm

Argh... really sorry to hear that. :|

Here are the contents of fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM:
Code: Select all
1b88f6ea5840f0618c74718ae7894006ee598285f55845006c10763ec42ec02d  fgdata-bare-b45ae04c153.git.tar.gz

and of fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM:
Code: Select all
0ff5947527d3361df2d8a182a4c32128342cc418c36b594f7e06bd1653da2893c00c6370ee517c46caf184170f40502f77e59c57262c175dca1dc06301e5557f  fgdata-bare-b45ae04c153.git.tar.gz

Verifying at least one of them is a good security measure even if you skip the gpg verification step or don't have a trust path to my OpenPGP key, because someone who would want to corrupt your download (by MITM or by hacking frougon.net) would also have to hack this forum to alter the checksums. And since the forum uses HTTPS, chances are good that you'll see the same checksums I posted. :)
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rleibner » Mon Jan 15, 2018 1:36 pm

Today, Monday, I am going to claim my ISP because I suppose something is not right.

Having downloaded 75%, wget reports:
Code: Select all
rodolfo@6200:~$ wget -c http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
--2018-01-15 10:10:57--  http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
Resolviendo frougon.net (frougon.net)... 217.70.180.136, 2001:4b98:dc0:950::136
Conectando con frougon.net (frougon.net)[217.70.180.136]:80... conectado.
Petición HTTP enviada, esperando respuesta... 206 Partial Content
Longitud: 2427139443 (2,3G), quedan 600438131 (573M) [application/x-gzip]
Guardando como: “fgdata-bare-b45ae04c153.git.tar.gz”

fgdata-bare-b45ae04  75%[+++++++++++++++     ]   1,70G  --.-KB/s    en 0s     

2018-01-15 10:11:58 (0,00 B/s) - Conexión cerrada en el byte 1826701312. Reintentando.
In English : Connection closed in byte 1826701312. Retrying.
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Re: FGdata downloading error

Postby rominet » Mon Jan 15, 2018 3:07 pm

Right, but it should be able to resume from the point where it was interrupted, right? That's the point of wget's -c option and the "206 Partial Content" HTTP response...

As a friend of mine likes to say, this “must converge”. :wink:
rominet
 
Posts: 548
Joined: Sat Nov 01, 2014 1:33 pm
Callsign: F-KATS
Version: Git next
OS: Debian GNU/Linux

Re: FGdata downloading error

Postby rleibner » Mon Jan 15, 2018 3:33 pm

should converge ...
I have tried dozens of times yesterday and today, always with the same result. Interestingly, the error is always in the same byte: 1826701312.
Rodolfo
*************************
Non-shared knowledge is lost knowledge
User avatar
rleibner
 
Posts: 222
Joined: Fri May 19, 2017 7:17 pm
Location: Uruguay - SUMU
Callsign: CX-BEX
Version: 2180.3.0
OS: Ubuntu 17.10

Next

Return to Linux

Who is online

Users browsing this forum: No registered users and 1 guest