FGdata downloading error

[rleibner]

Hi all,

I'm trying to update SimGear, FlightGear and FGData from 'next' using

Code: Select all

$ ../download_and_compile.sh -p n -j 4


the compilation seems ok (I didn't saw any error msg) but when downloading FGFS DATA (arriving next to 30%) the process stops with

Code: Select all

error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

Yesterday I tried it at least 5 times, always with a similar (and frustrating) result. I blame my poor internet connexion.

Is there any way to download DATA in batches (so I would not lose the downloaded KBs in each attempt)?

[rominet]

Hmm, not cool. If you run the command:

Code: Select all

git clone https://git.code.sf.net/p/flightgear/fgdata

several times in the same directory, does it restart from scratch every time?

Edit: this probably won't work, sorry (according to this stackoverflow page).

Otherwise, Parnikkapore used to seed Torrents for FG. If they contained even a non-up-to-date FGData with all the Git metadata (= the .git folder), you could probably bring it up to date (the rest to download would be small).

[rleibner]

Not cool at all
A torrent would be great, but I found only the 2017.3.1 version (the version that I have yet) and need to test some new changes availables at 'next'.

[rominet]

I can put a tarball for download over http (no https, sorry, but I can give checksums, sign them, and you can check commit ids ), but I don't know if the upload rate will be high enough for you. I downloaded a bare clone (we'll see later in case you get it), it is 2.3 GB, compressed it as .tar.xz but that didn't make it shrink. Trying 'git clone --depth=1', which should be smaller and apparently can be expanded later if you want access to older commits.

Note: the “download snapshot” feature of SourceForge is not very useful here, as it doesn't have the .git directory.

[rleibner]

Thanks, rominet.
Let us try that.

[Hooray]

Is the problem specific to sourceforge only, or are you seeing other connection issues ?
What about non SSL connections ?

In other words, don't you already have a working fgdata clone, i.e. wouldn't you only need the most recent changes ?
I suppose there are a number of recent "clones" that you could pull from, including some in your country or somewhere close to you, if that'd be of any help ?

Are you familiar with trace route to see if/where and how the connection is "broken" ?
Depending on the issue, you may be better of with a clone on a USB stick - I do remember, that we had this situation once, and a number of folks volunteered to send a flash disk by mail ... So that'd be another option.

Anyway, if the issue is fgdata-only, we could also post the corresponding patches and/or new files.

[rleibner]

Usually my download speed is near 200 kbps, but from SF it barely reaches 70-90kbps.

In other words, don't you already have a working fgdata clone, i.e. wouldn't you only need the most recent changes ?

Florent has fixed the issue " FGaddons and Canvas Image".
I quote him:

Your oscilloscope is gorgeous! Make sure you have SimGear, FlightGear
and FGData up-to-date from 'next', then use something like:

var addon = addons.getAddon("org.flightgear.addons.Oscilloscope");

[...]

var skin = root.createChild("image", 'skin').set("src",
addon.resourcePath("osc2.png"));

Alternatively, you can obtain the addons.Addon ghost (here, the value of
the 'addon' variable) from the add-on main() function: since about 24
hours (FGData commit a57aed195e4), this is the only parameter passed to
main().

Thanks to the resourcePath() method of addons.Addon objects, you don't
have to use the special '[addon=ADDON_ID]relative/path' syntax when you
need a resource from your add-on, and you don't get to hardcode the
add-on identifier either in all such paths (still, the lookup will only
find files inside *your* add-on).

[rominet]

The minimum size I managed to get is 1.3 GB, that was with:

Code: Select all

git clone --bare --depth=1 https://git.code.sf.net/p/flightgear/fgdata fgdata-bare-depth1

However, trying to work with the resulting repository ('git fetch --depth=3', etc.) quickly wanted to download a lot of data, so finally I decided for a bare clone with full history:

Code: Select all

git clone --bare https://git.code.sf.net/p/flightgear/fgdata fgdata.git-bare

This one takes 2.3 GB, not that bad considering that it will become 4.2 GB when made non-bare (bare = the normal files of the repo are not visible, but can be recreated from the Git internal data inside the repo).

So, the main file you need to get is fgdata-bare-b45ae04c153.git.tar.gz. It is currently being uploaded, which should be finished in a few hours. There are signature files in the same directory.

In the following, I assume that you've downloaded all files from that directory, and that it is now your current directory.

If you know how to use gpg[1], you can verify my OpenPGP signature on the *.SHA256SUM and *.SHA512SUM files, using the corresponding *.sign files:

Code: Select all

% gpg2 --verify fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM.sign fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM
gpg: Signature made Thu 11 Jan 2018 11:23:04 PM CET
gpg:                using RSA key 125B5A0FDB788FDD0EF41A9DC785B90B5053A3A2
gpg: Good signature from "Florent Rougon <...>" [ultimate]
gpg:                 aka "Florent Rougon <...>" [ultimate]
% gpg2 --verify fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM.sign fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM
gpg: Signature made Thu 11 Jan 2018 11:23:13 PM CET
gpg:                using RSA key 125B5A0FDB788FDD0EF41A9DC785B90B5053A3A2
gpg: Good signature from "Florent Rougon <...>" [ultimate]
gpg:                 aka "Florent Rougon <...>" [ultimate]
%

Once you trust the *.SHA256SUM and *.SHA512SUM files, you can use them to check the integrity of your downloaded fgdata-bare-b45ae04c153.git.tar.gz:

Code: Select all

% sha256sum -c fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM
fgdata-bare-b45ae04c153.git.tar.gz: OK
% sha512sum -c fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM
fgdata-bare-b45ae04c153.git.tar.gz: OK
%

Finally, let's unpack the tarball and set up this repository (i.e., make it non-bare and configure the 'origin' remote to fetch updates from the official FGData repo):

Code: Select all

tar -xzf fgdata-bare-b45ae04c153.git.tar.gz
git clone fgdata.git-bare fgdata
cd fgdata
git remote set-url origin https://git.code.sf.net/p/flightgear/fgdata
git remote set-head origin next

You should now be able to update it with a simple 'git pull'. You can also remove the fgdata.git-bare directory in order to save space.

Good luck.

Edit: I simplified the instructions by removing output, and kept the default remote name 'origin' so as to avoid future confusion.

Edit 2: for better security in case the gpg verification step is not practicable (for you or anyone else), here are the contents of fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM:

Code: Select all

1b88f6ea5840f0618c74718ae7894006ee598285f55845006c10763ec42ec02d  fgdata-bare-b45ae04c153.git.tar.gz

and of fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM:

Code: Select all

0ff5947527d3361df2d8a182a4c32128342cc418c36b594f7e06bd1653da2893c00c6370ee517c46caf184170f40502f77e59c57262c175dca1dc06301e5557f  fgdata-bare-b45ae04c153.git.tar.gz

[1] This requires my OpenPGP key, instructions to get it can be found here.

[rominet]

I had a problem with the upload, starting it again...

Edit: the upload is now completed.

[rominet]

By the way, I didn't split the tarball, but since it's a plain HTTP download, you can resume it if interrupted, for instance with wget's -c option:

Code: Select all

wget -c http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
--2018-01-12 22:37:31--  http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
Resolving frougon.net (frougon.net)... 217.70.180.136, 2001:4b98:dc0:950::136
Connecting to frougon.net (frougon.net)|217.70.180.136|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 2427139443 (2.3G), 2368543717 (2.2G) remaining [application/x-gzip]
Saving to: ‘fgdata-bare-b45ae04c153.git.tar.gz’

fgdata-bare-b45ae04   2%[                    ]  60.69M  2.40MB/s

(this was run after interrupting the first download with Ctrl-C)

[rleibner]

rominet, I am very grateful for your help. I am already downloading the tar file.

It seems that this new 2018 is testing my patience. Yesterday an intermittence in the power network spoiled my hdd.
I've reinstalled the OS and now downloading your tar file.
But it will take me a few days to install all my tools and get my work back.

I'll be in touch with you when ready.
Regards

[rominet]

Argh... really sorry to hear that.

Here are the contents of fgdata-bare-b45ae04c153.git.tar.gz.SHA256SUM:

Code: Select all

1b88f6ea5840f0618c74718ae7894006ee598285f55845006c10763ec42ec02d  fgdata-bare-b45ae04c153.git.tar.gz

and of fgdata-bare-b45ae04c153.git.tar.gz.SHA512SUM:

Code: Select all

0ff5947527d3361df2d8a182a4c32128342cc418c36b594f7e06bd1653da2893c00c6370ee517c46caf184170f40502f77e59c57262c175dca1dc06301e5557f  fgdata-bare-b45ae04c153.git.tar.gz

Verifying at least one of them is a good security measure even if you skip the gpg verification step or don't have a trust path to my OpenPGP key, because someone who would want to corrupt your download (by MITM or by hacking frougon.net) would also have to hack this forum to alter the checksums. And since the forum uses HTTPS, chances are good that you'll see the same checksums I posted.

[rleibner]

Today, Monday, I am going to claim my ISP because I suppose something is not right.

Having downloaded 75%, wget reports:

Code: Select all

rodolfo@6200:~$ wget -c http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
--2018-01-15 10:10:57--  http://frougon.net/tmp/FlightGear/fgdata/fgdata-bare-b45ae04c153.git.tar.gz
Resolviendo frougon.net (frougon.net)... 217.70.180.136, 2001:4b98:dc0:950::136
Conectando con frougon.net (frougon.net)[217.70.180.136]:80... conectado.
Petición HTTP enviada, esperando respuesta... 206 Partial Content
Longitud: 2427139443 (2,3G), quedan 600438131 (573M) [application/x-gzip]
Guardando como: “fgdata-bare-b45ae04c153.git.tar.gz”

fgdata-bare-b45ae04  75%[+++++++++++++++     ]   1,70G  --.-KB/s    en 0s     

2018-01-15 10:11:58 (0,00 B/s) - Conexión cerrada en el byte 1826701312. Reintentando.


In English : Connection closed in byte 1826701312. Retrying.

[rominet]

Right, but it should be able to resume from the point where it was interrupted, right? That's the point of wget's -c option and the "206 Partial Content" HTTP response...

As a friend of mine likes to say, this “must converge”.

[rleibner]

should converge ...
I have tried dozens of times yesterday and today, always with the same result. Interestingly, the error is always in the same byte: 1826701312.