I have added the request header.
The socket timeouts make me puzzled. The stackdump suggests that these
timeouts happen _after_ the http header has been received and the reception
of the message body fails.
This should not happen unless the connection gets interrupted in the middle
of a http request/response sequence (pulled plug, dsl going down or so), at
least it should be extremly rare.
Do you have any firewall that might play a role?
For the skipped directories: do you run with --quick? That would explain
why it does skip directories that have a valid .dirindex (this is by
intention)
Unless you have a complete error-free run of terrasync.py, don't use
--quick.
Regards
Torsten
<wkitty42@...> schrieb am Mi., 18. Mai 2016 um 14:09 Uhr
> On 05/18/2016 07:05 AM, Torsten Dreyer wrote:
> > Hi all,
> >
> > I have just pushed an improved version of terrasync.py.
> > This should be much faster than before as it uses the same connection
> for many
> > http requests.
>
> this should be a GoodThing<tm>...
>
> > It also checks the .dirindex'es local hash and only downloads the file
> if it has
> > changed.
>
> i'm not sure about this... it seems to stop at the top level even if the
> lower
> levels are not completed or even have .dirindex file... see below where it
> has
> been run again a full SVN that contains no .dirindex or .dirindex.tmp
> files...
>
> > It should handle connection aborts and restarts much more graceful.
>
> we must be missing something, then... i've had several of these socket
> timeouts
> and they seem to be really quick when they happen and not taking seconds or
> minutes of nothing before they trigger... on restarts, it skips the last
> one
> done... in the below, Airports is being worked on when the socket timeout
> occurs... the restart skips all the rest of Airports and moves on to
> Models...
> to get it to go back and do them properly, i have to delete .dirindex
> files
>
> please also note the PS at the bottom of this post...
>
>
> myuser@...:~$ ~/myflightgear/pyupdateTSScenery
>
> ********** Program start at 20160518_074237EDT **********
> processing
> processing //Airports
> processing /Airports/M
> processing /Airports/M/M
> processing /Airports/M/M/M
> processing /Airports/M/M/V
> [...]
> processing /Airports/L/F/E
> processing /Airports/L/F/Q
> processing /Airports/L/F/K
> Traceback (most recent call last):
> File "/home/myuser/myflightgear/terrasync.py", line 249, in <module>
> terraSync.start()
> File "/home/myuser/myflightgear/terrasync.py", line 161, in start
> self.updateDirectory("", "", None )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 121, in callback
> f.write(self.result.read())
> File "/usr/lib/python3.4/http/client.py", line 512, in read
> s = self._safe_read(self.length)
> File "/usr/lib/python3.4/http/client.py", line 662, in _safe_read
> chunk = self.fp.read(min(amt, MAXAMOUNT))
> File "/usr/lib/python3.4/socket.py", line 374, in readinto
> return self._sock.recv_into(b)
> socket.timeout: timed out
> RSLTCODE=1
> Removing left over .dirindex.tmp files...
> Process terminates.
> ********** Program stop at 20160518_074520EDT **********
> ********** Total elapsed time is 0h 2m 43s **********
>
>
> myuser@...:~$ ~/myflightgear/pyupdateTSScenery
>
> ********** Program start at 20160518_075048EDT **********
> processing
> processing //Airports
> processing //Models
> processing /Models/Airport
> processing /Models/Airport/Jetway
> processing /Models/Airport/Jetway/Airlines
> processing /Models/Airport/Vehicle
> processing /Models/Airport/Pushback
> processing /Models/Transport
> processing /Models/Effects
> processing /Models/Residential
> processing /Models/Sport
> processing /Models/Agriculture
> processing /Models/Power
> processing /Models/Aircraft
> processing /Models/Misc
> processing /Models/Boundaries
> processing /Models/Industrial
> processing /Models/Maritime
> processing /Models/Maritime/Misc
> processing /Models/Maritime/Civilian
> downloading /Models/Maritime/Civilian/LargeFerry.ac
> processing /Models/Maritime/Military
> processing /Models/Military
> processing /Models/Communications
> processing /Models/Fauna
> processing /Models/Commercial
> processing /Models/StreetFurniture
> processing /Models/Trees
> processing //Objects
> processing /Objects/e130n70
> processing /Objects/e130n70/e135n70
> processing /Objects/e130n70/e136n71
> [...]
> processing /Objects/w090n40/w081n44
> processing /Objects/w090n40/w085n43
> Traceback (most recent call last):
> File "/home/myuser/myflightgear/terrasync.py", line 249, in <module>
> terraSync.start()
> File "/home/myuser/myflightgear/terrasync.py", line 161, in start
> self.updateDirectory("", "", None )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 124, in callback
> self.mycallback(self)
> File "/home/myuser/myflightgear/terrasync.py", line 192, in
> handleDirindexRequest
> self.handleDirindexFile(dirindexRequest.dst)
> File "/home/myuser/myflightgear/terrasync.py", line 207, in
> handleDirindexFile
> self.updateDirectory( "/" + dirIndex.getPath() + "/" + d,
> join(dirIndex.getPath(),d), h )
> File "/home/myuser/myflightgear/terrasync.py", line 189, in
> updateDirectory
> self.httpGetter.get(request)
> File "/home/myuser/myflightgear/terrasync.py", line 62, in get
> httpGetCallback.callback()
> File "/home/myuser/myflightgear/terrasync.py", line 121, in callback
> f.write(self.result.read())
> File "/usr/lib/python3.4/http/client.py", line 512, in read
> s = self._safe_read(self.length)
> File "/usr/lib/python3.4/http/client.py", line 662, in _safe_read
> chunk = self.fp.read(min(amt, MAXAMOUNT))
> File "/usr/lib/python3.4/socket.py", line 374, in readinto
> return self._sock.recv_into(b)
> socket.timeout: timed out
> RSLTCODE=1
> Removing left over .dirindex.tmp files...
> Process terminates.
> ********** Program stop at 20160518_075328EDT **********
> ********** Total elapsed time is 0h 2m 40s **********
>
>
>
>
> PS: i had to add (again) the user agent to the headers to get the intrusion
> detection system to stop crying about the python user agent
>
> $ diff -u ~/flightgear-dev/next/flightgear/scripts/python/terrasync.py
> ~/myflightgear/terrasync.py
> --- /home/myuser/flightgear-dev/next/flightgear/scripts/python/terrasync.py
> 2016-05-18 07:19:07.563711948 -0400
> +++ /home/myuser/myflightgear/terrasync.py 2016-05-18
> 07:28:47.171726946 -0400
> @@ -41,7 +41,7 @@
> self.requests = []
> self.pendingRequests = []
> self.httpConnection =
> HTTPConnection(self.parsedBaseUrl.netloc,80, True)
> - self.httpRequestHeaders = headers =
>
> {'Host':self.parsedBaseUrl.netloc,'Content-Length':0,'Connection':'Keep-Alive'}
> + self.httpRequestHeaders = headers =
>
> {'Host':self.parsedBaseUrl.netloc,'Content-Length':0,'Connection':'Keep-Alive','User-Agent':'FlightGear
> Terrasyncer v2016.3.0'}
>
> def get(self, httpGetCallback):
>
>
>
> --
> NOTE: No off-list assistance is given without prior approval.
> *Please keep mailing list traffic on the list* unless
> private contact is specifically requested and granted.
>
>