Hi Connexion broken

[tutorialevideo]

Hi i get this all the time. can you share a update?
Thank you for your help.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 710, in _error_catcher
page parsed (1/0)
page parsed (2/29)
page parsed (3/29)
page parsed (4/29)
page parsed (5/29)
page parsed (6/29)
page parsed (7/29)
page parsed (8/29)
page parsed (9/29)
page parsed (10/29)
page parsed (11/29)
page parsed (12/29)
page parsed (13/29)
page parsed (14/29)
page parsed (15/29)
page parsed (16/29)
page parsed (17/29)
page parsed (18/29)
page parsed (19/29)
page parsed (20/29)
page parsed (21/29)
page parsed (22/29)
page parsed (23/29)
page parsed (24/29)
page parsed (25/29)
page parsed (26/29)
page parsed (27/29)
page parsed (28/29)
page parsed (29/29)
listing page(1) media(['GH010518.MP4', 'GH010517.MP4', 'GH010516.MP4', 'GH010515.MP4', 'GH010514.MP4', 'GH010513.MP4', 'GH010512.MP4', 'GOPR0511.JPG', 'GOPR0510.JPG', '', 'GOPR0509.JPG', 'GH010508.MP4', 'GH010507.MP4', 'GH010506.MP4', 'GH010505.MP4'])
downloading to ./download/1_page.zip
yield
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 1073, in read_chunked
self._update_chunk_length()
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 1008, in _update_chunk_length
raise InvalidChunkLength(self, line) from None
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 933, in stream
yield from self.read_chunked(amt, decode_content=decode_content)
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 1061, in read_chunked
with self._error_catcher():
File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/site-packages/urllib3/response.py", line 727, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/main.py", line 197, in
main()
File "/app/main.py", line 193, in main
gpp.download_media_ids(ids, filepath, progress_mode=args.progress_mode)
File "/app/main.py", line 135, in download_media_ids
for chunk in resp.iter_content(chunk_size=8192):
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[costa]

@itsankoff thank you for your work; i seem to experience the same issue (the first page is not downloading -- resulting in a 0-sized ZIP) -- the peculiar thing about this is, it worked before i uploaded manually some videos (which "auto-upload" failed to upload by itself). so, will really appreciate any ideas on your side.

btw, i think of integrating this in my automated video production pipeline, so it'd be really cool to have this working somehow.

thanks!

[costa]

@itsankoff a hint / a workaround -- just checked: increasing the page size may kinda work around the issue...

[costa]

@itsankoff

Reporting on usage and problems of this thing:

• in general, the problems stem from media library "API's" quirks (which need to be worked around)
• my own library's state (after manually re-uploading some failed auto-uploads) results in some problems downloading
  • (exiting quickly with no errors, but the downloaded ZIP size was 0)
  • I tried increasing the "page size" and worked around this issue successfully
• this issue (of broken connection) appears sporadically
  • my workaround that reduces the risk of this failure happening is increasing chunk_size (the iter_content parameter)
  • (seems to reduce the chances proportionally)
• I believe that retry option (validating the downloaded ZIP files) will be helpful with this tool

As a side note, I'm working on an automated video production pipeline -- described here https://medium.com/@costa/lively-video-reporting-proj-cont-bd4c4b9bd18e (using iCloud as a source) -- and I'll be glad to co-operate on that with users of tools like this.

cc @tutorialevideo

[georgik]

@tutorialevideo Please, try the version in PR: #9
It should be more resilient to failure. It contains retry mechanism.

[hubDreamer]

Hello, the container stops with a urllib3.exceptions.InvalidChunkLength error during download. It seems to happen after successfully downloading a few pages of media. It looks like the script cannot recover from this network error and crashes. Please see the log below.