Sharing large files on flaky networks with AWS S3 and BitTorrent

How to share a 3GB archive with a remote client when both might have unreliable connection that doesn’t keep up for the full download? Upload it to S3 using multipart upload with a resume capability and download it from S3 using BitTorrent.

(Note: I am not very proficient with Python so don’t judge the code too harshly.)

To upload it, we will use boto3 Multipart upload with small enough parts. If the connection fails, we can simply start again (supplying the upload id of the first attempt with --uploadid). See s3_multipart_upload.py for details:

$ ## The initial attempt:
$ ./s3_multipart_upload.py --path $(pwd)/my-file.zip \
    --key my-file.zip --bucket my-bucket.zip
Starting upload with id= GoN2qwUsc...
15000000 of 128924712 bytes uploaded (11.635%)
30000000 of 128924712 bytes uploaded (23.269%)
Traceback (most recent call last):
...
botocore.exceptions.EndpointConnectionError: ...


$ ## Resume:
$ ./s3_multipart_upload.py --path $(pwd)/my-file.zip \
    --key my-file.zip --bucket my-bucket.zip --uploadid "GoN2qwUsc..."
Continuing upload with id= GoN2qwUsc...
15000000 of 128924712 bytes uploaded (11.635%)
30000000 of 128924712 bytes uploaded (23.269%) # Actually not re-uploaded
45000000 of 128924712 bytes uploaded (34.904%)
60000000 of 128924712 bytes uploaded (46.539%)
75000000 of 128924712 bytes uploaded (58.173%)
90000000 of 128924712 bytes uploaded (69.808%)
105000000 of 128924712 bytes uploaded (81.443%)
120000000 of 128924712 bytes uploaded (93.078%)
128924712 of 128924712 bytes uploaded (100.000%)
{... 'Location': 'https://my-bucket.s3.eu-west-1.amazonaws.com/my-file.zip',
 'Bucket': 'my-bucket', 'Key': 'my-file.zip',
 'ETag': '"27f5a3d5becb0f0b9b27619caeb78f9d-9"'}

Finally, we get a torrent for the file:

get_torrent_file
import boto3
s3 = boto3.session.Session().client("s3")
res = s3.get_object_torrent(Bucket='my-bucket', Key='my-file.zip')
torrent = res['Body'].read()
f = open('my-file.torrent', 'w+b')
f.write(torrent)
f.close()

You can now share my-file.torrent with those who need it.

Caveats

S3 BitTorrent support only works for files up to 5GB. For large ones you want to find a different reliable download method. Also, the object needs to be public and it is the provider who pays for the data transfer, which might not be acceptable for you.


Tags: tools


Copyright © 2020 Jakub Holý
Powered by Cryogen
Theme by KingMob