Sharing large files on flaky networks with AWS S3 and BitTorrent
How to share a 3GB archive with a remote client when both might have unreliable connection that doesn’t keep up for the full download? Upload it to S3 using multipart upload with a resume capability and download it from S3 using BitTorrent.
(Note: I am not very proficient with Python so don’t judge the code too harshly.)
To upload it, we will use boto3 Multipart upload with small enough parts. If the connection fails, we can simply start again (supplying the upload id of the first attempt with --uploadid). See s3_multipart_upload.py for details:
$ ## The initial attempt:
$ ./s3_multipart_upload.py --path $(pwd)/my-file.zip \
--key my-file.zip --bucket my-bucket.zip
Starting upload with id= GoN2qwUsc...
15000000 of 128924712 bytes uploaded (11.635%)
30000000 of 128924712 bytes uploaded (23.269%)
Traceback (most recent call last):
...
botocore.exceptions.EndpointConnectionError: ...
$ ## Resume:
$ ./s3_multipart_upload.py --path $(pwd)/my-file.zip \
--key my-file.zip --bucket my-bucket.zip --uploadid "GoN2qwUsc..."
Continuing upload with id= GoN2qwUsc...
15000000 of 128924712 bytes uploaded (11.635%)
30000000 of 128924712 bytes uploaded (23.269%) # Actually not re-uploaded
45000000 of 128924712 bytes uploaded (34.904%)
60000000 of 128924712 bytes uploaded (46.539%)
75000000 of 128924712 bytes uploaded (58.173%)
90000000 of 128924712 bytes uploaded (69.808%)
105000000 of 128924712 bytes uploaded (81.443%)
120000000 of 128924712 bytes uploaded (93.078%)
128924712 of 128924712 bytes uploaded (100.000%)
{... 'Location': 'https://my-bucket.s3.eu-west-1.amazonaws.com/my-file.zip',
'Bucket': 'my-bucket', 'Key': 'my-file.zip',
'ETag': '"27f5a3d5becb0f0b9b27619caeb78f9d-9"'}Finally, we get a torrent for the file:
get_torrent_file
import boto3
s3 = boto3.session.Session().client("s3")
res = s3.get_object_torrent(Bucket='my-bucket', Key='my-file.zip')
torrent = res['Body'].read()
f = open('my-file.torrent', 'w+b')
f.write(torrent)
f.close()You can now share my-file.torrent with those who need it.