U k^ 2@sdZddlmZddlmZddlmZddlm Z ddlm Z ddl m Z ddlmZdd lmZdd lmZmZd ZeeZdd dZGddde Z GdddeZGdddeZd S)azAbstractions over S3's upload/download operations. This module provides high level abstractions for efficient uploads/downloads. It handles several things for the user: * Automatically switching to multipart transfers when a file is over a specific size threshold * Uploading/downloading a file in parallel * Progress callbacks to monitor transfers * Retries. While botocore handles retries for streaming uploads, it is not possible for it to handle retries for streaming downloads. This module handles retries for both cases so you don't need to implement any retry logic yourself. This module has a reasonable set of defaults. It also allows you to configure many aspects of the transfer process including: * Multipart threshold size * Max parallel downloads * Socket timeouts * Retry amounts There is no support for s3->s3 multipart copies at this time. .. _ref_s3transfer_usage: Usage ===== The simplest way to use this module is: .. code-block:: python client = boto3.client('s3', 'us-west-2') transfer = S3Transfer(client) # Upload /tmp/myfile to s3://bucket/key transfer.upload_file('/tmp/myfile', 'bucket', 'key') # Download s3://bucket/key to /tmp/myfile transfer.download_file('bucket', 'key', '/tmp/myfile') The ``upload_file`` and ``download_file`` methods also accept ``**kwargs``, which will be forwarded through to the corresponding client operation. Here are a few examples using ``upload_file``:: # Making the object public transfer.upload_file('/tmp/myfile', 'bucket', 'key', extra_args={'ACL': 'public-read'}) # Setting metadata transfer.upload_file('/tmp/myfile', 'bucket', 'key', extra_args={'Metadata': {'a': 'b', 'c': 'd'}}) # Setting content type transfer.upload_file('/tmp/myfile.json', 'bucket', 'key', extra_args={'ContentType': "application/json"}) The ``S3Transfer`` class also supports progress callbacks so you can provide transfer progress to users. Both the ``upload_file`` and ``download_file`` methods take an optional ``callback`` parameter. Here's an example of how to print a simple progress percentage to the user: .. code-block:: python class ProgressPercentage(object): def __init__(self, filename): self._filename = filename self._size = float(os.path.getsize(filename)) self._seen_so_far = 0 self._lock = threading.Lock() def __call__(self, bytes_amount): # To simplify we'll assume this is hooked up # to a single filename. with self._lock: self._seen_so_far += bytes_amount percentage = (self._seen_so_far / self._size) * 100 sys.stdout.write( " %s %s / %s (%.2f%%)" % ( self._filename, self._seen_so_far, self._size, percentage)) sys.stdout.flush() transfer = S3Transfer(boto3.client('s3', 'us-west-2')) # Upload /tmp/myfile to s3://bucket/key and print upload progress. transfer.upload_file('/tmp/myfile', 'bucket', 'key', callback=ProgressPercentage('/tmp/myfile')) You can also provide a TransferConfig object to the S3Transfer object that gives you more fine grained control over the transfer. For example: .. code-block:: python client = boto3.client('s3', 'us-west-2') config = TransferConfig( multipart_threshold=8 * 1024 * 1024, max_concurrency=10, num_download_attempts=10, ) transfer = S3Transfer(client, config) transfer.upload_file('/tmp/foo', 'bucket', 'key') ) ClientError)six)RetriesExceededError)TransferConfig)TransferManager)NonThreadedExecutor)BaseSubscriber)OSUtils)rS3UploadFailedErroriNcCsd}|jst}t||||S)aCreates a transfer manager based on configuration :type client: boto3.client :param client: The S3 client to use :type config: boto3.s3.transfer.TransferConfig :param config: The transfer config to use :type osutil: s3transfer.utils.OSUtils :param osutil: The os utility to use :rtype: s3transfer.manager.TransferManager :returns: A transfer manager based on parameters provided N) use_threadsrr)clientconfigosutilZ executor_clsrh/private/var/folders/sd/whlwsn6x1_qgglc0mjv25_695qk2gl/T/pip-install-4zq3fp6i/boto3/boto3/s3/transfer.pycreate_transfer_managersrcsNeZdZdddZdeddeddded ffd d Zfd d ZZS)rmax_request_concurrencymax_io_queue_size)max_concurrency max_io_queue dTc sJtt|j||||||d|jD]}t||t||j|q"||_dS)a5Configuration object for managed S3 transfers :param multipart_threshold: The transfer size threshold for which multipart uploads, downloads, and copies will automatically be triggered. :param max_concurrency: The maximum number of threads that will be making requests to perform a transfer. If ``use_threads`` is set to ``False``, the value provided is ignored as the transfer will only ever use the main thread. :param multipart_chunksize: The partition size of each part for a multipart transfer. :param num_download_attempts: The number of download attempts that will be retried upon errors with downloading an object in S3. Note that these retries account for errors that occur when streaming down the data from s3 (i.e. socket errors and read timeouts that occur after receiving an OK response from s3). Other retryable exceptions such as throttling errors and 5xx errors are already retried by botocore (this default is 5). This does not take into account the number of exceptions retried by botocore. :param max_io_queue: The maximum amount of read parts that can be queued in memory to be written for a download. The size of each of these read parts is at most the size of ``io_chunksize``. :param io_chunksize: The max size of each chunk in the io queue. Currently, this is size used when ``read`` is called on the downloaded stream as well. :param use_threads: If True, threads will be used when performing S3 transfers. If False, no threads will be used in performing transfers: all logic will be ran in the main thread. )multipart_thresholdrmultipart_chunksizenum_download_attemptsr io_chunksizeN)superr__init__ALIASsetattrgetattrr ) selfrrrrrrr alias __class__rrr s,  zTransferConfig.__init__cs8||jkr"tt||j||tt|||dSN)r!rr __setattr__)r$namevaluer&rrr)s zTransferConfig.__setattr__) __name__ __module__ __qualname__r!MBKBr r) __classcell__rrr&rrs;rc@sNeZdZejZejZdddZdddZdddZdd Z d d Z d d Z dS) S3TransferNcCsd|s|std|r*t|||gr*td|dkr8t}|dkrFt}|rR||_nt||||_dS)NzLEither a boto3.Client or s3transfer.manager.TransferManager must be providedzdManager cannot be provided with client, config, nor osutil. These parameters are mutually exclusive.) ValueErroranyrr _managerr)r$r r rmanagerrrrr szS3Transfer.__init__c Cst|tjstd||}|j|||||}z |Wn>tk r|}z t d|d ||g|fW5d}~XYnXdS)a(Upload a file to an S3 object. Variants have also been injected into S3 client, Bucket and Object. You don't have to use S3Transfer.upload_file() directly. .. seealso:: :py:meth:`S3.Client.upload_file` :py:meth:`S3.Client.upload_fileobj` Filename must be a stringzFailed to upload %s to %s: %s/N) isinstancer string_typesr3_get_subscribersr5uploadresultrr join) r$filenamebucketkeycallback extra_args subscribersfutureerrr upload_files(    zS3Transfer.upload_filec Cspt|tjstd||}|j|||||}z |Wn,tk rj}zt |j W5d}~XYnXdS)a0Download an S3 object to a file. Variants have also been injected into S3 client, Bucket and Object. You don't have to use S3Transfer.download_file() directly. .. seealso:: :py:meth:`S3.Client.download_file` :py:meth:`S3.Client.download_fileobj` r7N) r9rr:r3r;r5downloadr=S3TransferRetriesExceededErrorrZlast_exception) r$r@rAr?rCrBrDrErFrrr download_file!s   zS3Transfer.download_filecCs|sdSt|gSr()ProgressCallbackInvokerr$rBrrrr;<szS3Transfer._get_subscriberscCs|Sr(r)r$rrr __enter__AszS3Transfer.__enter__cGs|jj|dSr()r5__exit__)r$argsrrrrNDszS3Transfer.__exit__)NNNN)NN)NN) r,r-r.rZALLOWED_DOWNLOAD_ARGSZALLOWED_UPLOAD_ARGSr rGrJr;rMrNrrrrr2s   r2c@s eZdZdZddZddZdS)rKzA back-compat wrapper to invoke a provided callback via a subscriber :param callback: A callable that takes a single positional argument for how many bytes were transferred. cCs ||_dSr(Z _callbackrLrrrr Nsz ProgressCallbackInvoker.__init__cKs||dSr(rP)r$Zbytes_transferredkwargsrrr on_progressQsz#ProgressCallbackInvoker.on_progressN)r,r-r.__doc__r rRrrrrrKHsrK)N)rSZbotocore.exceptionsrZbotocore.compatrZs3transfer.exceptionsrrIZs3transfer.managerrZS3TransferConfigrZs3transfer.futuresrZs3transfer.subscribersrZs3transfer.utilsr Zboto3.exceptionsr r0r/robjectr2rKrrrr sp         J[