V!\c@sddlZddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z ddl m Z ddl m Z dd l mZdd l mZdd l mZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl#m%Z%ej&e'Z(de)fdYZ*de)fdYZ+de)fdYZ,dS( iN(tsix(tKBtMB(tALLOWED_DOWNLOAD_ARGS(t get_callbacks(tsignal_transferring(tsignal_not_transferring(tCallArgs(tOSUtils(t TaskSemaphore(tSlidingWindowSemaphore(tCancelledError(t FatalError(tIN_MEMORY_DOWNLOAD_TAG(tIN_MEMORY_UPLOAD_TAG(tBoundedExecutor(tTransferFuture(t TransferMeta(tTransferCoordinator(tDownloadSubmissionTask(tUploadSubmissionTask(tCopySubmissionTask(tDeleteSubmissionTask(t LeakyBucket(tBandwidthLimitertTransferConfigc BsJeZdededdddddeddddd ZdZRS(ii iiic Csz||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ |j dS(s(Configurations for the transfer mangager :param multipart_threshold: The threshold for which multipart transfers occur. :param max_request_concurrency: The maximum number of S3 API transfer-related requests that can happen at a time. :param max_submission_concurrency: The maximum number of threads processing a call to a TransferManager method. Processing a call usually entails determining which S3 API requests that need to be enqueued, but does **not** entail making any of the S3 API data transfering requests needed to perform the transfer. The threads controlled by ``max_request_concurrency`` is responsible for that. :param multipart_chunksize: The size of each transfer if a request becomes a multipart transfer. :param max_request_queue_size: The maximum amount of S3 API requests that can be queued at a time. A value of zero means that there is no maximum. :param max_submission_queue_size: The maximum amount of TransferManager method calls that can be queued at a time. A value of zero means that there is no maximum. :param max_io_queue_size: The maximum amount of read parts that can be queued to be written to disk per download. A value of zero means that there is no maximum. The default size for each element in this queue is 8 KB. :param io_chunksize: The max size of each chunk in the io queue. Currently, this is size used when reading from the downloaded stream as well. :param num_download_attempts: The number of download attempts that will be tried upon errors with downloading an object in S3. Note that these retries account for errors that occur when streamming down the data from s3 (i.e. socket errors and read timeouts that occur after recieving an OK response from s3). Other retryable exceptions such as throttling errors and 5xx errors are already retried by botocore (this default is 5). The ``num_download_attempts`` does not take into account the number of exceptions retried by botocore. :param max_in_memory_upload_chunks: The number of chunks that can be stored in memory at a time for all ongoing upload requests. This pertains to chunks of data that need to be stored in memory during an upload if the data is sourced from a file-like object. The total maximum memory footprint due to a in-memory upload chunks is roughly equal to: max_in_memory_upload_chunks * multipart_chunksize + max_submission_concurrency * multipart_chunksize ``max_submission_concurrency`` has an affect on this value because for each thread pulling data off of a file-like object, they may be waiting with a single read chunk to be submitted for upload because the ``max_in_memory_upload_chunks`` value has been reached by the threads making the upload request. :param max_in_memory_download_chunks: The number of chunks that can be buffered in memory and **not** in the io queue at a time for all ongoing dowload requests. This pertains specifically to file-like objects that cannot be seeked. The total maximum memory footprint due to a in-memory download chunks is roughly equal to: max_in_memory_download_chunks * multipart_chunksize :param max_bandwidth: The maximum bandwidth that will be consumed in uploading and downloading file content. The value is in terms of bytes per second. N( tmultipart_thresholdtmultipart_chunksizetmax_request_concurrencytmax_submission_concurrencytmax_request_queue_sizetmax_submission_queue_sizetmax_io_queue_sizet io_chunksizetnum_download_attemptstmax_in_memory_upload_chunkstmax_in_memory_download_chunkst max_bandwidtht_validate_attrs_are_nonzero( tselfRRRRRRR R!R"R#R$R%((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyt__init__0sW            cCsUxN|jjD]=\}}|dk r|dkrtd||fqqWdS(Nis9Provided parameter %s of value %s must be greater than 0.(t__dict__titemstNonet ValueError(R'tattrtattr_val((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyR&s N(t__name__t __module__RRR+R(R&(((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyR/s YtTransferManagercBs/eZeZdddddddddd d d d d ddddddgZeddddddddgZddd gZd-d-d-dZd-d-dZd-d-d Z d-d-d-d!Z d-d-d"Z d#Z d-d$Z d%Zd&Zd'Zd(Zd)Zed*d+Zed,ZRS(.tACLt CacheControltContentDispositiontContentEncodingtContentLanguaget ContentTypetExpirestGrantFullControlt GrantReadt GrantReadACPt GrantWriteACPtMetadatat RequestPayertServerSideEncryptiont StorageClasstSSECustomerAlgorithmtSSECustomerKeytSSECustomerKeyMD5t SSEKMSKeyIdtWebsiteRedirectLocationtCopySourceIfMatchtCopySourceIfModifiedSincetCopySourceIfNoneMatchtCopySourceIfUnmodifiedSincetCopySourceSSECustomerAlgorithmtCopySourceSSECustomerKeytCopySourceSSECustomerKeyMD5tMetadataDirectivetMFAt VersionIdc Csm||_||_|dkr-t|_n||_|dkrQt|_nt|_d|_t d|jj d|jj dit |jj t6t|jjt6d||_t d|jjd|jjd||_t d|jjddd||_d|_|jjdk r_tjd|jjt|jj}t||_n|jdS( sA transfer manager interface for Amazon S3 :param client: Client to be used by the manager :param config: TransferConfig to associate specific configurations :param osutil: OSUtils object to use for os-related behavior when using with transfer manager. :type executor_cls: s3transfer.futures.BaseExecutor :param executor_cls: The class of executor to use with the transfer manager. By default, concurrent.futures.ThreadPoolExecutor is used. itmax_sizetmax_num_threadsttag_semaphorest executor_clsisSetting max_bandwidth to %sN(t_clientt_configR+Rt_osutilRtTransferCoordinatorControllert_coordinator_controllert _id_counterRRRR R#RR R$R t_request_executorRRt_submission_executorR t _io_executort_bandwidth_limiterR%tloggertdebugRRt_register_handlers(R'tclienttconfigtosutilRSt leaky_bucket((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyR(sB                c Cs|dkri}n|dkr*g}n|j||jtd|d|d|d|d|}i}|jr|j|dscCs$|jtj|jSWdQXdS(s.The set of transfer coordinators being trackedN(RRvR(R'((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyttracked_transfer_coordinatorsGs cCs$|j|jj|WdQXdS(sAdds a transfer coordinator of a transfer to be canceled if needed :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator for the particular transfer N(RRtadd(R'R((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyRPs cCs$|j|jj|WdQXdS(sRemove a transfer coordinator from cancelation consideration Typically, this method is invoked by the transfer coordinator itself to remove its self when it completes its transfer. :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator for the particular transfer N(RRtremove(R'R((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyRZs RcCs(x!|jD]}|j||q WdS(sPCancels all inprogress transfers This cancels the inprogress transfers by calling cancel() on all tracked transfer coordinators. :param msg: The message to pass on to each transfer coordinator that gets cancelled. :param exc_type: The type of exception to set for the cancellation N(RR(R'tmsgRR((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyRgs cCsyy(d}x|jD]}|jqWWnJtk rdtjd|r^tjd|nntk rtnXdS(sWait until there are no more inprogress transfers This will not stop when failures are encountered and not propogate any of these errors from failed transfers, but it can be interrupted with a KeyboardInterrupt. s$Received KeyboardInterrupt in wait()s'On KeyboardInterrupt was waiting for %sN(R+RtresultRR^R_t Exception(R'R((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyRus    ( R/R0R(tpropertyRRRR RR(((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyRW=s   (-RvtloggingRtbotocore.compatRts3transfer.constantsRRRts3transfer.utilsRRRRRR R ts3transfer.exceptionsR R ts3transfer.futuresR RRRRRts3transfer.downloadRts3transfer.uploadRts3transfer.copiesRts3transfer.deleteRts3transfer.bandwidthRRt getLoggerR/R^tobjectRR1RW(((s8/tmp/pip-install-usGedi/s3transfer/s3transfer/manager.pyt s>   n