o dv@sddlZddlmZddlmZmZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZGdddZGd d d ZGd d d ZGd ddeZGdddeZGdddeZGddde ZGddde ZGddde ZdS)N)BytesIOreadableseekable)IN_MEMORY_UPLOAD_TAG)CompleteMultipartUploadTaskCreateMultipartUploadTaskSubmissionTaskTask)ChunksizeAdjusterDeferredOpenFile get_callbacksget_filtered_dictc@s.eZdZd ddZddZddZdd Zd S) AggregatedProgressCallbackcCs||_||_d|_dS)aAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total rN) _callbacks _threshold _bytes_seen)self callbacks thresholdrl/private/var/folders/v1/_jykv66s6qd26_69j1njbrl80000gr/T/pip-target-p1gutpg6/lib/python/s3transfer/upload.py__init__!s  z#AggregatedProgressCallback.__init__cCs*|j|7_|j|jkr|dSdSN)rr_trigger_callbacks)rbytes_transferredrrr__call__1s  z#AggregatedProgressCallback.__call__cCs|jdkr |dSdS)z@Flushes out any progress that has not been sent to its callbacksrN)rrrrrrflush6s  z AggregatedProgressCallback.flushcCs"|jD]}||jdqd|_dS)N)rr)rr)rcallbackrrrr;s  z-AggregatedProgressCallback._trigger_callbacksN)r)__name__ __module__ __qualname__rrrrrrrrr s   rc@sLeZdZdZddZdddZddd Zd d Zd d ZddZ ddZ dS)InterruptReaderaWrapper that can interrupt reading using an error It uses a transfer coordinator to propagate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dSr)_fileobj_transfer_coordinator)rfileobjtransfer_coordinatorrrrrOs zInterruptReader.__init__NcCs|jjr|jj|j|Sr)r& exceptionr%read)ramountrrrr*Ss zInterruptReader.readrcCs|j||dSr)r%seek)rwherewhencerrrr,]szInterruptReader.seekcCs |jSr)r%tellrrrrr/`s zInterruptReader.tellcCs|jdSr)r%closerrrrr0czInterruptReader.closecCs|Srrrrrr __enter__fzInterruptReader.__enter__cOs |dSr)r0)rargskwargsrrr__exit__i zInterruptReader.__exit__r)r) r!r"r#__doc__rr*r,r/r0r2r6rrrrr$As   r$c@sfeZdZdZdddZeddZddZd d Zd d Z d dZ ddZ ddZ ddZ ddZdS)UploadInputManageraJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. NcCs||_||_||_dSr)_osutilr&_bandwidth_limiterrosutilr(bandwidth_limiterrrrr}s zUploadInputManager.__init__cCtd)aDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. zmust implement _is_compatible()NotImplementedErrorclsZ upload_sourcerrr is_compatibles z UploadInputManager.is_compatiblecCr?)aWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. z%must implement store_body_in_memory()r@roperation_namerrrstores_body_in_memory z(UploadInputManager.stores_body_in_memorycCr?)zProvides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request z&must implement provide_transfer_size()r@rtransfer_futurerrrprovide_transfer_sizesz(UploadInputManager.provide_transfer_sizecCr?)aDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuration and size. False, otherwise. z*must implement requires_multipart_upload()r@rrJconfigrrrrequires_multipart_uploadrHz,UploadInputManager.requires_multipart_uploadcCr?)aReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. z$must implement get_put_object_body()r@rIrrrget_put_object_bodyrHz&UploadInputManager.get_put_object_bodycCr?)aYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. z)must implement yield_upload_part_bodies()r@)rrJ chunksizerrryield_upload_part_bodiessz+UploadInputManager.yield_upload_part_bodiescCs*t||j}|jr|jj||jdd}|S)NF)enabled)r$r&r;Zget_bandwith_limited_stream)rr'rrr _wrap_fileobjs z UploadInputManager._wrap_fileobjcCst|d}|r t|gSgS)Nprogress)r r)rrJrrrr_get_progress_callbackss  z*UploadInputManager._get_progress_callbackscCsdd|DS)NcSsg|]}|jqSr)r).0r rrr sz;UploadInputManager._get_close_callbacks..r)rZaggregated_progress_callbacksrrr_get_close_callbacksr1z'UploadInputManager._get_close_callbacksr)r!r"r#r8r classmethodrDrGrKrNrOrQrSrUrXrrrrr9ms    r9c@sdeZdZdZeddZddZddZdd Zd d Z d d Z ddZ ddZ ddZ ddZdS)UploadFilenameInputManagerzUpload utility for filenamescCs t|tSr) isinstancestrrBrrrrDs z(UploadFilenameInputManager.is_compatiblecCdS)NFrrErrrrGr3z0UploadFilenameInputManager.stores_body_in_memorycCs|j|j|jjjdSr)metarKr:Z get_file_size call_argsr'rIrrrrKsz0UploadFilenameInputManager.provide_transfer_sizecCs|jj|jkSr)r^sizemultipart_thresholdrLrrrrNr1z4UploadFilenameInputManager.requires_multipart_uploadcCsJ||\}}||}||}||}|jj}|jj|||||dS)Nr' chunk_sizefull_file_sizerclose_callbacks)&_get_put_object_fileobj_with_full_sizerSrUrXr^r`r:#open_file_chunk_reader_from_fileobj)rrJr' full_sizerrer`rrrrOs   z.UploadFilenameInputManager.get_put_object_bodyc cs|jj}|||}td|dD]5}||}||}||d}|j|jjj|||d\} } | | } |j j | || ||d} || fVqdS)N) start_byte part_sizerdrb) r^r`_get_num_partsrangerUrX'_get_upload_part_fileobj_with_full_sizer_r'rSr:rg) rrJrPrdZ num_parts part_numberrrerjr'rhZread_file_chunkrrrrQs.      z3UploadFilenameInputManager.yield_upload_part_bodiescCst|||jjd}|S)N)Z open_function)r r:open)rr'rjrrr_get_deferred_open_file1s z2UploadFilenameInputManager._get_deferred_open_filecCs"|jjj}|jj}||d|fS)Nr)r^r_r'r`rqrrJr'r`rrrrf7s zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizecKs |d}|d}||||fS)Nrjrd)rq)rr'r5rjrhrrrrn<szBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizecCstt|jjt|Sr)intmathceilr^r`float)rrJrkrrrrlAsz)UploadFilenameInputManager._get_num_partsN)r!r"r#r8rYrDrGrKrNrOrQrqrfrnrlrrrrrZs  rZc@s<eZdZdZeddZddZddZdd Zd d Z d S) UploadSeekableInputManagerz&Upload utility for an open file objectcCst|ot|SrrrBrrrrDHsz(UploadSeekableInputManager.is_compatiblecCs|dkrdSdS)N put_objectFTrrErrrrGLsz0UploadSeekableInputManager.stores_body_in_memorycCsD|jjj}|}|dd|}|||j||dS)Nr)r^r_r'r/r,rK)rrJr'Zstart_positionZ end_positionrrrrKRs   z0UploadSeekableInputManager.provide_transfer_sizecKs||d}t|t|fS)Nrk)r*rlen)rr'r5datarrrrn_szBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizecCs"|jjj}||jj}||fSr)r^r_r'r/r`rrrrrrfns zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN) r!r"r#r8rYrDrGrKrnrfrrrrrwEs  rwcsheZdZdZdfdd ZeddZddZd d Zd d Z d dZ ddZ dddZ ddZ ZS)UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Ncst|||d|_dS)N)superr _initial_datar< __class__rrrys z&UploadNonSeekableInputManager.__init__cCst|Sr)rrBrrrrD}sz+UploadNonSeekableInputManager.is_compatiblecCr])NTrrErrrrGr3z3UploadNonSeekableInputManager.stores_body_in_memorycCsdSrrrIrrrrKsz3UploadNonSeekableInputManager.provide_transfer_sizecCsP|jjdur |jj|jkS|jjj}|j}|||d|_t|j|kr&dSdS)NFT)r^r`rar_r'_readrrz)rrJrMr'rrrrrNs  z7UploadNonSeekableInputManager.requires_multipart_uploadcCs@||}||}|jjj}||j|||}d|_|Sr)rUrXr^r_r' _wrap_datarr*)rrJrrer'bodyrrrrOs   z1UploadNonSeekableInputManager.get_put_object_bodyc cs`|jjj}d} ||}||}|d7}|||}|s!dS||||}d}||fVq )NrTri)r^r_r'rUrXrr) rrJrPZ file_objectrorreZ part_contentZ part_objectrrrrQs      z6UploadNonSeekableInputManager.yield_upload_part_bodiesTcCsxt|jdkr ||S|t|jkr&|jd|}|r$|j|d|_|S|t|j}|j||}|r:d|_|S)a= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. rNr})rzrr*)rr'r+truncater{Zamount_to_readrrrrs z#UploadNonSeekableInputManager._readcCs,|t|}|jj|t|t|||dS)a Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. rb)rSrr:rgrz)rr{rrer'rrrrsz(UploadNonSeekableInputManager._wrap_datar)T)r!r"r#r8rrYrDrGrKrNrOrQrr __classcell__rrrrr|vs  *r|c@s\eZdZdZgdZddgZddZ ddd Zd d Zd d Z ddZ ddZ ddZ dS)UploadSubmissionTaskz.Task for submitting tasks to execute an upload)ChecksumAlgorithmZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5 RequestPayerExpectedBucketOwnerrrcCsDtttg}|jjj}|D] }||r|Sq td|t |)aoRetrieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. z&Input {} of type: {} is not supported.) rZrwr|r^r_r'rD RuntimeErrorformattype)rrJZupload_manager_resolver_chainr'Zupload_manager_clsrrr_get_upload_input_manager_clss   z2UploadSubmissionTask._get_upload_input_manager_clsNcCsf||||j|}|jjdur|||||s'|||||||dS|||||||dS)a :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N)rr&r^r`rKrN_submit_upload_request_submit_multipart_request)rclientrMr=request_executorrJr>upload_input_managerrrr_submit(s4   zUploadSubmissionTask._submitc CsN|jj}||d}|jj|t|j||||j|j|j ddd|ddS)Nrx)rr'bucketkey extra_argsT)r( main_kwargsis_finaltag) r^r__get_upload_task_tagr&submit PutObjectTaskrOrrr) rrrMr=rrJrr_Zput_object_tagrrrras(   z+UploadSubmissionTask._submit_upload_requestcCs|jj}|j|t|j||j|j|jdd}g} ||j} | |d} |jj } t } | |j | }|||}|D]!\}}| |jj|t|j|||j|j|| dd|id| dq<||j}|j|t|j||j|j|d|| dd d dS) N)rrrr)r(r upload_part)rr'rrror upload_id)r(rpending_main_kwargsr)rpartsT)r(rrr)r^r_r&rrrrr_extra_upload_part_argsrr`r Zadjust_chunksizeZmultipart_chunksizerQappendUploadPartTask_extra_complete_multipart_argsr)rrrMr=rrJrr_Zcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagr`ZadjusterrPZ part_iteratorror'Zcomplete_multipart_extra_argsrrrrsx    z.UploadSubmissionTask._submit_multipart_requestcC t||jSr)rUPLOAD_PART_ARGSrrrrrrs z,UploadSubmissionTask._extra_upload_part_argscCrr)rCOMPLETE_MULTIPART_ARGSrrrrrr7z3UploadSubmissionTask._extra_complete_multipart_argscCsd}||r t}|Sr)rGr)rrrFrrrrrs z)UploadSubmissionTask._get_upload_task_tagr) r!r"r#r8rrrrrrrrrrrrrrs ! 9#V rc@eZdZdZddZdS)rz Task to do a nonmultipart uploadcCsB|}|jd|||d|WddS1swYdS)aP :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the upload. )BucketKeyBodyNr)rx)rrr'rrrrrrr_mains "zPutObjectTask._mainNr!r"r#r8rrrrrr rc@r)rz+Task to upload a part in a multipart uploadc Cs|}|jd|||||d|} Wdn1swY| d} | |d} d|vrB|d} d| } | | vrB| | | | <| S)a :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param part_number: The number representing the part of the multipart upload :param extra_args: A dictionary of any extra arguments that may be used in the upload. :rtype: dict :returns: A dictionary representing a part:: {'Etag': etag_value, 'PartNumber': part_number} This value can be appended to a list to be used to complete the multipart upload. )rrZUploadId PartNumberrNETag)rrrZChecksumr)rupper)rrr'rrrrorrresponseetagZ part_metadataZalgorithm_nameZchecksum_memberrrrrs&    zUploadPartTask._mainNrrrrrrrr)rtiorZs3transfer.compatrrZs3transfer.futuresrZs3transfer.tasksrrr r Zs3transfer.utilsr r r rrr$r9rZrwr|rrrrrrrs"  !,{]1 j