B \vs@sddlZddlmZddlmZmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd l mZdd l mZdd l mZmZGd d d eZGdddeZGdddeZGdddeZGdddeZGdddeZGddde ZGddde ZGddde ZdS)N)six)seekablereadable)IN_MEMORY_UPLOAD_TAG)Task)SubmissionTask)CreateMultipartUploadTask)CompleteMultipartUploadTask) get_callbacks)get_filtered_dict)DeferredOpenFileChunksizeAdjusterc@s.eZdZd ddZddZddZdd Zd S) AggregatedProgressCallbackcCs||_||_d|_dS)aAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total rN) _callbacks _threshold _bytes_seen)self callbacks thresholdr7/tmp/pip-build-uw_ogi45/s3transfer/s3transfer/upload.py__init__s z#AggregatedProgressCallback.__init__cCs&|j|7_|j|jkr"|dS)N)rr_trigger_callbacks)rbytes_transferredrrr__call__-s z#AggregatedProgressCallback.__call__cCs|jdkr|dS)z@Flushes out any progress that has not been sent to its callbacksrN)rr)rrrrflush2s z AggregatedProgressCallback.flushcCs&x|jD]}||jdqWd|_dS)N)rr)rr)rcallbackrrrr7s z-AggregatedProgressCallback._trigger_callbacksN)r)__name__ __module__ __qualname__rrrrrrrrrs rc@sJeZdZdZddZdddZddZd d Zd d Zd dZ ddZ dS)InterruptReaderaWrapper that can interrupt reading using an error It uses a transfer coordinator to propagate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dS)N)_fileobj_transfer_coordinator)rfileobjtransfer_coordinatorrrrrJszInterruptReader.__init__NcCs|jjr|jj|j|S)N)r# exceptionr"read)ramountrrrr'NszInterruptReader.readcCs|j|dS)N)r"seek)rwhererrrr)XszInterruptReader.seekcCs |jS)N)r"tell)rrrrr+[szInterruptReader.tellcCs|jdS)N)r"close)rrrrr,^szInterruptReader.closecCs|S)Nr)rrrr __enter__aszInterruptReader.__enter__cOs |dS)N)r,)rargskwargsrrr__exit__dszInterruptReader.__exit__)N) rrr __doc__rr'r)r+r,r-r0rrrrr!=s  r!c@sfeZdZdZdddZeddZddZd d Zd d Z d dZ ddZ ddZ ddZ ddZdS)UploadInputManageraJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. NcCs||_||_||_dS)N)_osutilr#_bandwidth_limiter)rosutilr%bandwidth_limiterrrrrwszUploadInputManager.__init__cCs tddS)aDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. zmust implement _is_compatible()N)NotImplementedError)cls upload_sourcerrr is_compatible|s z UploadInputManager.is_compatiblecCs tddS)aWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. z%must implement store_body_in_memory()N)NotImplemented)roperation_namerrrstores_body_in_memorys z(UploadInputManager.stores_body_in_memorycCs tddS)zProvides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request z&must implement provide_transfer_size()N)r7)rtransfer_futurerrrprovide_transfer_sizesz(UploadInputManager.provide_transfer_sizecCs tddS)aDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuartion and size. False, otherwise. z*must implement requires_multipart_upload()N)r7)rr>configrrrrequires_multipart_uploads z,UploadInputManager.requires_multipart_uploadcCs tddS)aReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. z$must implement get_put_object_body()N)r7)rr>rrrget_put_object_bodys z&UploadInputManager.get_put_object_bodycCs tddS)aYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. z)must implement yield_upload_part_bodies()N)r7)rr> chunksizerrryield_upload_part_bodiessz+UploadInputManager.yield_upload_part_bodiescCs*t||j}|jr&|jj||jdd}|S)NF)Zenabled)r!r#r4Zget_bandwith_limited_stream)rr$rrr _wrap_fileobjs  z UploadInputManager._wrap_fileobjcCst|d}|rt|gSgS)Nprogress)r r)rr>rrrr_get_progress_callbackss  z*UploadInputManager._get_progress_callbackscCsdd|DS)NcSsg|] }|jqSr)r).0rrrr sz;UploadInputManager._get_close_callbacks..r)rZaggregated_progress_callbacksrrr_get_close_callbackssz'UploadInputManager._get_close_callbacks)N)rrr r1r classmethodr:r=r?rArBrDrErGrJrrrrr2hs   r2c@sdeZdZdZeddZddZddZdd Zd d Z d d Z ddZ ddZ ddZ ddZdS)UploadFilenameInputManagerzUpload utility for filenamescCs t|tjS)N) isinstancer string_types)r8r9rrrr:sz(UploadFilenameInputManager.is_compatiblecCsdS)NFr)rr<rrrr=sz0UploadFilenameInputManager.stores_body_in_memorycCs|j|j|jjjdS)N)metar?r3Z get_file_size call_argsr$)rr>rrrr?sz0UploadFilenameInputManager.provide_transfer_sizecCs|jj|jkS)N)rOsizemultipart_threshold)rr>r@rrrrAsz4UploadFilenameInputManager.requires_multipart_uploadcCsJ||\}}||}||}||}|jj}|jj|||||dS)N)r$ chunk_sizefull_file_sizerclose_callbacks)&_get_put_object_fileobj_with_full_sizerErGrJrOrQr3#open_file_chunk_reader_from_fileobj)rr>r$ full_sizerrUrQrrrrBs    z.UploadFilenameInputManager.get_put_object_bodyc cs|jj}|||}x|td|dD]j}||}||}||d}|j|jjj|||d\} } | | } |j j | || ||d} || fVq$WdS)N) start_byte part_sizerT)r$rSrTrrU) rOrQ_get_num_partsrangerGrJ'_get_upload_part_fileobj_with_full_sizerPr$rEr3rW) rr>rCrTZ num_parts part_numberrrUrZr$rXZread_file_chunkrrrrDs      z3UploadFilenameInputManager.yield_upload_part_bodiescCst|||jjd}|S)N)Z open_function)r r3open)rr$rZrrr_get_deferred_open_filesz2UploadFilenameInputManager._get_deferred_open_filecCs"|jjj}|jj}||d|fS)Nr)rOrPr$rQra)rr>r$rQrrrrV#s zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizecKs |d}|d}||||fS)NrZrT)ra)rr$r/rZrXrrrr^(szBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizecCstt|jjt|S)N)intmathceilrOrQfloat)rr>r[rrrr\-sz)UploadFilenameInputManager._get_num_partsN)rrr r1rKr:r=r?rArBrDrarVr^r\rrrrrLs rLc@s<eZdZdZeddZddZddZdd Zd d Z d S) UploadSeekableInputManagerz&Upload utility for an open file objectcCst|ot|S)N)rr)r8r9rrrr:4sz(UploadSeekableInputManager.is_compatiblecCs|dkr dSdSdS)N put_objectFTr)rr<rrrr=8sz0UploadSeekableInputManager.stores_body_in_memorycCsD|jjj}|}|dd|}|||j||dS)Nr)rOrPr$r+r)r?)rr>r$Zstart_positionZ end_positionrrrr?>s   z0UploadSeekableInputManager.provide_transfer_sizecKs ||d}t|t|fS)Nr[)r'rBytesIOlen)rr$r/datarrrr^JszBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizecCs"|jjj}||jj}||fS)N)rOrPr$r+rQ)rr>r$rQrrrrVYs zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN) rrr r1rKr:r=r?r^rVrrrrrf2s   rfcsheZdZdZdfdd ZeddZddZd d Zd d Z d dZ ddZ dddZ ddZ ZS)UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Ncstt||||d|_dS)N)superrlr _initial_data)rr5r%r6) __class__rrrcs  z&UploadNonSeekableInputManager.__init__cCst|S)N)r)r8r9rrrr:hsz+UploadNonSeekableInputManager.is_compatiblecCsdS)NTr)rr<rrrr=lsz3UploadNonSeekableInputManager.stores_body_in_memorycCsdS)Nr)rr>rrrr?osz3UploadNonSeekableInputManager.provide_transfer_sizecCsT|jjdk r|jj|jkS|jjj}|j}|||d|_t|j|krLdSdSdS)NFT)rOrQrRrPr$_readrorj)rr>r@r$rrrrrAts  z7UploadNonSeekableInputManager.requires_multipart_uploadcCs@||}||}|jjj}||j|||}d|_|S)N)rGrJrOrPr$ _wrap_dataror')rr>rrUr$bodyrrrrBs   z1UploadNonSeekableInputManager.get_put_object_bodyc csb|jjj}d}xN||}||}|d7}|||}|s>P||||}d}||fVqWdS)NrrY)rOrPr$rGrJrqrr) rr>rCZ file_objectr_rrUZ part_contentZ part_objectrrrrDs     z6UploadNonSeekableInputManager.yield_upload_part_bodiesTcCsxt|jdkr||S|t|jkrL|jd|}|rH|j|d|_|S|t|j}|j||}|rtd|_|S)a= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. rNrm)rjror')rr$r(truncaterkZamount_to_readrrrrqs z#UploadNonSeekableInputManager._readcCs.|t|}|jj|t|t|||dS)a Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. )r$rSrTrrU)rErrir3rWrj)rrkrrUr$rrrrrsz(UploadNonSeekableInputManager._wrap_data)N)T)rrr r1rrKr:r=r?rArBrDrqrr __classcell__rr)rprrlas   *rlc@s\eZdZdZddddgZdgZddZdd d Zd d Zd dZ ddZ ddZ ddZ dS)UploadSubmissionTaskz.Task for submitting tasks to execute an uploadZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5Z RequestPayercCsHtttg}|jjj}x|D]}||r|SqWtd|t|fdS)aoRetrieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. z&Input %s of type: %s is not supported.N) rLrfrlrOrPr$r: RuntimeErrortype)rr>Zupload_manager_resolver_chainr$Zupload_manager_clsrrr_get_upload_input_manager_clss    z2UploadSubmissionTask._get_upload_input_manager_clsNcCsd||||j|}|jjdkr*|||||sL|||||||n|||||||dS)a :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N)ryr#rOrQr?rA_submit_upload_request_submit_multipart_request)rclientr@r5request_executorr>r6upload_input_managerrrr_submit s     zUploadSubmissionTask._submitc CsN|jj}||d}|jj|t|j||||j|j|j ddd|ddS)Nrg)r|r$bucketkey extra_argsT)r% main_kwargsis_final)tag) rOrP_get_upload_task_tagr#submit PutObjectTaskrBrrr) rr|r@r5r}r>r~rPZput_object_tagrrrrz2sz+UploadSubmissionTask._submit_upload_requestcCs|jj}|j|t|j||j|j|jdd}g} ||j} | |d} |jj } t } | |j | }|||}xJ|D]B\}}| |jj|t|j|||j|j|| dd|id| dqzW||j}|j|t|j||j|j|d|| dd d dS) N)r|rrr)r%r upload_part)r|r$rrr_r upload_id)r%rpending_main_kwargs)r)rpartsT)r%rrr)rOrPr#rrrrr_extra_upload_part_argsrrQr Zadjust_chunksizeZmultipart_chunksizerDappendUploadPartTask_extra_complete_multipart_argsr )rr|r@r5r}r>r~rPZcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagrQZadjusterrCZ part_iteratorr_r$Zcomplete_multipart_extra_argsrrrr{LsZ  z.UploadSubmissionTask._submit_multipart_requestcCs t||jS)N)r UPLOAD_PART_ARGS)rrrrrrsz,UploadSubmissionTask._extra_upload_part_argscCs t||jS)N)r COMPLETE_MULTIPART_ARGS)rrrrrrsz3UploadSubmissionTask._extra_complete_multipart_argscCsd}||rt}|S)N)r=r)rr~r<rrrrrs z)UploadSubmissionTask._get_upload_task_tag)N) rrr r1rrryrrzr{rrrrrrrrvs &Mrvc@seZdZdZddZdS)rz Task to do a nonmultipart uploadc Cs,|}|jf|||d|WdQRXdS)aP :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the upload. )BucketKeyBodyN)rg)rr|r$rrrrsrrr_mains zPutObjectTask._mainN)rrr r1rrrrrrsrc@seZdZdZddZdS)rz+Task to upload a part in a multipart uploadc Cs>|"}|jf|||||d|} WdQRX| d} | |dS)a :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param part_number: The number representing the part of the multipart upload :param extra_args: A dictionary of any extra arguments that may be used in the upload. :rtype: dict :returns: A dictionary representing a part:: {'Etag': etag_value, 'PartNumber': part_number} This value can be appended to a list to be used to complete the multipart upload. )rrZUploadId PartNumberrNETag)rr)r) rr|r$rrrr_rrsresponseetagrrrrszUploadPartTask._mainN)rrr r1rrrrrrsr)rcZbotocore.compatrZs3transfer.compatrrZs3transfer.futuresrZs3transfer.tasksrrrr Zs3transfer.utilsr r r r objectrr!r2rLrfrlrvrrrrrr s*        !+yQ/D