ó V!¶\c@s‚ddlZddlmZddlmZmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd l mZdd l mZdd l mZmZd efd „ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZdS(iÿÿÿÿN(tsix(tseekabletreadable(tIN_MEMORY_UPLOAD_TAG(tTask(tSubmissionTask(tCreateMultipartUploadTask(tCompleteMultipartUploadTask(t get_callbacks(tget_filtered_dict(tDeferredOpenFiletChunksizeAdjustertAggregatedProgressCallbackcBs/eZdd„Zd„Zd„Zd„ZRS(iicCs||_||_d|_dS(sØAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total iN(t _callbackst _thresholdt _bytes_seen(tselft callbackst threshold((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt__init__s  cCs2|j|7_|j|jkr.|jƒndS(N(RRt_trigger_callbacks(Rtbytes_transferred((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt__call__-scCs |jdkr|jƒndS(s@Flushes out any progress that has not been sent to its callbacksiN(RR(R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytflush2scCs1x!|jD]}|d|jƒq Wd|_dS(NRi(R R(Rtcallback((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR7si(t__name__t __module__RRRR(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR s   tInterruptReadercBsPeZdZd„Zdd„Zd„Zd„Zd„Zd„Z d„Z RS( sÏWrapper that can interrupt reading using an error It uses a transfer coordinator to propagate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dS(N(t_fileobjt_transfer_coordinator(Rtfileobjttransfer_coordinator((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRJs cCs+|jjr|jj‚n|jj|ƒS(N(Rt exceptionRtread(Rtamount((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR!Ns cCs|jj|ƒdS(N(Rtseek(Rtwhere((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR#XscCs |jjƒS(N(Rttell(R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR%[scCs|jjƒdS(N(Rtclose(R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR&^scCs|S(N((R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt __enter__ascOs|jƒdS(N(R&(Rtargstkwargs((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt__exit__dsN( RRt__doc__RtNoneR!R#R%R&R'R*(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR=s      tUploadInputManagercBsqeZdZd d„Zed„ƒZd„Zd„Zd„Z d„Z d„Z d„Z d „Z d „ZRS( sJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. cCs||_||_||_dS(N(t_osutilRt_bandwidth_limiter(RtosutilRtbandwidth_limiter((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRws  cCstdƒ‚dS(sDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. smust implement _is_compatible()N(tNotImplementedError(tclst upload_source((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt is_compatible|s cCstdƒ‚dS(sÛWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. s%must implement store_body_in_memory()N(tNotImplemented(Rtoperation_name((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytstores_body_in_memoryˆs cCstdƒ‚dS(s¼Provides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request s&must implement provide_transfer_size()N(R2(Rttransfer_future((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytprovide_transfer_size—scCstdƒ‚dS(sÔDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuartion and size. False, otherwise. s*must implement requires_multipart_upload()N(R2(RR9tconfig((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytrequires_multipart_uploadŸs cCstdƒ‚dS(sÜReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. s$must implement get_put_object_body()N(R2(RR9((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytget_put_object_body®s cCstdƒ‚dS(sYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. s)must implement yield_upload_part_bodies()N(R2(RR9t chunksize((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pytyield_upload_part_bodies½scCs@t||jƒ}|jr<|jj||jdtƒ}n|S(Ntenabled(RRR/tget_bandwith_limited_streamtFalse(RR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt _wrap_fileobjÍs   cCs&t|dƒ}|r"t|ƒgSgS(Ntprogress(RR (RR9R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_get_progress_callbacksÔs cCsg|D]}|j^qS(N(R(Rtaggregated_progress_callbacksR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_get_close_callbacksÝsN(RRR+R,Rt classmethodR5R8R:R<R=R?RCRERG(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR-hs        tUploadFilenameInputManagercBsneZdZed„ƒZd„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z d „Z RS( sUpload utility for filenamescCst|tjƒS(N(t isinstanceRt string_types(R3R4((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR5ãscCstS(N(RB(RR7((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR8çscCs)|jj|jj|jjjƒƒdS(N(tmetaR:R.t get_file_sizet call_argsR(RR9((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR:ês  cCs|jj|jkS(N(RLtsizetmultipart_threshold(RR9R;((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR<ïsc Csy|j|ƒ\}}|j|ƒ}|j|ƒ}|j|ƒ}|jj}|jjd|d|d|d|d|ƒS(NRt chunk_sizetfull_file_sizeRtclose_callbacks(t&_get_put_object_fileobj_with_full_sizeRCRERGRLROR.t#open_file_chunk_reader_from_fileobj(RR9Rt full_sizeRRSRO((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR=òs  c csã|jj}|j||ƒ}x¾td|dƒD]©}|j|ƒ}|j|ƒ}||d}|j|jjjd|d|d|ƒ\} } |j | ƒ} |j j d| d|d| d|d|ƒ} || fVq2WdS( Nit start_bytet part_sizeRRRRQRRS( RLROt_get_num_partstrangeRERGt'_get_upload_part_fileobj_with_full_sizeRNRRCR.RU( RR9R>RRt num_partst part_numberRRSRWRRVtread_file_chunk((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR?s     cCst||d|jjƒ}|S(Nt open_function(R R.topen(RRRW((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_get_deferred_open_filescCs1|jjj}|jj}|j|dƒ|fS(Ni(RLRNRRORa(RR9RRO((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRT#s cKs*|d}|d}|j||ƒ|fS(NRWRR(Ra(RRR)RWRV((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR[(s  cCs#ttj|jjt|ƒƒƒS(N(tinttmathtceilRLROtfloat(RR9RX((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRY-s(RRR+RHR5R8R:R<R=R?RaRTR[RY(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRIás        tUploadSeekableInputManagercBsAeZdZed„ƒZd„Zd„Zd„Zd„ZRS(s&Upload utility for an open file objectcCst|ƒot|ƒS(N(RR(R3R4((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR54scCs|dkrtStSdS(Nt put_object(RBtTrue(RR7((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR88s cCs\|jjj}|jƒ}|jddƒ|jƒ}|j|ƒ|jj||ƒdS(Nii(RLRNRR%R#R:(RR9Rtstart_positiont end_position((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR:>s    cKs,|j|dƒ}tj|ƒt|ƒfS(NRX(R!RtBytesIOtlen(RRR)tdata((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR[JscCs/|jjj}|jƒ|jj}||fS(N(RLRNRR%RO(RR9RRO((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRTYs( RRR+RHR5R8R:R[RT(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRf2s   tUploadNonSeekableInputManagercBskeZdZd d„Zed„ƒZd„Zd„Zd„Z d„Z d„Z e d„Z d „ZRS( s7Upload utility for a file-like object that cannot seek.cCs)tt|ƒj|||ƒd|_dS(Nt(tsuperRnRt _initial_data(RR0RR1((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRcs cCs t|ƒS(N(R(R3R4((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR5hscCstS(N(Rh(RR7((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR8lscCsdS(N((RR9((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR:oscCsv|jjdk r%|jj|jkS|jjj}|j}|j||tƒ|_t |jƒ|krntSt SdS(N( RLROR,RPRNRt_readRBRqRlRh(RR9R;RR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR<ts cCs\|j|ƒ}|j|ƒ}|jjj}|j|j|jƒ||ƒ}d|_|S(N( RERGRLRNRt _wrap_dataRqR!R,(RR9RRSRtbody((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR=…s c cs|jjj}d}xttr‹|j|ƒ}|j|ƒ}|d7}|j||ƒ}|sbPn|j|||ƒ}d}||fVqWdS(Nii( RLRNRRhRERGRrRsR,( RR9R>t file_objectR]RRSt part_contentt part_object((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR?’s  cCs t|jƒdkr"|j|ƒS|t|jƒkra|j| }|r]|j||_n|S|t|jƒ}|j|j|ƒ}|rœd|_n|S(s= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. iRo(RlRqR!(RRR"ttruncateRmtamount_to_read((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRr¥s   c CsO|jtj|ƒƒ}|jjd|dt|ƒdt|ƒd|d|ƒS(s¸ Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. RRQRRRRS(RCRRkR.RURl(RRmRRSR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRsÏs N(RRR+R,RRHR5R8R:R<R=R?RhRrRs(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRnas      *tUploadSubmissionTaskcBskeZdZddddgZdgZd„Zd d„Zd„Zd„Z d „Z d „Z d „Z RS( s.Task for submitting tasks to execute an uploadtSSECustomerKeytSSECustomerAlgorithmtSSECustomerKeyMD5t RequestPayercCsbtttg}|jjj}x!|D]}|j|ƒr%|Sq%Wtd|t|ƒfƒ‚dS(soRetrieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. s&Input %s of type: %s is not supported.N( RIRfRnRLRNRR5t RuntimeErrorttype(RR9tupload_manager_resolver_chainRtupload_manager_cls((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_get_upload_input_manager_clsós   cCs‘|j|ƒ||j|ƒ}|jjdkr@|j|ƒn|j||ƒsq|j||||||ƒn|j||||||ƒdS(sÒ :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N( RƒRRLROR,R:R<t_submit_upload_requestt_submit_multipart_request(RtclientR;R0trequest_executorR9R1tupload_input_manager((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_submit s  c Csˆ|jj}|j|dƒ}|jj|td|jdi|d6|j|ƒd6|jd6|jd6|j d6d t ƒd |ƒdS( NRgRt main_kwargsR†Rtbuckettkeyt extra_argstis_finalttag( RLRNt_get_upload_task_tagRtsubmitt PutObjectTaskR=R‹RŒRRh( RR†R;R0R‡R9RˆRNtput_object_tag((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR„2s          c Cs½|jj}|jj|td|jdi|d6|jd6|jd6|jd6ƒƒ}g} |j|jƒ} |j |dƒ} |jj } t ƒ} | j |j | ƒ}|j||ƒ}x…|D]}\}}| j|jj|td|jdi|d6|d6|jd6|jd6|d 6| d6d i|d 6ƒd | ƒƒqÅW|j|jƒ}|jj|td|jdi|d6|jd6|jd6|d6d i|d 6| d 6dtƒƒdS(NRRŠR†R‹RŒRt upload_partRR]tpending_main_kwargst upload_idRtpartsRŽ(RLRNRR‘RR‹RŒRt_extra_upload_part_argsRROR tadjust_chunksizetmultipart_chunksizeR?tappendtUploadPartTaskt_extra_complete_multipart_argsRRh(RR†R;R0R‡R9RˆRNtcreate_multipart_futuret part_futurestextra_part_argstupload_part_tagROtadjusterR>t part_iteratorR]Rtcomplete_multipart_extra_args((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR…Lsd                      cCst||jƒS(N(R tUPLOAD_PART_ARGS(RR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR˜™scCst||jƒS(N(R tCOMPLETE_MULTIPART_ARGS(RR((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRžscCs"d}|j|ƒrt}n|S(N(R,R8R(RRˆR7R((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR¡s N( RRR+R¥R¦RƒR,R‰R„R…R˜RR(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRzås    &  M  R’cBseZdZd„ZRS(s Task to do a nonmultipart uploadc Cs2|&}|jd|d|d||WdQXdS(sP :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the upload. tBuckettKeytBodyN(Rg(RR†RR‹RŒRRt((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt_mainªs (RRR+Rª(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyR’¨sRœcBseZdZd„ZRS(s+Task to upload a part in a multipart uploadc CsX|4}|jd|d|d|d|d||} WdQX| d} i| d6|d6S(sÓ :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param part_number: The number representing the part of the multipart upload :param extra_args: A dictionary of any extra arguments that may be used in the upload. :rtype: dict :returns: A dictionary representing a part:: {'Etag': etag_value, 'PartNumber': part_number} This value can be appended to a list to be used to complete the multipart upload. R§R¨tUploadIdt PartNumberR©NtETag(R”( RR†RR‹RŒR–R]RRttresponsetetag((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRª¹s     (RRR+Rª(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyRœ·s(Rctbotocore.compatRts3transfer.compatRRts3transfer.futuresRts3transfer.tasksRRRRts3transfer.utilsRR R R tobjectR RR-RIRfRnRzR’Rœ(((s7/tmp/pip-install-usGedi/s3transfer/s3transfer/upload.pyt s& !+yQ/„Ã