ó 9(Zc@srddlZddlmZddlmZmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd l mZdd l mZmZd efd „ƒYZd efd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZdS(iÿÿÿÿN(tsix(tseekabletreadable(tIN_MEMORY_UPLOAD_TAG(tTask(tSubmissionTask(tCreateMultipartUploadTask(tCompleteMultipartUploadTask(t get_callbacks(tDeferredOpenFiletChunksizeAdjustertAggregatedProgressCallbackcBs/eZdd„Zd„Zd„Zd„ZRS(iicCs||_||_d|_dS(sØAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total iN(t _callbackst _thresholdt _bytes_seen(tselft callbackst threshold((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt__init__s  cCs2|j|7_|j|jkr.|jƒndS(N(RR t_trigger_callbacks(Rtbytes_transferred((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt__call__,scCs |jdkr|jƒndS(s@Flushes out any progress that has not been sent to its callbacksiN(RR(R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytflush1scCs1x!|jD]}|d|jƒq Wd|_dS(NRi(R R(Rtcallback((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR6si(t__name__t __module__RRRR(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR s   tInterruptReadercBsPeZdZd„Zdd„Zd„Zd„Zd„Zd„Z d„Z RS( sÏWrapper that can interrupt reading using an error It uses a transfer coordinator to propogate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dS(N(t_fileobjt_transfer_coordinator(Rtfileobjttransfer_coordinator((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRIs cCs+|jjr|jj‚n|jj|ƒS(N(Rt exceptionRtread(Rtamount((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR Ms cCs|jj|ƒdS(N(Rtseek(Rtwhere((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR"WscCs |jjƒS(N(Rttell(R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR$ZscCs|jjƒdS(N(Rtclose(R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR%]scCs|S(N((R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt __enter__`scOs|jƒdS(N(R%(Rtargstkwargs((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt__exit__csN( RRt__doc__RtNoneR R"R$R%R&R)(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR<s      tUploadInputManagercBsqeZdZd d„Zed„ƒZd„Zd„Zd„Z d„Z d„Z d„Z d „Z d „ZRS( sJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. cCs||_||_||_dS(N(t_osutilRt_bandwidth_limiter(RtosutilRtbandwidth_limiter((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRvs  cCstdƒ‚dS(sDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. smust implement _is_compatible()N(tNotImplementedError(tclst upload_source((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt is_compatible{s cCstdƒ‚dS(sÛWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. s%must implement store_body_in_memory()N(tNotImplemented(Rtoperation_name((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytstores_body_in_memory‡s cCstdƒ‚dS(s¼Provides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request s&must implement provide_transfer_size()N(R1(Rttransfer_future((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytprovide_transfer_size–scCstdƒ‚dS(sÔDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuartion and size. False, otherwise. s*must implement requires_multipart_upload()N(R1(RR8tconfig((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytrequires_multipart_uploadžs cCstdƒ‚dS(sÜReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. s$must implement get_put_object_body()N(R1(RR8((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytget_put_object_body­s cCstdƒ‚dS(sYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. s)must implement yield_upload_part_bodies()N(R1(RR8t chunksize((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pytyield_upload_part_bodies¼scCs@t||jƒ}|jr<|jj||jdtƒ}n|S(Ntenabled(RRR.tget_bandwith_limited_streamtFalse(RR((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt _wrap_fileobjÌs   cCs&t|dƒ}|r"t|ƒgSgS(Ntprogress(RR (RR8R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt_get_progress_callbacksÓs cCsg|D]}|j^qS(N(R(Rtaggregated_progress_callbacksR((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt_get_close_callbacksÜsN(RRR*R+Rt classmethodR4R7R9R;R<R>RBRDRF(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR,gs        tUploadFilenameInputManagercBsneZdZed„ƒZd„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z d „Z RS( sUpload utility for filenamescCst|tjƒS(N(t isinstanceRt string_types(R2R3((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR4âscCstS(N(RA(RR6((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR7æscCs)|jj|jj|jjjƒƒdS(N(tmetaR9R-t get_file_sizet call_argsR(RR8((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR9és  cCs|jj|jkS(N(RKtsizetmultipart_threshold(RR8R:((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR;îsc Csy|j|ƒ\}}|j|ƒ}|j|ƒ}|j|ƒ}|jj}|jjd|d|d|d|d|ƒS(NRt chunk_sizetfull_file_sizeRtclose_callbacks(t&_get_put_object_fileobj_with_full_sizeRBRDRFRKRNR-t#open_file_chunk_reader_from_fileobj(RR8Rt full_sizeRRRRN((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR<ñs  c csã|jj}|j||ƒ}x¾td|dƒD]©}|j|ƒ}|j|ƒ}||d}|j|jjjd|d|d|ƒ\} } |j | ƒ} |j j d| d|d| d|d|ƒ} || fVq2WdS( Nit start_bytet part_sizeRQRRPRRR( RKRNt_get_num_partstrangeRDRFt'_get_upload_part_fileobj_with_full_sizeRMRRBR-RT( RR8R=RQt num_partst part_numberRRRRVRRUtread_file_chunk((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR>s     cCst||d|jjƒ}|S(Nt open_function(R R-topen(RRRV((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt_get_deferred_open_filescCs1|jjj}|jj}|j|dƒ|fS(Ni(RKRMRRNR`(RR8RRN((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRS"s cKs*|d}|d}|j||ƒ|fS(NRVRQ(R`(RRR(RVRU((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRZ's  cCs#ttj|jjt|ƒƒƒS(N(tinttmathtceilRKRNtfloat(RR8RW((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRX,s(RRR*RGR4R7R9R;R<R>R`RSRZRX(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRHàs        tUploadSeekableInputManagercBsAeZdZed„ƒZd„Zd„Zd„Zd„ZRS(s&Upload utility for an open file objectcCst|ƒot|ƒS(N(RR(R2R3((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR43scCs|dkrtStSdS(Nt put_object(RAtTrue(RR6((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR77s cCs\|jjj}|jƒ}|jddƒ|jƒ}|j|ƒ|jj||ƒdS(Nii(RKRMRR$R"R9(RR8Rtstart_positiont end_position((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR9=s    cKs,|j|dƒ}tj|ƒt|ƒfS(NRW(R RtBytesIOtlen(RRR(tdata((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRZIscCs/|jjj}|jƒ|jj}||fS(N(RKRMRR$RN(RR8RRN((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRSXs( RRR*RGR4R7R9RZRS(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRe1s   tUploadNonSeekableInputManagercBskeZdZd d„Zed„ƒZd„Zd„Zd„Z d„Z d„Z e d„Z d „ZRS( s7Upload utility for a file-like object that cannot seek.cCs)tt|ƒj|||ƒd|_dS(Nt(tsuperRmRt _initial_data(RR/RR0((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRbs cCs t|ƒS(N(R(R2R3((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR4gscCstS(N(Rg(RR6((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR7kscCsdS(N((RR8((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR9nscCsv|jjdk r%|jj|jkS|jjj}|j}|j||tƒ|_t |jƒ|krntSt SdS(N( RKRNR+RORMRt_readRARpRkRg(RR8R:RR((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR;ss cCs\|j|ƒ}|j|ƒ}|jjj}|j|j|jƒ||ƒ}d|_|S(N( RDRFRKRMRt _wrap_dataRpR R+(RR8RRRRtbody((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR<„s c cs|jjj}d}xttr‹|j|ƒ}|j|ƒ}|d7}|j||ƒ}|sbPn|j|||ƒ}d}||fVqWdS(Nii( RKRMRRgRDRFRqRrR+( RR8R=t file_objectR\RRRt part_contentt part_object((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR>‘s  cCs t|jƒdkr"|j|ƒS|t|jƒkra|j| }|r]|j||_n|S|t|jƒ}|j|j|ƒ}|rœd|_n|S(s= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. iRn(RkRpR (RRR!ttruncateRltamount_to_read((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRq¤s   c CsO|jtj|ƒƒ}|jjd|dt|ƒdt|ƒd|d|ƒS(s¸ Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. RRPRQRRR(RBRRjR-RTRk(RRlRRRR((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRrÎs N(RRR*R+RRGR4R7R9R;R<R>RgRqRr(((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRm`s      *tUploadSubmissionTaskcBsYeZdZddddgZd„Zd d„Zd„Zd„Zd „Z d „Z RS( s.Task for submitting tasks to execute an uploadtSSECustomerKeytSSECustomerAlgorithmtSSECustomerKeyMD5t RequestPayercCsbtttg}|jjj}x!|D]}|j|ƒr%|Sq%Wtd|t|ƒfƒ‚dS(snRetieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. s&Input %s of type: %s is not supported.N( RHReRmRKRMRR4t RuntimeErrorttype(RR8tupload_manager_resolver_chainRtupload_manager_cls((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt_get_upload_input_manager_clsîs   cCs‘|j|ƒ||j|ƒ}|jjdkr@|j|ƒn|j||ƒsq|j||||||ƒn|j||||||ƒdS(sÒ :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N( R‚RRKRNR+R9R;t_submit_upload_requestt_submit_multipart_request(RtclientR:R/trequest_executorR8R0tupload_input_manager((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyt_submits  c Csˆ|jj}|j|dƒ}|jj|td|jdi|d6|j|ƒd6|jd6|jd6|j d6d t ƒd |ƒdS( NRfRt main_kwargsR…Rtbuckettkeyt extra_argstis_finalttag( RKRMt_get_upload_task_tagRtsubmitt PutObjectTaskR<RŠR‹RŒRg( RR…R:R/R†R8R‡RMtput_object_tag((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyRƒ-s          c Cs¤|jj}|jj|td|jdi|d6|jd6|jd6|jd6ƒƒ}g} |j|jƒ} |j |dƒ} |jj } t ƒ} | j |j | ƒ}|j||ƒ}x…|D]}\}}| j|jj|td|jdi|d6|d6|jd6|jd6|d 6| d6d i|d 6ƒd | ƒƒqÅW|jj|td|jdi|d6|jd6|jd6d i|d 6| d 6dtƒƒdS(NRR‰R…RŠR‹RŒt upload_partRR\tpending_main_kwargst upload_idRŽtpartsR(RKRMRRRRŠR‹RŒt_extra_upload_part_argsRRNR tadjust_chunksizetmultipart_chunksizeR>tappendtUploadPartTaskRRg(RR…R:R/R†R8R‡RMtcreate_multipart_futuret part_futurestextra_part_argstupload_part_tagRNtadjusterR=t part_iteratorR\R((s5/tmp/pip-build-wDUJoH/s3transfer/s3transfer/upload.pyR„Gs^                    cCsCi}x6|jƒD](\}}||jkr||| s$ !+yQ/„½