ó V!¶\c@s‰ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddl mZddl mZdd l mZdd l mZdd l mZdd l mZdd l mZddl mZddlmZddlmZejeƒZdefd„ƒYZdefd„ƒYZdefd„ƒYZ defd„ƒYZ!de!fd„ƒYZ"defd„ƒYZ#defd„ƒYZ$de$fd „ƒYZ%d!efd"„ƒYZ&d#efd$„ƒYZ'd%efd&„ƒYZ(d'efd(„ƒYZ)d)efd*„ƒYZ*d+efd,„ƒYZ+d-efd.„ƒYZ,dS(/iÿÿÿÿN(tsix(tseekable(tRetriesExceededError(tIN_MEMORY_DOWNLOAD_TAG(tS3_RETRYABLE_DOWNLOAD_ERRORS(t get_callbacks(tinvoke_progress_callbacks(tcalculate_num_parts(tcalculate_range_parameter(tFunctionContainer(tCountCallbackInvoker(tStreamReaderProgress(tDeferredOpenFile(tTask(tSubmissionTasktDownloadOutputManagercBs\eZdZd„Zed„ƒZd„Zd„Zd„Zd„Z d„Z d„Z RS( söBase manager class for handling various types of files for downloads This class is typically used for the DownloadSubmissionTask class to help determine the following: * Provides the fileobj to write to downloads to * Get a task to complete once everything downloaded has been written The answers/implementations differ for the various types of file outputs that may be accepted. All implementations must subclass and override public methods from this class. cCs||_||_||_dS(N(t_osutilt_transfer_coordinatort _io_executor(tselftosutilttransfer_coordinatort io_executor((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt__init__6s  cCstdƒ‚dS(seDetermines if the target for the download is compatible with manager :param download_target: The target for which the upload will write data to. :param osutil: The os utility to be used for the transfer :returns: True if the manager can handle the type of target specified otherwise returns False. smust implement is_compatible()N(tNotImplementedError(tclstdownload_targetR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt is_compatible;s cCsdS(s¨Get the tag (if any) to associate all GetObjectTasks :rtype: s3transfer.futures.TaskTag :returns: The tag to associate all GetObjectTasks with N(tNone(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pytget_download_task_tagIscCstdƒ‚dS(sGet file-like object to use for io writes in the io executor :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request returns: A file-like object to write to s*must implement get_fileobj_for_io_writes()N(R(Rttransfer_future((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pytget_fileobj_for_io_writesQscCs)|jj|j|j|||ƒƒdS(sQueue IO write for submission to the IO executor. This method accepts an IO executor and information about the downloaded data, and handles submitting this to the IO executor. This method may defer submission to the IO executor if necessary. N(RtsubmitRtget_io_write_task(Rtfileobjtdatatoffset((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pytqueue_file_io_task[s cCs(t|jdi|d6|d6|d6ƒS(sGet an IO write task for the requested set of data This task can be ran immediately or be submitted to the IO executor for it to run. :type fileobj: file-like object :param fileobj: The file-like object to write to :type data: bytes :param data: The data to write out :type offset: integer :param offset: The offset to write the data to in the file-like object :returns: An IO task to be used to write data to a file-like object t main_kwargsR"R#R$(t IOWriteTaskR(RR"R#R$((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR!is  cCstdƒ‚dS(s©Get the final io task to complete the download This is needed because based on the architecture of the TransferManager the final tasks will be sent to the IO executor, but the executor needs a final task for it to signal that the transfer is done and all done callbacks can be run. :rtype: s3transfer.tasks.Task :returns: A final task to completed in the io executor s"must implement get_final_io_task()N(R(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pytget_final_io_taskƒs cCs5t|ddd|jjƒ}|jj|jƒ|S(Ntmodetwbt open_function(R RtopenRtadd_failure_cleanuptclose(Rtfilenametf((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt_get_fileobj_from_filename‘s( t__name__t __module__t__doc__Rt classmethodRRRR%R!R(R1(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR)s      tDownloadFilenameOutputManagercBs;eZd„Zed„ƒZd„Zd„Zd„ZRS(cCs;tt|ƒj|||ƒd|_d|_d|_dS(N(tsuperR6RRt_final_filenamet_temp_filenamet _temp_fileobj(RRRR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR›s    cCst|tjƒS(N(t isinstanceRt string_types(RRR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢scCsC|jjj}||_|jj|ƒ|_|jƒ|_|jS(N( tmetat call_argsR"R8Rtget_temp_filenameR9t_get_temp_fileobjR:(RRR"((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¦s  cCs:td|jdi|jd6|jd6|jd6dtƒS(NRR&R"tfinal_filenameRtis_final(tIORenameFileTaskRR:R8RtTrue(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR(­s    cCs2|j|jƒ}|jj|jj|jƒ|S(N(R1R9RR-Rt remove_file(RR0((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR@»s (R2R3RR5RRR(R@(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR6šs    tDownloadSeekableOutputManagercBs)eZed„ƒZd„Zd„ZRS(cCs t|ƒS(N(R(RRR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRÃscCs |jjjS(N(R=R>R"(RR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRÇscCstd|jƒS(NR(tCompleteDownloadNOOPTaskR(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR(Ës(R2R3R5RRR((((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRFÂs t DownloadNonSeekableOutputManagercBsPeZdd„Zed„ƒZd„Zd„Zd„Zd„Z d„Z RS(cCsPtt|ƒj|||ƒ|dkr4tƒ}n||_tjƒ|_dS(N( R7RHRRt DeferQueuet _defer_queuet threadingtLockt_io_submit_lock(RRRRt defer_queue((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRÓs     cCs t|dƒS(Ntwrite(thasattr(RRR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRÜscCstS(N(R(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRàscCs |jjjS(N(R=R>R"(RR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRãscCstd|jƒS(NR(RGR(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR(æsc Csw|jh|jj||ƒ}xK|D]C}|d}tjd|d|ƒtt|ƒj|||ƒq&WWdQXdS(NR#s%Queueing IO offset %s for fileobj: %sR$(RMRJtrequest_writestloggertdebugR7RHR%(RR"R#R$twritesRO((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR%ês     cCs!t|jdi|d6|d6ƒS(NR&R"R#(tIOStreamingWriteTaskR(RR"R#R$((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR!õs  N( R2R3RRR5RRRR(R%R!(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRHÒs     t$DownloadSpecialFilenameOutputManagercBs5eZdd„Zed„ƒZd„Zd„ZRS(cCs,tt|ƒj||||ƒd|_dS(N(R7RVRRt_fileobj(RRRRRN((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRscCst|tjƒo|j|ƒS(N(R;RR<tis_special_file(RRR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRscCs(|jjj}|j|ƒ|_|jS(N(R=R>R"R1RW(RRR/((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR sc Cs&td|jdtdi|jd6ƒS(NRRBR&R"(t IOCloseTaskRRDRW(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR(s N(R2R3RRR5RRR((((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRVÿs  tDownloadSubmissionTaskcBsGeZdZd„Zdd„Zd„Zd„Zd„Zd„Z RS(s/Task for submitting tasks to execute a downloadcCshttttg}|jjj}x$|D]}|j||ƒr(|Sq(Wtd|t |ƒfƒ‚dS(sÔRetrieves a class for managing output for a download :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :type osutil: s3transfer.utils.OSUtils :param osutil: The os utility associated to the transfer :rtype: class of DownloadOutputManager :returns: The appropriate class to use for managing a specific type of input for downloads. s'Output %s of type: %s is not supported.N( RVR6RFRHR=R>R"Rt RuntimeErrorttype(RRRtdownload_manager_resolver_chainR"tdownload_manager_cls((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt _get_download_output_manager_clss  c Csà|jjdkr_|jd|jjjd|jjj|jjj}|jj|dƒn|j ||ƒ||j |ƒ} |jj|j krº|j |||||| ||ƒn"|j |||||| ||ƒdS(s :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type io_executor: s3transfer.futures.BoundedExecutor :param io_executor: The io executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for :type bandwidth_limiter: s3transfer.bandwidth.BandwidthLimiter :param bandwidth_limiter: The bandwidth limiter to use when downloading streams tBuckettKeyt ContentLengthN(R=tsizeRt head_objectR>tbuckettkeyt extra_argstprovide_transfer_sizeR_Rtmultipart_thresholdt_submit_download_requestt_submit_ranged_download_request( RtclienttconfigRtrequest_executorRRtbandwidth_limitertresponsetdownload_output_manager((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt_submit7s"    c  CsÏ|jj} |j|ƒ} t|dƒ} |jƒ} |jƒ} |jj|td|jdi |d6| j d6| j d6| d6| j d6| d 6|j d 6|d 6|j d 6|d 6d| gƒd| ƒdS(NtprogressRR&RlReRfR"Rgt callbackst max_attemptsRqt io_chunksizeRotdone_callbacksttag(R=R>RRRR(RR tImmediatelyWriteIOGetObjectTaskReRfRgtnum_download_attemptsRv(RRlRmRRnRRqRRoR>R"tprogress_callbackstget_object_tagt final_task((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRjms.             c  Cs^|jj} t|dƒ} |j|ƒ} |j} t|jj| ƒ} |jƒ}t|j ||ƒƒ}xát | ƒD]Ó}t | || ƒ}i|d6}|j | j ƒ|jƒ|jj|td|jdi |d6| jd6| jd6| d6|d 6| d 6|jd 6|| d 6|d 6|jd6|d6d|jgƒd|ƒqyW|jƒdS(NRstRangeRR&RlReRfR"RgRtRut start_indexRqRvRoRwRx(R=R>RRtmultipart_chunksizeRRcRR t&_get_final_io_task_submission_callbacktrangeRtupdateRgt incrementRR t GetObjectTaskReRfRzRvt decrementtfinalize(RRlRmRRnRRqRRoR>R{R"t part_sizet num_partsR|tfinalize_download_invokertitrange_parameterRg((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRk—sF               cCs"|jƒ}t|jj||ƒS(N(R(R RR (Rtdownload_managerRR}((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRÖs cCsE||}||dkr#d}n||d}d||f}|S(Nits bytes=%s-%s((RRˆt part_indexR‰t start_ranget end_ranget range_param((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt_calculate_range_paramÜs   N( R2R3R4R_RRrRjRkRR“(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRZs  5 * ? R…cBs eZddd„Zd„ZRS(ic Cs/d} xt|ƒD]} y¶|jd|d||}t|d|ƒ}| rh| j||jƒ}n| }t|| ƒ}xJ|D]B}|jjƒsÂ|j||||ƒ|t |ƒ7}q„dSq„WdSWqt k r}t j d|| |dt ƒ|} t|| |ƒqqXqWt| ƒ‚dS(sßDownloads an object and places content into io queue :param client: The client to use when calling GetObject :param bucket: The bucket to download from :param key: The key to download from :param fileobj: The file handle to write content to :param exta_args: Any extra arguements to include in GetObject request :param callbacks: List of progress callbacks to invoke on download :param max_attempts: The number of retries to do when downloading :param download_output_manager: The download output manager associated with the current download. :param io_chunksize: The size of each io chunk to read from the download stream and queue in the io queue. :param start_index: The location in the file to start writing the content of the key to. :param bandwidth_limiter: The bandwidth limiter to use when throttling the downloading of data in streams. R`RatBodyNsCRetrying exception caught (%s), retrying request, (attempt %s / %s)texc_info(RR‚t get_objectR tget_bandwith_limited_streamRtDownloadChunkIteratortdonet _handle_iotlenRRRRSRDRR(RRlReRfR"RgRtRuRqRvRRotlast_exceptionR‹Rptstreaming_bodyt current_indextchunkstchunkte((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt_mainès:      cCs|j|||ƒdS(N(R%(RRqR"R tindex((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRš%sN(R2R3RR¢Rš(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR…çs;RycBseZdZd„ZRS(sGetObjectTask that immediately writes to the provided file object This is useful for downloads where it is known only one thread is downloading the object so there is no reason to go through the overhead of using an IO queue and executor. cCs |j|||ƒ}|ƒdS(N(R!(RRqR"R R£ttask((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRš0s(R2R3R4Rš(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRy)sR'cBseZd„ZRS(cCs|j|ƒ|j|ƒdS(sÖPulls off an io queue to write contents to a file :param fileobj: The file handle to write content to :param data: The data to write :param offset: The offset to write the data to. N(tseekRO(RR"R#R$((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢6s (R2R3R¢(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR'5sRUcBseZdZd„ZRS(s/Task for writing data to a non-seekable stream.cCs|j|ƒdS(sÛWrite data to a fileobj. Data will be written directly to the fileboj without any prior seeking. :param fileobj: The fileobj to write content to :param data: The data to write N(RO(RR"R#((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢Ds (R2R3R4R¢(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRUAsRCcBseZdZd„ZRS(sA task to rename a temporary file to its final filename :param fileobj: The file handle that content was written to. :param final_filename: The final name of the file to rename to upon completion of writing the contents. :param osutil: OS utility cCs!|jƒ|j|j|ƒdS(N(R.t rename_filetname(RR"RAR((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢Ys (R2R3R4R¢(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRCQsRYcBseZdZd„ZRS(siA task to close out a file once the download is complete. :param fileobj: The fileobj to close. cCs|jƒdS(N(R.(RR"((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢cs(R2R3R4R¢(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRY^sRGcBs,eZdZddded„Zd„ZRS(s¯A NOOP task to serve as an indicator that the download is complete Note that the default for is_final is set to True because this should always be the last task. c Cs5tt|ƒjd|d|d|d|d|ƒdS(NRR&tpending_main_kwargsRwRB(R7RGR(RRR&R¨RwRB((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRms cCsdS(N((R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR¢xsN(R2R3R4RRDRR¢(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRGgs  R˜cBs)eZd„Zd„Zd„ZeZRS(cCs||_||_d|_dS(sžIterator to chunk out a downloaded S3 stream :param body: A readable file-like object :param chunksize: The amount to read each time iN(t_bodyt _chunksizet _num_reads(Rtbodyt chunksize((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR}s  cCs|S(N((R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt__iter__‡scCsN|jj|jƒ}|jd7_|r.|S|jdkrA|Stƒ‚dS(Ni(R©treadRªR«t StopIteration(RR ((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt__next__Šs(R2R3RR®R±tnext(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR˜|s  RIcBs eZdZd„Zd„ZRS(s"IO queue that defers write requests until they are queued sequentially. This class is used to track IO data for a *single* fileobj. You can send data to this queue, and it will defer any IO write requests until it has the next contiguous block available (starting at 0). cCs"g|_tƒ|_d|_dS(Ni(t_writestsett_pending_offsetst _next_offset(R((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyR£s  cCså||jkrgSg}||jkr,gStj|j||fƒ|jj|ƒx‰|jrà|jdd|jkràtj|jƒ}|ji|dd6|dd6ƒ|jj|dƒ|jt |dƒ7_qXW|S(sÐRequest any available writes given new incoming data. You call this method by providing new data along with the offset associated with the data. If that new data unlocks any contiguous writes that can now be submitted, this method will return all applicable writes. This is done with 1 method call so you don't have to make two method calls (put(), get()) which acquires a lock each method call. iR$iR#( R¶RµtheapqtheappushR³taddtheappoptappendtremoveR›(RR$R#RTt next_write((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRQ¨s &#(R2R3R4RRQ(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyRIšs (-tloggingtostsocketRKR·tbotocore.compatRts3transfer.compatRts3transfer.exceptionsRts3transfer.futuresRts3transfer.utilsRRRRRR R R R ts3transfer.tasksR Rt getLoggerR2RRtobjectRR6RFRHRVRZR…RyR'RURCRYRGR˜RI(((s9/tmp/pip-install-usGedi/s3transfer/s3transfer/download.pyt sF     q(-ÏB