ó V!¶\c@sÌddlZddlZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd „ƒYZd efd „ƒYZd efd „ƒYZ defd„ƒYZ defd„ƒYZ dS(iÿÿÿÿNtRequestExceededExceptioncBseZd„ZRS(cCs<||_||_d||f}tt|ƒj|ƒdS(s Error when requested amount exceeds what is allowed The request that raised this error should be retried after waiting the time specified by ``retry_time``. :type requested_amt: int :param requested_amt: The originally requested byte amount :type retry_time: float :param retry_time: The length in time to wait to retry for the requested amount s<Request amount %s exceeded the amount available. Retry in %sN(t requested_amtt retry_timetsuperRt__init__(tselfRRtmsg((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRs   (t__name__t __module__R(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRst RequestTokencBseZdZRS(sDA token to pass as an identifier when consuming from the LeakyBucket(RRt__doc__(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR (st TimeUtilscBseZd„Zd„ZRS(cCs tjƒS(sgGet the current time back :rtype: float :returns: The current time in seconds (ttime(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR .scCs tj|ƒS(swSleep for a designated time :type value: float :param value: The time to sleep for in seconds (R tsleep(Rtvalue((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR 6s(RRR R (((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR -s tBandwidthLimitercBs eZdd„Zed„ZRS(cCs1||_||_|dkr-tƒ|_ndS(sLimits bandwidth for shared S3 transfers :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use limit bandwidth :type time_utils: TimeUtils :param time_utils: Time utility to use for interacting with time. N(t _leaky_buckett _time_utilstNoneR (Rt leaky_buckett time_utils((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR@s   cCs2t||j||jƒ}|s.|jƒn|S(sÎWraps a fileobj in a bandwidth limited stream wrapper :type fileobj: file-like obj :param fileobj: The file-like obj to wrap :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type enabled: boolean :param enabled: Whether bandwidth limiting should be enabled to start (tBandwidthLimitedStreamRRtdisable_bandwidth_limiting(Rtfileobjttransfer_coordinatortenabledtstream((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pytget_bandwith_limited_streamNs    N(RRRRtTrueR(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR?s RcBszeZddd„Zd„Zd„Zd„Zd„Zd„Zd„Z d „Z d „Z d „Z d „Z d „ZRS(iicCsj||_||_||_||_|dkr?tƒ|_nt|_tƒ|_ d|_ ||_ dS(s[Limits bandwidth for reads on a wrapped stream :type fileobj: file-like object :param fileobj: The file like object to wrap :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use to throttle reads on the stream :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time iN( t_fileobjRt_transfer_coordinatorRRR Rt_bandwidth_limiting_enabledR t_request_tokent _bytes_seent_bytes_threshold(RRRRRtbytes_threshold((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRes        cCs t|_dS(s0Enable bandwidth limiting on reads to the streamN(RR(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pytenable_bandwidth_limiting‚scCs t|_dS(s1Disable bandwidth limiting on reads to the streamN(tFalseR(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR†scCsd|js|jj|ƒS|j|7_|j|jkrJ|jj|ƒS|jƒ|jj|ƒS(shRead a specified amount Reads will only be throttled if bandwidth limiting is enabled. (RRtreadR!R"t_consume_through_leaky_bucket(Rtamount((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR&Šs  cCsvxo|jjsey*|jj|j|jƒd|_dSWqtk ra}|jj|j ƒqXqW|jj‚dS(Ni( Rt exceptionRtconsumeR!R RRR R(Rte((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR'žs  cCs|jƒdS(s6Signal that data being read is being transferred to S3N(R$(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pytsignal_transferring¯scCs|jƒdS(s:Signal that data being read is not being transferred to S3N(R(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pytsignal_not_transferring³scCs|jj|ƒdS(N(Rtseek(Rtwhere((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR.·scCs |jjƒS(N(Rttell(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR0ºscCs0|jr|jr|jƒn|jjƒdS(N(RR!R'Rtclose(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR1½s cCs|S(N((R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyt __enter__ÇscOs|jƒdS(N(R1(Rtargstkwargs((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyt__exit__ÊsNi(RRRRR$RR&R'R,R-R.R0R1R2R5(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRds         t LeakyBucketcBsGeZdddd„Zd„Zd„Zd„Zd„Zd„ZRS(cCsŽt|ƒ|_||_|dkr3tƒ|_ntjƒ|_||_|dkrft ƒ|_n||_ |dkrŠt ƒ|_ ndS(s9A leaky bucket abstraction to limit bandwidth consumption :type rate: int :type rate: The maximum rate to allow. This rate is in terms of bytes per second. :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time :type rate_tracker: BandwidthRateTracker :param rate_tracker: Tracks bandwidth consumption :type consumption_scheduler: ConsumptionScheduler :param consumption_scheduler: Schedules consumption retries when necessary N( tfloatt _max_rateRRR t threadingtLockt_lockt _rate_trackertBandwidthRateTrackert_consumption_schedulertConsumptionScheduler(Rtmax_rateRt rate_trackertconsumption_scheduler((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRÏs      cCs€|jq|jjƒ}|jj|ƒr>|j|||ƒS|j||ƒrf|j|||ƒn|j||ƒSWdQXdS(scConsume an a requested amount :type amt: int :param amt: The amount of bytes to request to consume :type request_token: RequestToken :param request_token: The token associated to the consumption request that is used to identify the request. So if a RequestExceededException is raised the token should be used in subsequent retry consume() request. :raises RequestExceededException: If the consumption amount would exceed the maximum allocated bandwidth :rtype: int :returns: The amount consumed N( R;RR R>t is_scheduledt,_release_requested_amt_for_scheduled_requestt_projected_to_exceed_max_ratet!_raise_request_exceeded_exceptiont_release_requested_amt(Rtamtt request_tokenttime_now((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR*ís  cCs"|jj||ƒ}||jkS(N(R<tget_projected_rateR8(RRHRJtprojected_rate((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRE scCs |jj|ƒ|j||ƒS(N(R>tprocess_scheduled_consumptionRG(RRHRIRJ((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRDs cCsD|t|jƒ}|jj|||ƒ}td|d|ƒ‚dS(NRR(R7R8R>tschedule_consumptionR(RRHRIRJtallocated_timeR((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRFs  cCs|jj||ƒ|S(N(R<trecord_consumption_rate(RRHRJ((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRGsN( RRRRR*RERDRFRG(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR6Îs     R?cBs,eZd„Zd„Zd„Zd„ZRS(cCsi|_d|_dS(s*Schedules when to consume a desired amountiN(t _tokens_to_scheduled_consumptiont _total_wait(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR!s cCs ||jkS(sÙIndicates if a consumption request has been scheduled :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. (RQ(Rttoken((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRC&scCs4|j|7_i|jd6|d6|j|<|jS(s´Schedules a wait time to be able to consume an amount :type amt: int :param amt: The amount of bytes scheduled to be consumed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. :type time_to_consume: float :param time_to_consume: The desired time it should take for that specific request amount to be consumed in regardless of previously scheduled consumption requests :rtype: float :returns: The amount of time to wait for the specific request before actually consuming the specified amount. t wait_durationttime_to_consume(RRRQ(RRHRSRU((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRN/s  cCs3|jj|ƒ}t|j|ddƒ|_dS(sàProcesses a scheduled consumption request that has completed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. RUiN(RQtpoptmaxRR(RRStscheduled_retry((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRMIs(RRRRCRNRM(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR? s  R=cBsGeZdd„Zed„ƒZd„Zd„Zd„Zd„ZRS(gš™™™™™é?cCs||_d|_d|_dS(s’Tracks the rate of bandwidth consumption :type a: float :param a: The constant to use in calculating the exponentional moving average of the bandwidth rate. Specifically it is used in the following calculation: current_rate = alpha * new_rate + (1 - alpha) * current_rate This value of this constant should be between 0 and 1. N(t_alphaRt _last_timet _current_rate(Rtalpha((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRVs  cCs|jdkrdS|jS(smThe current transfer rate :rtype: float :returns: The current tracked transfer rate gN(RZRR[(R((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyt current_ratefscCs#|jdkrdS|j||ƒS(sZGet the projected rate using a provided amount and time :type amt: int :param amt: The proposed amount to consume :type time_at_consumption: float :param time_at_consumption: The proposed time to consume at :rtype: float :returns: The consumption rate if that amt and time were consumed gN(RZRt*_calculate_exponential_moving_average_rate(RRHttime_at_consumption((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRKqs cCsG|jdkr%||_d|_dS|j||ƒ|_||_dS(sRecord the consumption rate based off amount and time point :type amt: int :param amt: The amount that got consumed :type time_at_consumption: float :param time_at_consumption: The time at which the amount was consumed gN(RZRR[R^(RRHR_((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyRP‚s   cCs+||j}|dkr#tdƒS||S(Nitinf(RZR7(RRHR_t time_delta((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyt_calculate_rate“s   cCs/|j||ƒ}|j|d|j|jS(Ni(RbRYR[(RRHR_tnew_rate((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR^s( RRRtpropertyR]RKRPRbR^(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyR=Us     ( R R9t ExceptionRtobjectR R RRR6R?R=(((s:/tmp/pip-install-usGedi/s3transfer/s3transfer/bandwidth.pyt s  %jR5