B ㊇ck@sddlZddlZddlZddlmZddlZddlmZddlm Z m Z e e Z GdddZGdddZGd d d ZGd d d ZGd ddZGdddZdS)N)tee)PaginationError) merge_dictsset_value_from_jmespathc@s8eZdZdZddZddZddZdd Zd d Zd S) TokenEncoderabEncodes dictionaries into opaque strings. This for the most part json dumps + base64 encoding, but also supports having bytes in the dictionary in addition to the types that json can handle by default. This is intended for use in encoding pagination tokens, which in some cases can be complex structures and / or contain bytes. c Cs`yt|}Wn:ttfk rH||g\}}||d<t|}YnXt|ddS)a.Encodes a dictionary to an opaque string. :type token: dict :param token: A dictionary containing pagination information, particularly the service pagination token(s) but also other boto metadata. :rtype: str :returns: An opaque string boto_encoded_keyszutf-8) jsondumps TypeErrorUnicodeDecodeError_encodebase64 b64encodeencodedecode)selftoken json_stringZ encoded_token encoded_keysrl/private/var/folders/8c/hx9_v10d5x38qmnzt13b7b8j1k3n5b/T/pip-target-x6xd5gna/lib/python/botocore/paginate.pyr&s zTokenEncoder.encodecCsNt|tr|||St|tr,|||St|trB|||S|gfSdS)z@Encode bytes in given data, keeping track of the path traversed.N) isinstancedict _encode_dictlist _encode_listbytes _encode_bytes)rdatapathrrrr Cs      zTokenEncoder._encodec CsTg}g}xBt|D]6\}}||g}|||\}} |||| qW||fS)z@Encode any bytes in a list, noting the index of what is encoded.) enumerater appendextend) rrrnew_dataencodedivaluenew_path new_value new_encodedrrrrNs  zTokenEncoder._encode_listc CsRi}g}x@|D]4\}}||g}|||\}} |||<|| qW||fS)z@Encode any bytes in a dict, noting the index of what is encoded.)itemsr r") rrrr#r$keyr&r'r(r)rrrrYs zTokenEncoder._encode_dictcCst|d|gfS)zBase64 encode a byte string.zutf-8)r rr)rrrrrrrdszTokenEncoder._encode_bytesN) __name__ __module__ __qualname____doc__rr rrrrrrrrs     rc@s0eZdZdZddZddZddZdd Zd S) TokenDecoderzDecodes token strings back into dictionaries. This performs the inverse operation to the TokenEncoder, accepting opaque strings and decoding them into a useable form. cCsHt|dd}t|}|dd}|dkr8|S|||SdS)adDecodes an opaque string to a dictionary. :type token: str :param token: A token string given by the botocore pagination interface. :rtype: dict :returns: A dictionary containing pagination information, particularly the service pagination token(s) but also other boto metadata. zutf-8rN)r b64decoderrrloadspop_decode)rrrZ decoded_tokenrrrrrps   zTokenDecoder.decodecCs<x6|D].}|||}t|d}||||qW|S)z&Find each encoded value and decode it.zutf-8) _path_getr r1r _path_set)rrrr+r$decodedrrrr4s   zTokenDecoder._decodecCs|}x|D] }||}q W|S)zReturn the nested data at the given path. For instance: data = {'foo': ['bar', 'baz']} path = ['foo', 0] ==> 'bar' r)rrrdsteprrrr5s   zTokenDecoder._path_getcCs$|||dd}|||d<dS)zSet the value of a key in the given data. Example: data = {'foo': ['bar', 'baz']} path = ['foo', 1] value = 'bin' ==> data = {'foo': ['bar', 'bin']} N)r5)rrrr& containerrrrr6s zTokenDecoder._path_setN)r,r-r.r/rr4r5r6rrrrr0is r0c@seZdZddZddZdS)PaginatorModelcCs|d|_dS)NZ pagination)_paginator_config)rZpaginator_configrrr__init__szPaginatorModel.__init__cCs4y|j|}Wn tk r.td|YnX|S)Nz*Paginator for operation does not exist: %s)r=KeyError ValueError)rZoperation_nameZsingle_paginator_configrrr get_paginators zPaginatorModel.get_paginatorN)r,r-r.r>rArrrrr<sr<c@seZdZdZddZeddZeddZejddZed d Z d d Z d dZ ddZ ddZ ddZddZddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)S)* PageIteratorzAn iterable object to pagiante API results. Please note it is NOT a python iterator. Use ``iter`` to wrap this as a generator. c Csb||_||_||_||_||_||_||_| |_| |_| |_ d|_ ||_ i|_ t |_t|_dS)N)_method _input_token _output_token _more_results _result_keys _max_items _limit_key_starting_token _page_size _op_kwargs _resume_token_non_aggregate_key_exprs_non_aggregate_partr_token_encoderr0_token_decoder) rmethod input_token output_token more_results result_keysnon_aggregate_keys limit_key max_itemsZstarting_token page_size op_kwargsrrrr>szPageIterator.__init__cCs|jS)N)rG)rrrrrVszPageIterator.result_keyscCs|jS)z&Token to specify to resume pagination.)rM)rrrr resume_tokenszPageIterator.resume_tokencCsnt|tstd|d|kr0t|jdg}n t|j}t|}||kr^|j||_n td|dS)NzBad starting token: %sboto_truncate_amount) rrr@sortedrDkeysrPrrM)rr&Z token_keys dict_keysrrrr\s    cCs|jS)N)rO)rrrrnon_aggregate_partszPageIterator.non_aggregate_partccs|j}d}dd|jD}|jdk r0|d}d}d}|jd}d}||x0||}||} |r|jdk r|| ||}d}| | nd}| | } | dkrg} t | } d} |j dk r|| |j } | dkr| | || |||VPqT|V|| 7}|| }tdd|Dr(P|j dk rH||j krH||_P|dk rp||krpd|} t| d ||||}qTWdS) NcSsi|] }d|qS)Nr).0r+rrr sz)PageIterator.__iter__..rTFcss|]}|dkVqdS)Nr)rbtrrr 5sz(PageIterator.__iter__..z(The same next token was received twice: )message)rLrDrJ_parse_starting_tokenrV_inject_starting_params _make_request_extract_parsed_response_handle_first_request _record_non_aggregate_key_valuessearchlenrH_truncate_response_get_next_tokenallvaluesr\r_inject_token_into_kwargs)rcurrent_kwargsZprevious_next_token next_tokenZ total_itemsZ first_requestprimary_result_keystarting_truncationresponseparsedZcurrent_responseZnum_current_responsetruncate_amountrfrrr__iter__sf                   zPageIterator.__iter__ccsBt|}x2|D]*}||}t|tr4|EdHq|VqWdS)aApplies a JMESPath expression to a paginator Each page of results is searched using the provided JMESPath expression. If the result is not a list, it is yielded directly. If the result is a list, each element in the result is yielded individually (essentially implementing a flatmap in which the JMESPath search is the mapping function). :type expression: str :param expression: JMESPath expression to apply to each page. :return: Returns an iterator that yields the individual elements of applying a JMESPath expression to each page of results. N)jmespathcompilermrr)r expressioncompiledpageresultsrrrrmKs      zPageIterator.searchcCs |jf|S)N)rC)rrtrrrridszPageIterator._make_requestcCs|S)Nr)rrxrrrrjgsz%PageIterator._extract_parsed_responsecCs6i}x&|jD]}||}t||j|q W||_dS)N)rNrmrr~rO)rrxrWr~resultrrrrljs   z-PageIterator._record_non_aggregate_key_valuescCs<|jdk r"|d}||||jdk r8|j||j<dS)Nr)rJrgrsrKrI)rr[rurrrrhss     z$PageIterator._inject_starting_paramscCsBx<|D]0\}}|dk r,|dkr,|||<q ||kr ||=q WdS)NNone)r*)rr[runamerrrrrss  z&PageIterator._inject_token_into_kwargsc Cs|d}||}t|ttfr2||d}nd}t||j|xh|jD]^}||krZqL||}t|trtg}n(t|trd}nt|tt frd}nd}t||j|qLW|S)Nr) rgrmrrstrrr~rVintfloat) rryrvrwZall_datarrsampleZ empty_valuerrrrks&      z"PageIterator._handle_first_requestc CsR||}|dkrg}t||}|d|}t||j||||d<||_dS)Nr])rmrnrr~r\) rryrvrzrwruoriginalZamount_to_keep truncatedrrrros     zPageIterator._truncate_responsecCs^|jdk r|j|siSi}x:t|j|jD](\}}||}|rN|||<q.d||<q.W|S)N)rFrmziprErD)rryZ next_tokensrTZ input_keyrurrrrps     zPageIterator._get_next_tokencCs&t|t|j}ddt||jDS)NcSsg|]\}}t||qSr)ResultKeyIterator)rbr% result_keyrrr sz1PageIterator.result_key_iters..)rrnrVr)rZ teed_resultsrrrresult_key_itersszPageIterator.result_key_iterscCsi}x|D]}|}t|tr0t|dkr0|d}x||jD]r}||}|dkrPq8||}|dkrrt||j|q8t|tr||q8t|t t t fr8t||j||q8Wq Wt ||j |jdk r|j|d<|S)NrZ NextToken)rtuplernrVrmrr~rr"rrrrrar\)rZcomplete_resultrxrZresult_expressionZ result_valueZexisting_valuerrrbuild_full_results6         zPageIterator.build_full_resultc Csn|jdkrdS|j}y,|j|}d}d|kr>|d}|d=Wn$ttfk rd|\}}YnX||fS)Nrr])rJrQrgetr@r _parse_starting_token_deprecated)rruindexrrrrgs    z"PageIterator._parse_starting_tokencCstd|j|jdkrdS|jd}g}d}t|t|jdkrvyt|}Wntk rt|jg}YnXx*|D]"}|dkr| dq|| |q|W| ||fS)z| This handles parsing of old style starting tokens, and attempts to coerce them into the new style. zCAttempting to fall back to old starting token parser. For token: %sNZ___rrr) logdebugrJsplitrnrDrr3r@r!"_convert_deprecated_starting_token)rpartsrurpartrrrrs$     z-PageIterator._parse_starting_token_deprecatedcCslt|}t|j}||kr*td|jn2||kr\tdxt||D]}|dqJWtt |j|S)zb This attempts to convert a deprecated starting token into the new style. zBad starting token: %szaOld format starting token does not contain all input tokens. Setting the rest, in order, as None.N) rnrDr@rJrrranger!rr)rZdeprecated_tokenZlen_deprecated_tokenZlen_input_tokenr%rrrr<s z/PageIterator._convert_deprecated_starting_tokenN)r,r-r.r/r>propertyrVr\setterrar{rmrirjrlrhrsrkrorprrrgrrrrrrrBs*   O   1rBc@sdeZdZeZddZeddZddZddZ d d Z d d Z d dZ ddZ ddZddZdS) PaginatorcCsj||_||_||_||j|_||j|_||j|_| |j|_ | |j|_ | |j|_dS)N)_modelrC_pagination_cfg_get_output_tokensrE_get_input_tokensrD_get_more_results_tokenrF_get_non_aggregate_keys_non_aggregate_keys_get_result_keysrG_get_limit_keyrI)rrRpagination_configmodelrrrr>Rs zPaginator.__init__cCs|jS)N)rG)rrrrrV_szPaginator.result_keyscCs.g}x$|dgD]}|t|qW|S)NrW)rr!r|r})rconfigr_r+rrrrcsz!Paginator._get_non_aggregate_keyscCs>g}|d}t|ts|g}x|D]}|t|q"W|S)NrT)rrr!r|r})rroutputrTrrrris  zPaginator._get_output_tokenscCs|jd}t|ts|g}|S)NrS)rrr)rrrSrrrrrs  zPaginator._get_input_tokenscCs |d}|dk rt|SdS)NrU)rr|r})rrrUrrrrxs z!Paginator._get_more_results_tokencCs8|d}|dk r4t|ts"|g}dd|D}|SdS)NrcSsg|]}t|qSr)r|r})rbZrkrrrrsz.Paginator._get_result_keys..)rrr)rrrrrrr}s   zPaginator._get_result_keyscCs |dS)NrX)r)rrrrrrszPaginator._get_limit_keyc KsB||}||j|j|j|j|j|j|j|d|d|d| S)zCreate paginator object for an operation. This returns an iterable object. Iterating over this object will yield a single page of a response at a time. MaxItems StartingTokenPageSize) _extract_paging_paramsPAGE_ITERATOR_CLSrCrDrErFrGrrI)rkwargsZ page_paramsrrrpaginates zPaginator.paginatecCs|di}|dd}|dk r(t|}|dd}|dk r|jdkrPtdd|jjj}||j}|jdkrt |t st |}nt|}||dd|dS) NZPaginationConfigrrzTPageSize parameter is not supported for the pagination interface for this operation.)rfstringr)rrr) r3rrrIrrZ input_shapemembersZ type_namerr)rrrrYrZZ input_membersZlimit_key_shaperrrrs$          z Paginator._extract_paging_paramsN)r,r-r.rBrr>rrVrrrrrrrrrrrrrOs  rc@s eZdZdZddZddZdS)raIterates over the results of paginated responses. Each iterator is associated with a single result key. Iterating over this object will give you each element in the result key list. :param pages_iterator: An iterator that will give you pages of results (a ``PageIterator`` class). :param result_key: The JMESPath expression representing the result key. cCs||_||_dS)N)_pages_iteratorr)rZpages_iteratorrrrrr>szResultKeyIterator.__init__ccs6x0|jD]&}|j|}|dkr$g}|EdHqWdS)N)rrrm)rrrrrrr{s   zResultKeyIterator.__iter__N)r,r-r.r/r>r{rrrrrs r)r rlogging itertoolsrr|Zbotocore.exceptionsrZbotocore.utilsrr getLoggerr,rrr0r<rBrrrrrrs    NDj