ó <¿CVc@s«ddlmZmZddlmZddlmZmZddlm Z ddl m Z de fd„ƒYZ de fd „ƒYZ d „Zed kr§eƒnd S( iÿÿÿÿ(tprint_functiontunicode_literals(t Nonterminal(tTreet ImmutableTree(t unicode_repr(tParserItRecursiveDescentParsercBs¤eZdZdd„Zd„Zd„Zd„Zd„Zdd„Z d„Z d d „Z dd „Z d „Z d „Zd„Zd„Zd„Zdd„ZRS(uq A simple top-down CFG parser that parses texts by recursively expanding the fringe of a Tree, and matching it against a text. ``RecursiveDescentParser`` uses a list of tree locations called a "frontier" to remember which subtrees have not yet been expanded and which leaves have not yet been matched against the text. Each tree location consists of a list of child indices specifying the path from the root of the tree to a subtree or a leaf; see the reference documentation for Tree for more information about tree locations. When the parser begins parsing a text, it constructs a tree containing only the start symbol, and a frontier containing the location of the tree's root node. It then extends the tree to cover the text, using the following recursive procedure: - If the frontier is empty, and the text is covered by the tree, then return the tree as a possible parse. - If the frontier is empty, and the text is not covered by the tree, then return no parses. - If the first element of the frontier is a subtree, then use CFG productions to "expand" it. For each applicable production, add the expanded subtree's children to the frontier, and recursively find all parses that can be generated by the new tree and frontier. - If the first element of the frontier is a token, then "match" it against the next token from the text. Remove the token from the frontier, and recursively find all parses that can be generated by the new tree and frontier. :see: ``nltk.grammar`` icCs||_||_dS(u³ Create a new ``RecursiveDescentParser``, that uses ``grammar`` to parse texts. :type grammar: CFG :param grammar: The grammar used to parse texts. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; and higher numbers will produce more verbose tracing output. N(t_grammart_trace(tselftgrammarttrace((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyt__init__6s cCs|jS(N(R(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR FscCs{t|ƒ}|jj|ƒ|jjƒjƒ}t|gƒ}dg}|jrh|j|||ƒn|j|||ƒS(N(( tlistRtcheck_coveragetstarttsymbolRR t _trace_startt_parse(R ttokensRt initial_treetfrontier((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pytparseIs   ccsát|ƒdkrHt|ƒdkrH|jr@|j||ƒn|Vn•t|ƒdkry|jrÝ|j||ƒqÝndt||dtƒr¸xJ|j|||ƒD] }|Vq¦Wn%x"|j|||ƒD] }|VqÎWdS(u2 Recursively expand and match each elements of ``tree`` specified by ``frontier``, to cover ``remaining_text``. Return a list of all parses found. :return: An iterator of all parses that can be generated by matching and expanding the elements of ``tree`` specified by ``frontier``. :rtype: iter(Tree) :type tree: Tree :param tree: A partial structure for the text that is currently being parsed. The elements of ``tree`` that are specified by ``frontier`` have not yet been expanded or matched. :type remaining_text: list(str) :param remaining_text: The portion of the text that is not yet covered by ``tree``. :type frontier: list(tuple(int)) :param frontier: A list of the locations within ``tree`` of all subtrees that have not yet been expanded, and all leaves that have not yet been matched. This list sorted in left-to-right order of location within the tree. iN(tlenR t_trace_succeedt_trace_backtrackt isinstanceRt_expandt_match(R tremaining_textttreeRtresult((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRXs$   ccsÒ||d}t|ƒdkr«||dkr«|jdtƒ}|d||d<|jr{|j||d|dƒnxP|j|d||dƒD] }|Vq™Wn#|jrÎ|j|||d ƒndS(uû :rtype: iter(Tree) :return: an iterator of all parses that can be generated by matching the first element of ``frontier`` against the first token in ``rtext``. In particular, if the first element of ``frontier`` has the same type as the first token in ``rtext``, then substitute the token into ``tree``; and return all parses that can be generated by matching and expanding the remaining elements of ``frontier``. If the first element of ``frontier`` does not have the same type as the first token in ``rtext``, then return empty list. :type tree: Tree :param tree: A partial structure for the text that is currently being parsed. The elements of ``tree`` that are specified by ``frontier`` have not yet been expanded or matched. :type rtext: list(str) :param rtext: The portion of the text that is not yet covered by ``tree``. :type frontier: list of tuple of int :param frontier: A list of the locations within ``tree`` of all subtrees that have not yet been expanded, and all leaves that have not yet been matched. itdeepiN(RtcopytTrueR t _trace_matchRR(R trtextRRt tree_leaftnewtreeR ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR‡s" $  c cs5|dkr|jjƒ}n |g}x|D]ÿ}|jƒjƒ}|||djƒkr.|j|ƒ}|ddkrˆ|}n |jdtƒ}|||d elt[1] ... elt[n]]`` return a tree that has a node ``lhs.symbol``, and ``n`` children. For each nonterminal element ``elt[i]`` in the production, the tree token has a childless subtree with node value ``elt[i].symbol``; and for each terminal element ``elt[j]``, the tree token has a leaf token with type ``elt[j]``. :param production: The CFG production that licenses the tree token that should be returned. :type production: Production (R.RRtappendRRR*(R R0tchildrentelt((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR,æs icCs ||_dS(uP Set the level of tracing output that should be generated when parsing a text. :type trace: int :param trace: The trace level. A trace level of ``0`` will generate no tracing output; and higher trace levels will produce more verbose tracing output. :rtype: None N(R (R R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR ÿs cCsç|dkrtdddƒnt|tƒrÍt|ƒdkrettt|jƒƒƒddƒnx{tt|ƒƒD]N}|dk rµ||dkrµ|j |||dƒqx|j ||ƒqxWntt|ƒddƒdS(uÆ Print trace output displaying the fringe of ``tree``. The fringe of ``tree`` consists of all of its leaves and all of its childless subtrees. :rtype: None u*tendu iiN(( tprintRRRRRR+R-R(t _trace_fringe(R RttreelocR2((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR9 s %cCsz|jdkr&td|ddƒntdddƒt|ƒdkr_|j||dƒn |j|ƒtdƒdS( uÏ Print trace output displaying the parser's current state. :param operation: A character identifying the operation that generated the current state. :rtype: None iu %c [R7u u [iu]N(R R8RR9(R RRt operation((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyt _trace_tree!s cCs\tddj|ƒƒ|jdkr3tdƒn|jdkrX|j||dƒndS(Nu Parsing %ru iuStart:i(R8tjoinR R<(R RRttext((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR/s  cCsI|jdkr td|ƒn|jdkrE|j||dƒndS(Niu Expand: %siuE(R R8R<(R RRR0((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR/4scCsI|jdkr td|ƒn|jdkrE|j||dƒndS(Niu Match: %riuM(R R8R<(R RRttok((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR$8scCse|jdkrtdƒn|jdkr<td|ƒn|jdkra|j||dƒndS(Niu GOOD PARSE:iuFound a parse: %su+(R R8R<(R RR((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR<s  cCs;|jdkr7|r*td|dƒq7tdƒndS(NiuBacktrack: %r match failediu Backtrack(R R8(R RRttoks((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRAsN(t__name__t __module__t__doc__R R RRRR(RR,R R9R<RR/R$RR(((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRs "    / + 4       tSteppingRecursiveDescentParsercBs¶eZdZdd„Zd„Zd„Zd„Zd„Zd„Zd„Z d „Z dd „Z d „Z d „Zd „Zd„Zd„Zd„Zd„Zd„Zd„ZRS(u’ A ``RecursiveDescentParser`` that allows you to step through the parsing process, performing a single operation at a time. The ``initialize`` method is used to start parsing a text. ``expand`` expands the first element on the frontier using a single CFG production, and ``match`` matches the first element on the frontier against the next text token. ``backtrack`` undoes the most recent expand or match operation. ``step`` performs a single expand, match, or backtrack operation. ``parses`` returns the set of parses that have been found by the parser. :ivar _history: A list of ``(rtext, tree, frontier)`` tripples, containing the previous states of the parser. This history is used to implement the ``backtrack`` operation. :ivar _tried_e: A record of all productions that have been tried for a given tree. This record is used by ``expand`` to perform the next untried production. :ivar _tried_m: A record of what tokens have been matched for a given tree. This record is used by ``step`` to decide whether or not to match a token. :see: ``nltk.grammar`` icCsX||_||_d|_d|_dg|_i|_i|_g|_g|_ dS(N(( RR R(t_rtextt_treet _frontiert_tried_et_tried_mt_historyt_parses(R R R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR as        cCs|jƒ}tj|ƒS(N(R"Rtconvert(R Rtc((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyt_freezens cCs<t|ƒ}|j|ƒx|jƒdk r1qW|jƒS(N(Rt initializetstepR(tparses(R R((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRts   cCsŒ||_|jjƒjƒ}t|gƒ|_dg|_i|_i|_g|_ g|_ |j rˆ|j |j|j|jƒndS(uÀ Start parsing a given text. This sets the parser's tree to the start symbol, its frontier to the root node, and its remaining text to ``token['SUBTOKENS']``. N(( RERRRRRFRGRHRIRJRKR R(R RR((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRO{s       cCs|jS(u} :return: The portion of the text that is not yet covered by the tree. :rtype: list(str) (RE(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRscCs|jS(uÐ :return: A list of the tree locations of all subtrees that have not yet been expanded, and all leaves that have not yet been matched. :rtype: list(tuple(int)) (RG(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyR•scCs|jS(u× :return: A partial structure for the text that is currently being parsed. The elements specified by the frontier have not yet been expanded or matched. :rtype: Tree (RF(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRžscCsq|jƒr+|jƒ}|dk r+|Sn|jƒ}|dk rG|S|jƒrm|j|j|jƒtSdS(uw Perform a single parsing operation. If an untried match is possible, then perform the match, and return the matched token. If an untried expansion is possible, then perform the expansion, and return the production that it is based on. If backtracking is possible, then backtrack, and return True. Otherwise, return None. :return: None if no operation was performed; a token if a match was performed; a production if an expansion was performed; and True if a backtrack operation was performed. :rtype: Production or String or bool N( t untried_matchtmatchR(texpandt backtrackRRFRGR#(R ttokenR0((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRP§s      cCsÎt|jƒdkrdSt|j|jdtƒs:dS|dkrU|jƒ}n |g}g}xc|D][}|jj|j |jƒgƒj |ƒx*|j |j |j|j|ƒD]}|SWqkWdS(u— Expand the first element of the frontier. In particular, if the first element of the frontier is a subtree whose node type is equal to ``production``'s left hand side, then add a child to that subtree for each element of ``production``'s right hand side. If ``production`` is not specified, then use the first untried expandable production. If all expandable productions have been tried, do nothing. :return: The production used to expand the frontier, if an expansion was performed. If no expansion was performed, return None. :rtype: Production or None iN( RRGR(RRFRtuntried_expandable_productionsRHt setdefaultRNR4RRE(R R0R)RQtprodt_result((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRTÆs   (( cCs¬|jd}|jj|j|jƒgƒj|ƒt|jƒdkrNdSt |j|jdt ƒrodSx6|j |j|j|jƒD]}|j dddSWdS(up Match the first element of the frontier. In particular, if the first element of the frontier has the same type as the next text token, then substitute the text token into the tree. :return: The token matched, if a match operation was performed. If no match was performed, return None :rtype: str or None iiÿÿÿÿN( RERIRXRNRFR4RRGR(RRRRJ(R R?RZ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRSís (%cCs>t|jƒdkrtS|jjƒ\|_|_|_tS(u| Return the parser to its state before the most recent match or expand operation. Calling ``undo`` repeatedly return the parser to successively earlier states. If no match or expand operations have been performed, ``undo`` will make no changes. :return: true if an operation was successfully undone. :rtype: bool i(RRJtFalsetpopRERFRGR#(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRUs !cCs”t|jƒdkrgS|j|jd}t|jƒdksRt|tƒ rVgSg|jjƒD]*}|jƒjƒ|j ƒkrf|^qfS(u¦ :return: A list of all the productions for which expansions are available for the current parser state. :rtype: list(Production) i( RRGRFRRRR)R*RR+(R tfrontier_childtp((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pytexpandable_productionsscCsJ|jj|j|jƒgƒ}g|jƒD]}||kr.|^q.S(u® :return: A list of all the untried productions for which expansions are available for the current parser state. :rtype: list(Production) (RHtgetRNRFR_(R ttried_expansionsR^((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRW&s!cCsKt|jƒdkrtS|jj|j|jƒgƒ}|jd|kS(u :return: Whether the first element of the frontier is a token that has not yet been matched. :rtype: bool i(RRER[RIR`RNRF(R t tried_matches((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRR1s!cCs(t|jƒdko't|jƒdkS(u{ :return: Whether the parser's current state represents a complete parse. :rtype: bool i(RRGRE(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pytcurrently_complete<scCs‘|jj|j|j|jfƒ||_||_||_t|ƒdkrŠt|ƒdkrŠ|jj|ƒ|j|j|jƒndgS(u¢ A stub version of ``_parse`` that sets the parsers current state to the given arguments. In ``RecursiveDescentParser``, the ``_parse`` method is used to recursively continue parsing a text. ``SteppingRecursiveDescentParser`` overrides it to capture these recursive calls. It records the parser's old state in the history (to allow for backtracking), and updates the parser's new state using the given arguments. Finally, it returns ``[1]``, which is used by ``match`` and ``expand`` to detect whether their operations were successful. :return: ``[1]`` :rtype: list of int ii(RJR4RERFRGRRKR(R RRR((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRDs"   $cCs t|jƒS(u‰ :return: An iterator of the parses that have been found by this parser so far. :rtype: list of Tree (titerRK(R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRQ_scCs ||_dS(u~ Change the grammar used to parse texts. :param grammar: The new grammar. :type grammar: CFG N(R(R R ((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyt set_grammargsN(RARBRCR RNRRORRRRPR(RTRSRUR_RWRRRcRRQRe(((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pyRDIs&      '      cCsddlm}m}|jdƒ}x|jƒD]}t|ƒq2Wdjƒ}|j|ddƒ}x!|j|ƒD]}t|ƒqwWdS(u: A demonstration of the recursive descent parser. iÿÿÿÿ(RtCFGuÐ S -> NP VP NP -> Det N | Det N PP VP -> V NP | V NP PP PP -> P NP NP -> 'I' N -> 'man' | 'park' | 'telescope' | 'dog' Det -> 'the' | 'a' P -> 'in' | 'with' V -> 'saw' uI saw a man in the parkR iN(tnltkRRft fromstringR)R8tsplitR(RRfR RYtsenttparserR^((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pytdemots  u__main__N(t __future__RRt nltk.grammarRt nltk.treeRRt nltk.compatRtnltk.parse.apiRRRDRlRA(((sm/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/recursivedescent.pytsÿ7ÿ,