ó <¿CVc@s¥ddlmZmZddlmZddlmZddlmZddl m Z de fd„ƒYZ de fd „ƒYZ d „Z ed kr¡e ƒnd S( iÿÿÿÿ(tprint_functiontunicode_literals(t Nonterminal(tTree(t unicode_repr(tParserItShiftReduceParsercBs}eZdZdd„Zd„Zd„Zd„Zd„Zdd„Z dd „Z d d „Z d „Z d „Z d„ZRS(u A simple bottom-up CFG parser that uses two operations, "shift" and "reduce", to find a single parse for a text. ``ShiftReduceParser`` maintains a stack, which records the structure of a portion of the text. This stack is a list of strings and Trees that collectively cover a portion of the text. For example, while parsing the sentence "the dog saw the man" with a typical grammar, ``ShiftReduceParser`` will produce the following stack, which covers "the dog saw":: [(NP: (Det: 'the') (N: 'dog')), (V: 'saw')] ``ShiftReduceParser`` attempts to extend the stack to cover the entire text, and to combine the stack elements into a single tree, producing a complete parse for the sentence. Initially, the stack is empty. It is extended to cover the text, from left to right, by repeatedly applying two operations: - "shift" moves a token from the beginning of the text to the end of the stack. - "reduce" uses a CFG production to combine the rightmost stack elements into a single Tree. Often, more than one operation can be performed on a given stack. In this case, ``ShiftReduceParser`` uses the following heuristics to decide which operation to perform: - Only shift if no reductions are available. - If multiple reductions are available, then apply the reduction whose CFG production is listed earliest in the grammar. Note that these heuristics are not guaranteed to choose an operation that leads to a parse of the text. Also, if multiple parses exists, ``ShiftReduceParser`` will return at most one of them. :see: ``nltk.grammar`` icCs ||_||_|jƒdS(u² Create a new ``ShiftReduceParser``, that uses ``grammar`` to parse texts. :type grammar: Grammar :param grammar: The grammar used to parse texts. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; and higher numbers will produce more verbose tracing output. N(t_grammart_tracet_check_grammar(tselftgrammarttrace((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyt__init__<s  cCs|jS(N(R(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR Msccsçt|ƒ}|jj|ƒg}|}|jr[tddj|ƒƒ|j||ƒnx?t|ƒdkrœ|j||ƒx|j ||ƒr˜qƒWq^Wt|ƒdkrã|dj ƒ|jj ƒj ƒkrã|dVqãndS(Nu Parsing %ru ii( tlistRtcheck_coverageRtprinttjoint _trace_stacktlent_shiftt_reducetlabeltstarttsymbol(R ttokenststacktremaining_text((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pytparsePs  %cCsB|j|dƒ|j|dƒ|jr>|j||ƒndS(u¶ Move a token from the beginning of ``remaining_text`` to the end of ``stack``. :type stack: list(str and Tree) :param stack: A list of strings and Trees, encoding the structure of the text that has been parsed so far. :type remaining_text: list(str) :param remaining_text: The portion of the text that is not yet covered by ``stack``. :rtype: None iN(tappendtremoveRt _trace_shift(R RR((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyRis  cCs½t|ƒt|ƒkrtSxštt|ƒƒD]†}t||tƒr†t||tƒs_tS||jƒ||jƒkrµtSq/t||tƒrtS||||kr/tSq/WtS(u7 :rtype: bool :return: true if the right hand side of a CFG production matches the rightmost elements of the stack. ``rhs`` matches ``rightmost_stack`` if they are the same length, and each element of ``rhs`` matches the corresponding element of ``rightmost_stack``. A nonterminal element of ``rhs`` matches any Tree whose node value is equal to the nonterminal's symbol. A terminal element of ``rhs`` matches any string whose type is equal to the terminal. :type rhs: list(terminal and Nonterminal) :param rhs: The right hand side of a CFG production. :type rightmost_stack: list(string and Tree) :param rightmost_stack: The rightmost elements of the parser's stack. ( RtFalsetranget isinstanceRRRRtTrue(R trhstrightmost_stackti((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyt _match_rhszs cCs¼|dkr|jjƒ}n |g}xŽ|D]†}t|jƒƒ}|j|jƒ|| ƒr.t|jƒjƒ|| ƒ}|g|| )|j r°|j |||ƒn|Sq.WdS(u§ Find a CFG production whose right hand side matches the rightmost stack elements; and combine those stack elements into a single Tree, with the node specified by the production's left-hand side. If more than one CFG production matches the stack, then use the production that is listed earliest in the grammar. The new Tree replaces the elements in the stack. :rtype: Production or None :return: If a reduction is performed, then return the CFG production that the reduction is based on; otherwise, return false. :type stack: list(string and Tree) :param stack: A list of strings and Trees, encoding the structure of the text that has been parsed so far. :type remaining_text: list(str) :param remaining_text: The portion of the text that is not yet covered by ``stack``. N( tNoneRt productionsRR$R'RtlhsRRt _trace_reduce(R RRt productionR)trhslenttree((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR–s     icCs ||_dS(uP Set the level of tracing output that should be generated when parsing a text. :type trace: int :param trace: The trace level. A trace level of ``0`` will generate no tracing output; and higher trace levels will produce more verbose tracing output. :rtype: None N(R(R R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR Ãsu cCsŽd|d}xT|D]L}t|tƒrM|tt|jƒƒƒd7}q|t|ƒd7}qW|ddj|ƒd7}t|ƒdS(u' Print trace output displaying the given stack and text. :rtype: None :param marker: A character that is printed to the left of the stack. This is used with trace level 2 to print 'S' before shifted stacks and 'R' before reduced stacks. u u [ u u* u]N(R"RRRRRR(R RRtmarkertstelt((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyRÓs  #cCso|jdkr$td|dƒn|jdkrI|j||dƒn"|jdkrk|j||ƒndS(ud Print trace output displaying that a token has been shifted. :rtype: None iu Shift %r:iÿÿÿÿuSiN(RRR(R RR((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyRås cCsŒ|jdkrAdj|jƒƒ}td|jƒ|fƒn|jdkrf|j||dƒn"|jdkrˆ|j||ƒndS(u Print trace output displaying that ``production`` was used to reduce ``stack``. :rtype: None iu uReduce %r <- %suRiN(RRR$RR*R(R RR,RR$((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR+ïscCsŸ|jjƒ}x‰tt|ƒƒD]u}xlt|dt|ƒƒD]Q}||jƒ}||jƒ}|t|ƒ |krBtd||ƒqBqBWq"WdS(u¿ Check to make sure that all of the CFG productions are potentially useful. If any productions can never be used, then print a warning. :rtype: None iuWarning: %r will never be usedN(RR)R!RR$R(R R)R&tjtrhs1trhs2((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR üs N(t__name__t __module__t__doc__R R RRR'R(RR RRR+R (((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyRs(      -   tSteppingShiftReduceParsercBs€eZdZdd„Zd„Zd„Zd„Zd„Zd„Zd„Z dd „Z d „Z d „Z d „Zd „ZRS(u' A ``ShiftReduceParser`` that allows you to setp through the parsing process, performing a single operation at a time. It also allows you to change the parser's grammar midway through parsing a text. The ``initialize`` method is used to start parsing a text. ``shift`` performs a single shift operation, and ``reduce`` performs a single reduce operation. ``step`` will perform a single reduce operation if possible; otherwise, it will perform a single shift operation. ``parses`` returns the set of parses that have been found by the parser. :ivar _history: A list of ``(stack, remaining_text)`` pairs, containing all of the previous states of the parser. This history is used to implement the ``undo`` operation. :see: ``nltk.grammar`` icCs1||_||_d|_d|_g|_dS(N(RRR(t_stackt_remaining_textt_history(R R R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR $s     cCs6t|ƒ}|j|ƒx|jƒr+qW|jƒS(N(Rt initializetsteptparses(R R((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR+s   cCs|jS(uQ :return: The parser's stack. :rtype: list(str and Tree) (R9(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR2scCs|jS(u~ :return: The portion of the text that is not yet covered by the stack. :rtype: list(str) (R:(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR9scCsg|_||_g|_dS(uˆ Start parsing a given text. This sets the parser's stack to ``[]`` and sets its remaining text to ``tokens``. N(R9R:R;(R R((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR<As  cCs|jƒp|jƒS(uó Perform a single parsing operation. If a reduction is possible, then perform that reduction, and return the production that it is based on. Otherwise, if a shift is possible, then perform it, and return True. Otherwise, return False. :return: False if no operation was performed; True if a shift was performed; and the CFG production used to reduce if a reduction was performed. :rtype: Production or bool (treducetshift(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR=Js cCsQt|jƒdkrtS|jj|j|jfƒ|j|j|jƒtS(u Move a token from the beginning of the remaining text to the end of the stack. If there are no more tokens in the remaining text, then do nothing. :return: True if the shift operation was successful. :rtype: bool i(RR:R R;RR9RR#(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR@Ys cCsS|jj|j|jfƒ|j|j|j|ƒ}|sO|jjƒn|S(uƒ Use ``production`` to combine the rightmost stack elements into a single Tree. If ``production`` does not match the rightmost stack elements, then do nothing. :return: The production used to reduce the stack, if a reduction was performed. If no reduction was performed, return None. :rtype: Production or None (R;RR9R:Rtpop(R R,t return_val((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR?gs  cCs8t|jƒdkrtS|jjƒ\|_|_tS(u| Return the parser to its state before the most recent shift or reduce operation. Calling ``undo`` repeatedly return the parser to successively earlier states. If no shift or reduce operations have been performed, ``undo`` will make no changes. :return: true if an operation was successfully undone. :rtype: bool i(RR;R RAR9R:R#(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pytundozs cCsfg}xY|jjƒD]H}t|jƒƒ}|j|jƒ|j| ƒr|j|ƒqqW|S(u¢ :return: A list of the productions for which reductions are available for the current parser state. :rtype: list(Production) (RR)RR$R'R9R(R R)R,R-((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pytreducible_productions‰s  ccset|jƒdkrat|jƒdkra|jdjƒ|jjƒjƒkra|jdVndS(u‡ :return: An iterator of the parses that have been found by this parser so far. :rtype: iter(Tree) iiN(RR:R9RRRR(R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR>–s(cCs ||_dS(u~ Change the grammar used to parse texts. :param grammar: The new grammar. :type grammar: CFG N(R(R R ((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyt set_grammar¤sN(R5R6R7R RRRR<R=R@R(R?RCRDR>RE(((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pyR8s         cCsnddlm}m}|jdƒ}djƒ}|j|ddƒ}x!|j|ƒD]}t|ƒqVWdS(u5 A demonstration of the shift-reduce parser. iÿÿÿÿ(RtCFGuÐ S -> NP VP NP -> Det N | Det N PP VP -> V NP | V NP PP PP -> P NP NP -> 'I' N -> 'man' | 'park' | 'telescope' | 'dog' Det -> 'the' | 'a' P -> 'in' | 'with' V -> 'saw' uI saw a man in the parkR iN(tnltkRRFt fromstringtsplitRR(RRFR tsenttparsertp((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pytdemo±s  u__main__N(t __future__RRt nltk.grammarRt nltk.treeRt nltk.compatRtnltk.parse.apiRRR8RMR5(((sh/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/parse/shiftreduce.pytsÿŸ