ó <¿CVc@ sddlmZmZddlmZddlZddlmZddlmZddl m Z m Z m Z e de fd„ƒYƒZe d e fd „ƒYƒZe d efd „ƒYƒZe d efd„ƒYƒZe defd„ƒYƒZe defd„ƒYƒZe defd„ƒYƒZe defd„ƒYƒZe defd„ƒYƒZe defd„ƒYƒZejdd%ƒZd„Ze defd„ƒYƒZe d efd!„ƒYƒZd"„Zd#„Zed$kreƒndS(&iÿÿÿÿ(tprint_functiontunicode_literals(tdivisionN(tTree(t ChunkParserI(tpython_2_unicode_compatiblet string_typest unicode_reprt ChunkStringcB s´eZdZdZdeZdZdZdeZdeZe j deƒZ e j dƒZ e j d ƒZ d d „Zd „Zd „Zdd„Zd„Zd„Zd„ZRS(uL A string-based encoding of a particular chunking of a text. Internally, the ``ChunkString`` class uses a single string to encode the chunking of the input text. This string contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is:: {
}{
}<.>{
}<.> ``ChunkString`` are created from tagged texts (i.e., lists of ``tokens`` whose type is ``TaggedType``). Initially, nothing is chunked. The chunking of a ``ChunkString`` can be modified with the ``xform()`` method, which uses a regular expression to transform the string representation. These transformations should only add and remove braces; they should *not* modify the sequence of angle-bracket delimited tags. :type _str: str :ivar _str: The internal string representation of the text's encoding. This string representation contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is:: {
}{
}<.>{
}<.> :type _pieces: list(tagged tokens and chunks) :ivar _pieces: The tagged tokens and chunks encoded by this ``ChunkString``. :ivar _debug: The debug level. See the constructor docs. :cvar IN_CHUNK_PATTERN: A zero-width regexp pattern string that will only match positions that are in chunks. :cvar IN_CHINK_PATTERN: A zero-width regexp pattern string that will only match positions that are in chinks. u [^\{\}<>]u(<%s+?>)u (?=[^\{]*\})u(?=[^\}]*(\{|$))u (\{%s+?\})+?u(%s+?)+?u^(\{?%s\}?)*?$u[^\{\}]+u(\{\})*$icC se|jƒ|_||_g|jD]}|j|ƒ^q#}ddj|ƒd|_||_dS(u} Construct a new ``ChunkString`` that encodes the chunking of the text ``tagged_tokens``. :type chunk_struct: Tree :param chunk_struct: The chunk structure to be further chunked. :type debug_level: int :param debug_level: The level of debugging which should be applied to transformations on the ``ChunkString``. The valid levels are: - 0: no checks - 1: full check on to_chunkstruct - 2: full check on to_chunkstruct and cursory check after each transformation. - 3: full check on to_chunkstruct and full check after each transformation. We recommend you use at least level 1. You should probably use level 3 if you use any non-standard subclasses of ``RegexpChunkRule``. uN(tlabelt _root_labelt_piecest_tagtjoint_strt_debug(tselft chunk_structt debug_levelttokttags((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt__init__Hs  %cC s@t|tƒr|dSt|tƒr0|jƒStdƒ‚dS(Niu4chunk structures must contain tagged tokens or trees(t isinstancettupleRR t ValueError(RR((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR cs  c C s tjj|ƒs%td|ƒ‚ntjjd|ƒ}x`tdt|ƒdƒD]D}||d|dd!}tjj|ƒsUtd|ƒ‚qUqUW|dkr­dSt j d|ƒdd!}g|j D]}|j |ƒ^qÐ}||krtd ƒ‚ndS( u” Check to make sure that ``s`` still corresponds to some chunked version of ``_pieces``. :type verify_tags: bool :param verify_tags: Whether the individual tags should be checked. If this is false, ``_verify`` will check to make sure that ``_str`` encodes a chunked version of *some* list of tokens. If this is true, then ``_verify`` will check to make sure that the tags in ``_str`` match those in ``_pieces``. :raise ValueError: if the internal string representation of this ``ChunkString`` is invalid or not consistent with _pieces. u2Transformation generated invalid chunkstring: %suiiˆiNu [\{\}<>]+iÿÿÿÿu9Transformation generated invalid chunkstring: tag changed( Rt_VALIDtmatchRt _BRACKETStsubtrangetlent_BALANCED_BRACKETStretsplitR R ( Rtst verify_tagstbracketstitsubstrttags1tpiecettags2((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt_verifyls ! % uCHUNKcC sÄ|jdkr%|j|jdƒng}d}d}xztjd|jƒD]c}|jdƒ}|j|||!}|r•|jt||ƒƒn ||7}||7}| }qMWt|j |ƒS(uÄ Return the chunk structure encoded by this ``ChunkString``. :rtype: Tree :raise ValueError: If a transformation has generated an invalid chunkstring. iiu[{}]u<( RR*RR R!tcountR tappendRR (Rt chunk_labeltpiecestindextpiece_in_chunkR(tlengtht subsequence((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pytto_chunkstruct“s   cC sctj|||jƒ}tjdd|ƒ}|jdkrV|j||jdƒn||_dS(uH Apply the given transformation to the string encoding of this ``ChunkString``. In particular, find all occurrences that match ``regexp``, and replace them using ``repl`` (as done by ``re.sub``). This transformation should only add and remove braces; it should *not* modify the sequence of angle-bracket delimited tags. Furthermore, this transformation may not result in improper bracketing. Note, in particular, that bracketing may not be nested. :type regexp: str or regexp :param regexp: A regular expression matching the substring that should be replaced. This will typically include a named group, which can be used by ``repl``. :type repl: str :param repl: An expression specifying what should replace the matched substring. Typically, this will include a named replacement group, specified by ``regexp``. :rtype: None :raise ValueError: If this transformation generated an invalid chunkstring. u\{\}uiiN(R RRRR*(RtregexptreplR"((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pytxform³s cC sdt|jƒS(u¸ Return a string representation of this ``ChunkString``. It has the form:: }{
}'> :rtype: str u(RR(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt__repr__Ús cC sNtjdd|jƒ}tjdd|ƒ}|ddkrJd|}n|S(u( Return a formatted representation of this ``ChunkString``. This representation will include extra spaces to ensure that tags will line up with the representation of other ``ChunkStrings`` for the same text, regardless of the chunking. :rtype: str u>(?!\})u> u([^\{])}'->''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str uu>(RRGtpatternRE(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7Bs cC sªtjd|ƒ}|jdƒjƒ}|jdƒp9ddjƒ}y3|s^tdƒ‚n|ddkr’|d d kr’t|dd !|ƒS|dd krÆ|d dkrÆt|dd !|ƒSd |kr÷|jd ƒ\}}t|||ƒSd |kr(|jd ƒ\}}t |||ƒStjd |ƒrhtjd|ƒ\}}}t ||||ƒStd|ƒ‚Wn*ttj fk r¥td|ƒ‚nXdS(u^ Create a RegexpChunkRule from a string description. Currently, the following formats are supported:: {regexp} # chunk rule }regexp{ # chink rule regexp}{regexp # split rule regexp{}regexp # merge rule Where ``regexp`` is a regular expression for the rule. Any text following the comment marker (``#``) will be used as the rule's description: >>> from nltk.chunk.regexp import RegexpChunkRule >>> RegexpChunkRule.fromstring('{
?+}') ?+'> u&(?P(\\.|[^#])*)(?P#.*)?uruleucommentuiuEmpty chunk patterniu{iÿÿÿÿu}u}{u{}u[^{}]*{[^{}]*}[^{}]*u[{}]uIllegal chunk pattern: %sN( R RtgrouptstripRt ChunkRulet ChinkRuleR!t SplitRulet MergeRuletChunkRuleWithContextterror(R"tmtruletcommenttlefttrighttchunk((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt fromstringQs,    ( R:R;R<RRJRHR7t staticmethodRZ(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRDøs    RNcB s eZdZd„Zd„ZRS(u) A rule specifying how to add chunks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any substring that matches this tag pattern and that is not already part of a chunk, and create a new chunk containing that substring. cC sE||_tjdt|ƒtjfƒ}tj||d|ƒdS(u± Construct a new ``ChunkRule``. :type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will chunk any substring that matches this tag pattern and that is not already part of a chunk. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)%su {\g}N(t_patternR RCttag_pattern2re_patternRR@RDR(Rt tag_patternRHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR‰s    cC sdt|jƒdS(u0 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u (RR\(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7s (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRN€s ROcB s eZdZd„Zd„ZRS(u0 A rule specifying how to remove chinks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any substring that matches this tag pattern and that is contained in a chunk, and remove it from that chunk, thus creating two new chunks. cC sE||_tjdt|ƒtjfƒ}tj||d|ƒdS(uò Construct a new ``ChinkRule``. :type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will find any substring that matches this tag pattern and that is contained in a chunk, and remove it from that chunk, thus creating two new chunks. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)%su }\g{N(R\R RCR]RR?RDR(RR^RHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR´s    cC sdt|jƒdS(u0 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u (RR\(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7Ès (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRO«s t UnChunkRulecB s eZdZd„Zd„ZRS(uä A rule specifying how to remove chunks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any complete chunk that matches this tag pattern, and un-chunk it. cC s<||_tjdt|ƒƒ}tj||d|ƒdS(u  Construct a new ``UnChunkRule``. :type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will find any complete chunk that matches this tag pattern, and un-chunk it. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u\{(?P%s)\}u \gN(R\R RCR]RDR(RR^RHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRßs  cC sdt|jƒdS(u2 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u(RR\(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7ñs (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR_×s RQcB s eZdZd„Zd„ZRS(u~ A rule specifying how to merge chunks in a ``ChunkString``, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose end matches left pattern, and immediately followed by a chunk whose beginning matches right pattern. It will then merge those two chunks into a single chunk. cC swtjt|ƒƒtjt|ƒƒ||_||_tjdt|ƒt|ƒfƒ}tj||d|ƒdS(uÊ Construct a new ``MergeRule``. :type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches ``left_tag_pattern``, and immediately followed by a chunk whose beginning matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches this pattern, and immediately followed by a chunk whose beginning matches ``right_tag_pattern``. It will then merge those two chunks into a single chunk. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)}{(?=%s)u\gN(R RCR]t_left_tag_patternt_right_tag_patternRDR(Rtleft_tag_patterntright_tag_patternRHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR s    cC s&dt|jƒdt|jƒdS(u< Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u (RR`Ra(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7-s (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRQs  #RPcB s eZdZd„Zd„ZRS(uw A rule specifying how to split chunks in a ``ChunkString``, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk that matches the left pattern followed by the right pattern. It will then split the chunk into two new chunks, at the point between the two pattern matches. cC swtjt|ƒƒtjt|ƒƒ||_||_tjdt|ƒt|ƒfƒ}tj||d|ƒdS(uß Construct a new ``SplitRule``. :type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk containing a substring that matches ``left_tag_pattern`` followed by this pattern. It will then split the chunk into two new chunks at the point between these two matching patterns. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk containing a substring that matches this pattern followed by ``right_tag_pattern``. It will then split the chunk into two new chunks at the point between these two matching patterns. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)(?=%s)u \g}{N(R RCR]R`RaRDR(RRbRcRHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRGs    cC s&dt|jƒdt|jƒdS(u2 Return a string representation of this rule. It has the form:: ', '
'> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u (RR`Ra(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7is (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRP=s  "tExpandLeftRulecB s eZdZd„Zd„ZRS(u A rule specifying how to expand chunks in a ``ChunkString`` to the left, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose beginning matches right pattern, and immediately preceded by a chink whose end matches left pattern. It will then expand the chunk to incorporate the new material on the left. cC swtjt|ƒƒtjt|ƒƒ||_||_tjdt|ƒt|ƒfƒ}tj||d|ƒdS(uâ Construct a new ``ExpandRightRule``. :type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose beginning matches ``right_tag_pattern``, and immediately preceded by a chink whose end matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose beginning matches this pattern, and immediately preceded by a chink whose end matches ``left_tag_pattern``. It will then expand the chunk to incorporate the new material on the left. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)\{(?P%s)u{\g\gN(R RCR]R`RaRDR(RRbRcRHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRƒs    cC s&dt|jƒdt|jƒdS(uA Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u(RR`Ra(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7¦s (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRdys  #tExpandRightRulecB s eZdZd„Zd„ZRS(u  A rule specifying how to expand chunks in a ``ChunkString`` to the right, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose end matches left pattern, and immediately followed by a chink whose beginning matches right pattern. It will then expand the chunk to incorporate the new material on the right. cC swtjt|ƒƒtjt|ƒƒ||_||_tjdt|ƒt|ƒfƒ}tj||d|ƒdS(uã Construct a new ``ExpandRightRule``. :type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches ``left_tag_pattern``, and immediately followed by a chink whose beginning matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches this pattern, and immediately followed by a chink whose beginning matches ``right_tag_pattern``. It will then expand the chunk to incorporate the new material on the right. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u(?P%s)\}(?P%s)u\g\g}N(R RCR]R`RaRDR(RRbRcRHR4((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRÀs    cC s&dt|jƒdt|jƒdS(uB Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u(RR`Ra(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7ãs (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRe¶s  #RRcB s eZdZd„Zd„ZRS(u© A rule specifying how to add chunks to a ``ChunkString``, using three matching tag patterns: one for the left context, one for the chunk, and one for the right context. When applied to a ``ChunkString``, it will find any substring that matches the chunk tag pattern, is surrounded by substrings that match the two context patterns, and is not already part of a chunk; and create a new chunk containing the substring that matched the chunk tag pattern. Caveat: Both the left and right context are consumed when this rule matches; therefore, if you need to find overlapping matches, you will need to apply your rule more than once. cC s¨tjt|ƒƒtjt|ƒƒtjt|ƒƒ||_||_||_tjdt|ƒt|ƒt|ƒtjfƒ}d}tj ||||ƒdS(uQ Construct a new ``ChunkRuleWithContext``. :type left_context_tag_pattern: str :param left_context_tag_pattern: A tag pattern that must match the left context of ``chunk_tag_pattern`` for this rule to apply. :type chunk_tag_pattern: str :param chunk_tag_pattern: A tag pattern that must match for this rule to apply. If the rule does apply, then this pattern also identifies the substring that will be made into a chunk. :type right_context_tag_pattern: str :param right_context_tag_pattern: A tag pattern that must match the right context of ``chunk_tag_pattern`` for this rule to apply. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. u((?P%s)(?P%s)(?P%s)%su\g{\g}\gN( R RCR]t_left_context_tag_patternt_chunk_tag_patternt_right_context_tag_patternRR@RDR(Rtleft_context_tag_patterntchunk_tag_patterntright_context_tag_patternRHR4t replacement((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRs       cC sd|j|j|jfS(uF Return a string representation of this rule. It has the form:: ', '', '
'> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str u#(RfRgRh(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7)s  (R:R;R<RR7(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRRós &u^((%s|<%s>)*)$u [^\{\}<>]+cC sªtjdd|ƒ}tjdd|ƒ}tjdd|ƒ}tj|ƒsatd|ƒ‚nd„}|tjƒ}||ƒ}tjd ||ƒ}||ƒ}|S( u€ Convert a tag pattern to a regular expression pattern. A "tag pattern" is a modified version of a regular expression, designed for matching sequences of tags. The differences between regular expression patterns and tag patterns are: - In tag patterns, ``'<'`` and ``'>'`` act as parentheses; so ``'+'`` matches one or more repetitions of ``''``, not ``''``. - Whitespace in tag patterns is ignored. So ``'
| '`` is equivalant to ``'
|'`` - In tag patterns, ``'.'`` is equivalant to ``'[^{}<>]'``; so ``''`` matches any single tag starting with ``'NN'``. In particular, ``tag_pattern2re_pattern`` performs the following transformations on the given pattern: - Replace '.' with '[^<>{}]' - Remove any whitespace - Add extra parens around '<' and '>', to make '<' and '>' act like parentheses. E.g., so that in '+', the '+' has scope over the entire ''; and so that in '', the '|' has scope over 'NN' and 'IN', but not '<' or '>'. - Check to make sure the resulting pattern is valid. :type tag_pattern: str :param tag_pattern: The tag pattern to convert to a regular expression pattern. :raise ValueError: If ``tag_pattern`` is not a valid tag pattern. In particular, ``tag_pattern`` should not include braces; and it should not contain nested or mismatched angle-brackets. :rtype: str :return: A regular expression pattern corresponding to ``tag_pattern``. u\suuu)>)uBad tag pattern: %rcS s#t|ƒ}|jƒdj|ƒS(Nu(tlisttreverseR (R8tlst((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt reverse_strvs  u\.(?!\\(\\\\)*($|[^\\]))(R RtCHUNK_TAG_PATTERNRRRR=(R^Rpttc_revtreversed((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR]Cs%   tRegexpChunkParsercB sYeZdZdddd„Zd„Zd„Zd d„Zd„Zd „Z d „Z RS( un A regular expression based chunk parser. ``RegexpChunkParser`` uses a sequence of "rules" to find chunks of a single type within a text. The chunking of the text is encoded using a ``ChunkString``, and each rule acts by modifying the chunking in the ``ChunkString``. The rules are all implemented using regular expression matching and substitution. The ``RegexpChunkRule`` class and its subclasses (``ChunkRule``, ``ChinkRule``, ``UnChunkRule``, ``MergeRule``, and ``SplitRule``) define the rules that are used by ``RegexpChunkParser``. Each rule defines an ``apply()`` method, which modifies the chunking encoded by a given ``ChunkString``. :type _rules: list(RegexpChunkRule) :ivar _rules: The list of rules that should be applied to a text. :type _trace: int :ivar _trace: The default level of tracing. uNPuSicC s(||_||_||_||_dS(u¦ Construct a new ``RegexpChunkParser``. :type rules: list(RegexpChunkRule) :param rules: The sequence of rules that should be used to generate the chunking for a tagged text. :type chunk_label: str :param chunk_label: The node value that should be used for chunk subtrees. This is typically a short string describing the type of information contained by the chunk, such as ``"NP"`` for base noun phrases. :type root_label: str :param root_label: The node value that should be used for the top node of the chunk structure. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. N(t_rulest_tracet _chunk_labelR (RtrulesR-t root_labelttrace((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRœs   cC sˆtdƒt|ƒxm|jD]b}|j|ƒ|r_td|jƒdt|ƒdƒntd|jƒdƒt|ƒqWdS(u¶ Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn. Generate trace output between each rule. If ``verbose`` is true, then generate verbose output. :type chunkstr: ChunkString :param chunkstr: The chunk string to which each rule should be applied. :type verbose: bool :param verbose: Whether output should be verbose. :rtype: None u# Input:u#u (u):u:N(tprintRuRJRHR(RRItverboseRU((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt _trace_apply¶s   (cC s%x|jD]}|j|ƒq WdS(u÷ Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn. :param chunkstr: The chunk string to which each rule should be applied. :type chunkstr: ChunkString :rtype: None N(RuRJ(RRIRU((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt_notrace_applyÍs cC sÆt|ƒdkr,tdƒt|jgƒSy|jƒWn#tk r_t|j|ƒ}nX|dkrx|j}nt|ƒ}|r©|dk}|j ||ƒn |j |ƒ|j |j ƒS(uQ :type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or highter will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :rtype: Tree :return: a chunk structure that encodes the chunks in a given tagged sentence. A chunk is a non-overlapping linguistic group, such as a noun phrase. The set of chunks identified in the chunk structure depends on the rules used to define this ``RegexpChunkParser``. iuWarning: parsing empty textiN( RR{RR R tAttributeErrortNoneRvRR}R~R3Rw(RRRzRIR|((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pytparseÛs      cC s|jS(uu :return: the sequence of rules used by ``RegexpChunkParser``. :rtype: list(RegexpChunkRule) (Ru(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRxscC sdt|jƒS(uy :return: a concise string representation of this ``RegexpChunkParser``. :rtype: str u!(RRu(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7 scC s²dt|jƒ}d}x,|jD]!}t|t|jƒƒƒ}q#W|dkrpdt|d ƒd}nd}x1|jD]&}|||jƒt|ƒf7}q€W|d S( um :return: a verbose string representation of this ``RegexpChunkParser``. :rtype: str u!RegexpChunkParser with %d rules: ii#u %ius%s u %s %s iÿÿÿÿ(RRutmaxRHtreprR(RR"tmarginRUtformat((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR9s $N( R:R;R<RR}R~R€RRxR7R9(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRt†s   *  t RegexpParsercB sPeZdZdddd„Zd„Zd„Zd d„Zd„Zd „Z RS( u÷ A grammar based chunk parser. ``chunk.RegexpParser`` uses a set of regular expression patterns to specify the behavior of the parser. The chunking of the text is encoded using a ``ChunkString``, and each rule acts by modifying the chunking in the ``ChunkString``. The rules are all implemented using regular expression matching and substitution. A grammar contains one or more clauses in the following form:: NP: {} # chunk determiners and adjectives }<[\.VI].*>+{ # chink any tag beginning with V, I, or . <.*>}{
# split a chunk at a determiner {} # merge chunk ending with det/adj # with one starting with a noun The patterns of a clause are executed in order. An earlier pattern may introduce a chunk boundary that prevents a later pattern from executing. Sometimes an individual pattern will match on multiple, overlapping extents of the input. As with regular expression substitution more generally, the chunker will identify the first match possible, then continue looking for matches after this one has ended. The clauses of a grammar are also executed in order. A cascaded chunk parser is one having more than one clause. The maximum depth of a parse tree created by this chunk parser is the same as the number of clauses in the grammar. When tracing is turned on, the comment portion of a line is displayed each time the corresponding pattern is applied. :type _start: str :ivar _start: The start symbol of the grammar (the root node of resulting trees) :type _stages: int :ivar _stages: The list of parsing stages corresponding to the grammar uSiicC s±||_g|_||_||_t|tƒrI|j|||ƒndd}yt|ƒ}Wnt|ƒ‚nXx,|D]$}t|t ƒs|t|ƒ‚q|q|W||_dS(u× Create a new chunk parser, from the given start state and set of chunk patterns. :param grammar: The grammar, or a list of RegexpChunkParser objects :type grammar: str or list(RegexpChunkParser) :param root_label: The top node of the tree being created :type root_label: str or Nonterminal :param loop: The number of times to run through the patterns :type loop: int :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. u>Expected string or list of RegexpChunkParsers for the grammar.N( Rvt_stagest_grammart_loopRRt _read_grammarRmt TypeErrorRt(RtgrammarRytloopRzttype_errtelt((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRSs     cC säg}d}x»|jdƒD]ª}|jƒ}tjd|ƒ}|r|j||||ƒ|jdƒjƒ}g}|jdƒjƒ}n|dks|jdƒr°qn|jt j |ƒƒqW|j||||ƒdS(u[ Helper function for __init__: read the grammar if it is a string. u u*(?P(\.|[^:])*)(:(?P.*))u nonterminaluruleuu#N( R€R!RMR Rt _add_stageRLt startswithR,RDRZ(RRŒRyRzRxtlhstlineRT((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRŠvs cC sV|gkrR|s!tdƒ‚nt|d|d|d|ƒ}|jj|ƒndS(uN Helper function for __init__: add a new stage to the parser. uExpected stage marker (eg NP:)R-RyRzN(RRtR‡R,(RRxR’RyRztparser((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR“s  cC s_|dkr|j}nx@t|jƒD]/}x&|jD]}|j|d|ƒ}q8Wq(W|S(uœ Apply the chunk parser to this input. :type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked (this tree is modified, and is also returned) :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or highter will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :return: the chunked output. :rtype: Tree RzN(R€RvRR‰R‡R(RRRzR%R”((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyRžs  cC sdt|jƒS(un :return: a concise string representation of this ``chunk.RegexpParser``. :rtype: str u#(RR‡(R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR7µscC sCdt|jƒ}d}x|jD]}|d|7}q#W|d S(ut :return: a verbose string representation of this ``RegexpParser``. :rtype: str u#chunk.RegexpParser with %d stages: iu%s iÿÿÿÿ(RR‡(RR"R„R”((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR9¼s N( R:R;R<RRŠRR€RR7R9(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyR†)s)#   c C s`ddlm}ddlm}|jƒ}x‰|jdƒD]x}t|ƒ|jƒ}|sdq<n|j|ƒ}|j ƒ}|j |d|ƒddƒ}|j ||ƒtƒq<Wtdd d d ƒtd |ƒtd dƒtd|j ƒddddƒtd|j ƒddddƒtd|jƒdƒ|jƒrÁtdƒ|jƒ} x1| d D]%}tddjtt|ƒƒƒqpWt|jƒƒdkrÁtdƒqÁn|jƒr?tdƒ|jƒ} x1| d D]%}tddjtt|ƒƒƒqîWt|jƒƒdkr?tdƒq?ntd d d dƒtƒdS(uè Demonstration code for evaluating a chunk parser, using a ``ChunkScore``. This function assumes that ``text`` contains one sentence per line, and that each sentence has the form expected by ``tree.chunk``. It runs the given chunk parser on each sentence in the text, and scores the result. It prints the final score (precision, recall, and f-measure); and reports the set of chunks that were missed and the set of chunks that were incorrect. (At most 10 missing chunks and 10 incorrect chunks are reported). :param chunkparser: The chunkparser to be tested :type chunkparser: ChunkParserI :param text: The chunked tagged text that should be used for evaluation. :type text: str iÿÿÿÿ(RY(Ru uSRziu/u=iKu\uScoringu-iMuPrecision: %5.1f%%idu itenduRecall: %5.1f%%iuF-Measure: %5.1f%%uMissed:i u u ...u Incorrect:Nu u (tnltkRYt nltk.treeRt ChunkScoreR!R{RMt tagstr2treetleavesRtscoret precisiontrecallt f_measuretmissedR tmapR8Rt incorrect( t chunkparserttextRYRt chunkscoretsentencetgoldttokensttestRŸR¡((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pyt demo_evalÌsF      !!   #   #c C sÎddlm}m}d}tddƒtdƒt|ƒtddƒtƒd}|j|ƒ}t||ƒd}|j|ƒ}t||ƒd }|j|ƒ}t||ƒd }|j|ƒ}t||ƒdd lm}tƒtd ƒ|jd ƒ}t|j||j ddd!ƒƒƒtƒtdƒd}|j|ƒ}t|j||j dƒd ƒƒtƒtdƒd }|j|ƒ}t|j d"d#d$d%d&d'd(d)gƒƒd S(*u« A demonstration for the ``RegexpChunkParser`` class. A single text is parsed with four different chunk parsers, using a variety of rules and strategies. iÿÿÿÿ(RYRuö [ the/DT little/JJ cat/NN ] sat/VBD on/IN [ the/DT mat/NN ] ./. [ John/NNP ] saw/VBD [the/DT cats/NNS] [the/DT dog/NN] chased/VBD ./. [ John/NNP ] thinks/VBZ [ Mary/NN ] saw/VBD [ the/DT cat/NN ] sit/VB on/IN [ the/DT mat/NN ]./. u*iKuEvaluation text:u NP: # NP stage {
?*} # chunk determiners, adjectives and nouns {+} # chunk proper nouns uÀ NP: {<.*>} # start by chunking each tag }<[\.VI].*>+{ # unchunk any verbs, prepositions or periods {} # merge det/adj with nouns uy NP: {
?*} # chunk determiners, adjectives and nouns VP: {?} # VP = verb words u6 NP: {<.*>*} # start by chunking everything }<[\.VI].*>+{ # chink any verbs, prepositions or periods <.*>}{
# separate on determiners PP: {} # PP = preposition + noun phrase VP: {*} # VP = verb words + NPs and PPs (t conll2000uDemonstration of empty grammar:uutest.txtt chunk_typesuNPu6Demonstration of accuracy evaluation using CoNLL tags:iu#Demonstration of tagged token inpututheuDTulittleuJJucatuNNusatuVBDuonuINumatu.N(uNP(utheuDT(ulittleuJJ(ucatuNN(usatuVBD(uonuIN(utheuDT(umatuNN(u.u.( R–RYRR{R†R©t nltk.corpusRªtaccuracyt chunked_sentsR(RYRR£RŒtcpRª((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pytdemo sJ         #  u__main__(u [^\{\}<>]+u [^\{\}<>]+( t __future__RRRR R—Rtnltk.chunk.apiRt nltk.compatRRRtobjectRRDRNROR_RQRPRdReRRRCRqR]RtR†R©R°R:(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/chunk/regexp.pytsJ â‡*+(<;<<K   C¢¢ > ]