ó <¿CVc@sïdZddlmZddlZddlmZddlmZddlm Z e defd„ƒYƒZ d e fd „ƒYZ d e fd „ƒYZ d e fd„ƒYZ eeejejBejBd„Ze ƒjZe ƒjZdS(uû Regular-Expression Tokenizers A ``RegexpTokenizer`` splits a string into substrings using a regular expression. For example, the following tokenizer forms tokens out of alphabetic sequences, money expressions, and any other non-whitespace sequences: >>> from nltk.tokenize import RegexpTokenizer >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> tokenizer = RegexpTokenizer('\w+|\$[\d\.]+|\S+') >>> tokenizer.tokenize(s) ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] A ``RegexpTokenizer`` can use its regexp to match delimiters instead: >>> tokenizer = RegexpTokenizer('\s+', gaps=True) >>> tokenizer.tokenize(s) ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks.'] Note that empty tokens are not returned when the delimiter appears at the start or end of the string. The material between the tokens is discarded. For example, the following tokenizer selects just the capitalized words: >>> capword_tokenizer = RegexpTokenizer('[A-Z]\w+') >>> capword_tokenizer.tokenize(s) ['Good', 'New', 'York', 'Please', 'Thanks'] This module contains several subclasses of ``RegexpTokenizer`` that use pre-defined regular expressions. >>> from nltk.tokenize import BlanklineTokenizer >>> # Uses '\s*\n\s*\n\s*': >>> BlanklineTokenizer().tokenize(s) ['Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.', 'Thanks.'] All of the regular expression tokenizers are also available as functions: >>> from nltk.tokenize import regexp_tokenize, wordpunct_tokenize, blankline_tokenize >>> regexp_tokenize(s, pattern='\w+|\$[\d\.]+|\S+') ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] >>> wordpunct_tokenize(s) ['Good', 'muffins', 'cost', '$', '3', '.', '88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] >>> blankline_tokenize(s) ['Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.', 'Thanks.'] Caution: The function ``regexp_tokenize()`` takes the text as its first argument, and the regular expression pattern as its second argument. This differs from the conventions used by Python's ``re`` functions, where the pattern is always the first argument. (This is for consistency with the other NLTK tokenizers.) iÿÿÿÿ(tunicode_literalsN(t TokenizerI(tregexp_span_tokenize(tpython_2_unicode_compatibletRegexpTokenizercBsUeZdZeeejejBejBd„Z d„Z d„Z d„Z d„Z RS(u× A tokenizer that splits a string using a regular expression, which matches either the tokens or the separators between tokens. >>> tokenizer = RegexpTokenizer('\w+|\$[\d\.]+|\S+') :type pattern: str :param pattern: The pattern used to build this tokenizer. (This pattern may safely contain capturing parentheses.) :type gaps: bool :param gaps: True if this tokenizer's pattern should be used to find separators between tokens; False if this tokenizer's pattern should be used to find the tokens themselves. :type discard_empty: bool :param discard_empty: True if any empty tokens `''` generated by the tokenizer should be discarded. Empty tokens can only be generated if `_gaps == True`. :type flags: int :param flags: The regexp flags used to compile this tokenizer's pattern. By default, the following flags are used: `re.UNICODE | re.MULTILINE | re.DOTALL`. cCsCt|d|ƒ}||_||_||_||_d|_dS(Nupattern(tgetattrt_patternt_gapst_discard_emptyt_flagstNonet_regexp(tselftpatterntgapst discard_emptytflags((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyt__init__fs     cCs+|jdkr'tj|jƒ|_ndS(N(R R tretcompileR(R ((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyt _check_regexpqscCsl|jƒ|jrX|jrEg|jj|ƒD]}|r/|^q/S|jj|ƒSn|jj|ƒSdS(N(RRRR tsplittfindall(R ttextttok((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyttokenizeus    )ccs‹|jƒ|jr\xqt||jƒD]/\}}|joD||ks&||fVq&q&Wn+x(tj|j|ƒD]}|jƒVqrWdS(N(RRRR RRtfinditertspan(R Rtlefttrighttm((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyt span_tokenize‚s  cCs)d|jj|j|j|j|jfS(Nu3%s(pattern=%r, gaps=%r, discard_empty=%r, flags=%r)(t __class__t__name__RRRR (R ((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyt__repr__s(R!t __module__t__doc__tFalsetTrueRtUNICODEt MULTILINEtDOTALLRRRRR"(((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyRLs  tWhitespaceTokenizercBseZdZd„ZRS(uÔ Tokenize a string on whitespace (space, tab, newline). In general, users should use the string ``split()`` method instead. >>> from nltk.tokenize import WhitespaceTokenizer >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> WhitespaceTokenizer().tokenize(s) ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks.'] cCstj|ddtƒdS(Nu\s+R(RRR&(R ((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyRžs(R!R#R$R(((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyR*’s tBlanklineTokenizercBseZdZd„ZRS(u¸ Tokenize a string, treating any sequence of blank lines as a delimiter. Blank lines are defined as lines containing no characters, except for space or tab characters. cCstj|ddtƒdS(Nu \s*\n\s*\n\s*R(RRR&(R ((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyR§s(R!R#R$R(((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyR+¡stWordPunctTokenizercBseZdZd„ZRS(uß Tokenize a text into a sequence of alphabetic and non-alphabetic characters, using the regexp ``\w+|[^\w\s]+``. >>> from nltk.tokenize import WordPunctTokenizer >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> WordPunctTokenizer().tokenize(s) ['Good', 'muffins', 'cost', '$', '3', '.', '88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] cCstj|dƒdS(Nu \w+|[^\w\s]+(RR(R ((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyRµs(R!R#R$R(((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pyR,ªs cCs"t||||ƒ}|j|ƒS(ur Return a tokenized copy of *text*. See :class:`.RegexpTokenizer` for descriptions of the arguments. (RR(RR RRRt tokenizer((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pytregexp_tokenize¼s(R$t __future__RRtnltk.tokenize.apiRtnltk.tokenize.utilRt nltk.compatRRR*R+R,R%R&R'R(R)R.Rtblankline_tokenizetwordpunct_tokenize(((sf/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/regexp.pytCs E