ó <¿CVc@sVdZddlmZddlmZdefd„ƒYZdefd„ƒYZdS( s Tokenizer Interface iÿÿÿÿ(t overridden(tstring_span_tokenizet TokenizerIcBs2eZdZd„Zd„Zd„Zd„ZRS(s† A processing interface for tokenizing a string. Subclasses must define ``tokenize()`` or ``tokenize_sents()`` (or both). cCs0t|jƒr#|j|gƒdStƒ‚dS(sN Return a tokenized copy of *s*. :rtype: list of str iN(Rttokenize_sentstNotImplementedError(tselfts((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyttokenizescCs tƒ‚dS(s· Identify the tokens using integer offsets ``(start_i, end_i)``, where ``s[start_i:end_i]`` is the corresponding token. :rtype: iter(tuple(int, int)) N(R(RR((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyt span_tokenize scCs g|D]}|j|ƒ^qS(s« Apply ``self.tokenize()`` to each element of ``strings``. I.e.: return [self.tokenize(s) for s in strings] :rtype: list(list(str)) (R(RtstringsR((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyR)sccs)x"|D]}t|j|ƒƒVqWdS(sÁ Apply ``self.span_tokenize()`` to each element of ``strings``. I.e.: return [self.span_tokenize(s) for s in strings] :rtype: iter(list(tuple(int, int))) N(tlistR(RR R((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pytspan_tokenize_sents3s (t__name__t __module__t__doc__RRRR (((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyRs  tStringTokenizercBs eZdZd„Zd„ZRS(sxA tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses). cCs|j|jƒS(N(tsplitt_string(RR((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyRDsccs&xt||jƒD] }|VqWdS(N(RR(RRtspan((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyRGs(R R RRR(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyR?s N(Rtnltk.internalsRtnltk.tokenize.utilRtobjectRR(((sc/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/api.pyt s/