Zdd >Zdd >Zdd >Zdd >Zdd >ZeeeZeeeZieddf6eddf6eddf6eddf6eddf6eddf6ZdZdZdZdZdZdZdZde fdYZ!ej"dej#Z$dZ%de fdYZ&e de fd YZ'd!e fd"YZ(d#e(fd$YZ)d%e(efd&YZ*d'Z+d(Z,e*e)d)Z-dS(*u Punkt Sentence Tokenizer This tokenizer divides a text into a list of sentences, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences. It must be trained on a large collection of plaintext in the target language before it can be used. The NLTK data package includes a pre-trained Punkt tokenizer for English. >>> import nltk.data >>> text = ''' ... Punkt knows that the periods in Mr. Smith and Johann S. Bach ... do not mark sentence boundaries. And sometimes sentences ... can start with non-capitalized words. i is a good variable ... name. ... ''' >>> sent_detector = nltk.data.load('tokenizers/punkt/english.pickle') >>> print('\n-----\n'.join(sent_detector.tokenize(text.strip()))) Punkt knows that the periods in Mr. Smith and Johann S. Bach do not mark sentence boundaries. ----- And sometimes sentences can start with non-capitalized words. ----- i is a good variable name. (Note that whitespace from the original text, including newlines, is retained in the output.) Punctuation following sentences is also included by default (from NLTK 3.0 onwards). It can be excluded with the realign_boundaries flag. >>> text = ''' ... (How does it deal with this parenthesis?) "It should be part of the ... previous sentence." "(And the same with this one.)" ('And this one!') ... "('(And (this)) '?)" [(and this. )] ... ''' >>> print('\n-----\n'.join( ... sent_detector.tokenize(text.strip()))) (How does it deal with this parenthesis?) ----- "It should be part of the previous sentence." ----- "(And the same with this one.)" ----- ('And this one!') ----- "('(And (this)) '?)" ----- [(and this. )] >>> print('\n-----\n'.join( ... sent_detector.tokenize(text.strip(), realign_boundaries=False))) (How does it deal with this parenthesis? ----- ) "It should be part of the previous sentence. ----- " "(And the same with this one. ----- )" ('And this one! ----- ') "('(And (this)) '? ----- )" [(and this. ----- )] However, Punkt is designed to learn parameters (a list of abbreviations, etc.) unsupervised from a corpus similar to the target domain. The pre-packaged models may therefore be unsuitable: use ``PunktSentenceTokenizer(text)`` to learn parameters from the given text. :class:`.PunktTrainer` learns parameters such as a list of abbreviations (without supervision) from portions of text. Using a ``PunktTrainer`` directly allows for incremental training and modification of the hyper-parameters used to decide what is considered an abbreviation, etc. The algorithm for this tokenizer is described in:: Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence Boundary Detection. Computational Linguistics 32: 485-525. i(tprint_functiontunicode_literalsN(t defaultdict(t unicode_reprtpython_2_unicode_compatiblet string_types(tFreqDist(t TokenizerIiiiiiiuinitialuupperuinternaluunknownulowerudefault decisionuknown collocation (both words)u%abbreviation + orthographic heuristicu(abbreviation + frequent sentence starteru initial + orthographic heuristicu(initial + special orthographic heuristictPunktLanguageVarscBseZdZdZdZdZdZedZd Z e j d e j Z d Zd Zd ZdZdZdZdZdZRS(uX Stores variables, mostly regular expressions, which may be language-dependent for correct application of the algorithm. An extension of this class may modify its properties to suit a language other than English; an instance can then be passed as an argument to PunktSentenceTokenizer and PunktTrainer constructors. u_re_period_contextu_re_word_tokenizercCsdS(Ni((tself((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt __getstate__scCsdS(Ni((R tstate((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt __setstate__su.u?u!cCsdtjdj|jS(Nu[%s]u(tretescapetjointsent_end_chars(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt_re_sent_end_charssu,:;u["\')\]}]+?(?:\s+|(?=--)|$)u[^\(\"\`{\[:;&\#\*@\)}\]\-,]u(?:[?!)\";}\]\*:@\'\({\[])u (?:\-{2,}|\.{2,}|(?:\.\s){2,}\.)u( %(MultiChar)s | (?=%(WordStart)s)\S+? # Accept word characters until end is found (?= # Sequences marking a word's end \s| # White-space $| # End-of-string %(NonWord)s|%(MultiChar)s| # Punctuation ,(?=$|\s|%(NonWord)s|%(MultiChar)s) # Comma if at end of word ) | \S )cCsky |jSWnYtk rftj|ji|jd6|jd6|jd6tjtj B|_|jSXdS(u?Compiles and returns a regular expression for word tokenizationuNonWordu MultiCharu WordStartN( t_re_word_tokenizertAttributeErrorR tcompilet_word_tokenize_fmtt_re_non_word_charst_re_multi_char_punctt_re_word_starttUNICODEtVERBOSE(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt_word_tokenizer_res     cCs|jj|S(u=Tokenize a string to split off punctuation other than periods(Rtfindall(R ts((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt word_tokenizesu: \S* # some word material %(SentEndChars)s # a potential sentence ending (?=(?P %(NonWord)s # either other punctuation | \s+(?P\S+) # or whitespace and some other token ))cCsWy |jSWnEtj|ji|jd6|jd6tjtjB|_|jSXdS(ujCompiles and returns a regular expression to find contexts including possible sentence boundaries.uNonWordu SentEndCharsN(t_re_period_contextR Rt_period_context_fmtRRRR(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytperiod_context_res   (u_re_period_contextu_re_word_tokenizer(u.u?u!(t__name__t __module__t__doc__t __slots__R R RtpropertyRtinternal_punctuationR Rt MULTILINEtre_boundary_realignmentRRRRRRR R!(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs"      u[^\W\d]ccsIt|}t|}x|D]}||fV|}qW|dfVdS(u Yields pairs of tokens from the given iterator such that each input token will appear as the first element in a yielded tuple. The last pair will have None as its second element. N(titertnexttNone(tittprevtel((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt _pair_iter/s      tPunktParameterscBsMeZdZdZdZdZdZdZdZdZ RS(uCStores data used to perform sentence boundary detection with Punkt.cCs7t|_t|_t|_tt|_dS(N(tsett abbrev_typest collocationst sent_startersRtintt ortho_context(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt__init__Cs    cCst|_dS(N(R2R3(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt clear_abbrevsWscCst|_dS(N(R2R4(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytclear_collocationsZscCst|_dS(N(R2R5(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytclear_sent_starters]scCstt|_dS(N(RR6R7(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytclear_ortho_context`scCs|j|c|Osu %s(%s,%s %s)(RKRIRRRNt __class__R"(R ttypestrtpropvals((R se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt__repr__s +  cCsO|j}|jr|d7}n|jr5|d7}n|jrK|d7}n|S(uO A string representation akin to that used by Kiss and Strunk. uuu(RItabbrtellipsisRX(R tres((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt__str__s       (R"R#R$RNR%R8R RR_RSRRdRfRJR&RWRYR[R]R^RaRcReRgRjRpRt(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRHys,     tPunktBaseClasscBsAeZdZeeedZdZdZdZ RS(uP Includes common components of PunktTrainer and PunktSentenceTokenizer. cCs||_||_||_dS(N(t_paramst _lang_varst_Token(R t lang_varst token_clsRP((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR8 s   ccst}x|jdD]x}|jrt|jj|}|jt|d|dtVt}x%|D]}|j|VqmWqt}qWdS(uB Divide the given text into tokens, using the punkt word segmentation regular expression, and generate the resulting list of tokens augmented as three-tuples with two boolean values for whether the given token occurs at the start of a paragraph or a new line, respectively. u t parastartt linestartN( tFalsetsplittstripR*RwRRxR+tTrue(R t plaintextR{tlinet line_tokstt((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt_tokenize_wordss  ccs'x |D]}|j||VqWdS(u Perform the first pass of annotation, which makes decisions based purely based on the word type of each word: - '?', '!', and '.' are marked as sentence breaks. - sequences of two or more periods are marked as ellipsis. - any word ending in '.' that's a known abbreviation is marked as an abbreviation. - any other word ending in '.' is marked as a sentence break. Return these annotations as a tuple of three sets: - sentbreak_toks: The indices of all sentence breaks. - abbrev_toks: The indices of all abbreviations. - ellipsis_toks: The indices of all ellipsis marks. N(t_first_pass_annotation(R ttokenstaug_tok((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt_annotate_first_pass0s  cCs|j}||jjkr't|_n|jr<t|_nv|jr|jd r|d j |j j ks|d j j dd|j j krt|_ qt|_ndS(uC Performs type-based annotation on a single token. u..iu-N(RIRwRRRXRaRrRMRLRURvR3R~Rq(R RRI((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyREs    )  ( R"R#R$RRHR1R8RRR(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRus    t PunktTrainercBs(eZdZdeeedZdZdZ eZ dZ dZ dZ eZeZdZeedZeed Zd Zd Zed Zd d d d dZdZdZdZdZdZedZedZdZ dZ!dZ"dZ#dZ$RS(u<Learns parameters used in Punkt sentence boundary detection.cCs{tj|d|d|t|_d|_t|_t|_d|_t|_ |rw|j ||dtndS(NRyRzitfinalize( RuR8Rt _type_fdistt_num_period_tokst_collocation_fdistt_sent_starter_fdistt_sentbreak_countRt _finalizedttrain(R t train_texttverboseRyRz((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR8bs      cCs|js|jn|jS(ul Calculates and returns parameters for sentence boundary detection as derived from training.(Rtfinalize_trainingRv(R ((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt get_paramss  g333333?igQ@iicCs3|j|j|||r/|j|ndS(u8 Collects training data from a given text. If finalize is True, it will determine all the parameters for sentence boundary detection. If not, this will be delayed until get_params() or finalize_training() is called. If verbose is True, abbreviations found will be listed. N(t _train_tokensRR(R ttextRR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs cs:jfd|D||r6j|ndS(uE Collects training data from a given list of tokens. c3s|]}j|VqdS(N(Rx(RlR(R (se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pys sN(RR(R RRR((R se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt train_tokenss c Cs#t|_t|}x?|D]7}|j|jcd7<|jr|jd7_qqW|j|}x|j|D]\}}}||j kr|r |j j j ||rt d||fqq qv|sv|j j j||r t d||fq qvqvWt|j|}|j||j|j|7_xt|D]\}} |j sT| rwqTn|j|| r|j j j |j|rt d|jqn|j| |r|j| jcd7s(R2(R R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRscCs|jjxJ|jD]<\}}|jjj||rtd||fqqW|jjxY|jD]K\\}}}|jjj||f|rttd|||fqtqtWt |_ dS(u~ Uses data that has been gathered in training to determine likely collocations and sentence starters. u Sent Starter: [%6.4f] %ru Collocation: [%6.4f] %r+%rN( RvR;t_find_sent_startersR5RRR:t_find_collocationsR4RR(R RR=tllttyp1ttyp2((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs  icCs|dkrl|jj}|jjxD|jD]6}|j|}||kr/|||jj| count_removed). iiN(RR,(R tfdistt thresholdRst num_removedRIR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRDs     cCsd}t|}x|D]}|jr=|dkr=d}n|jr[|dkr[d}n|j}tj||jfd}|r|jj||n|j r|j p|j sd}qd}q|j s|j rd}qd}qWdS(u Collect information about whether each token type occurs with different case patterns (i) overall, (ii) at sentence-initial positions, and (iii) at sentence-internal positions. uinternaluunknownuinitialiN(RR{R|RYt _ORTHO_MAPtgetR^RvR?RXRcReRrRq(R RtcontextRR=R>((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRZs$         c csTxM|D]E}tj| s|dkr/qn|jdri||jjkrVqn|d }t}n||jjkrqnt}|jdd}t||d}|j |d}|j |}|j |||j ||j j }t j| } |} t|jp)t j|| } || | | } || |fVqWdS(u (Re)classifies each given token if - it is period-final and not a known abbreviation; or - it is not period-final and is otherwise a known abbreviation by checking whether its previous classification still holds according to the heuristics of section 3. Yields triples (abbr, score, is_add) where abbr is the type in question, score is its log-likelihood with penalties applied, and is_add specifies whether the present type is a candidate for inclusion or exclusion as an abbreviation, such that: - (is_add and score >= 0.3) suggests a new abbreviation; and - (not is_add and score < 0.3) suggests excluding an abbreviation. u ##number##u.iiN(RhRiRLRvR3RR}RRVRt_dunning_log_likelihoodRtNtmathtexpR6tIGNORE_ABBREV_PENALTYtpow( R ttypesR=Rt num_periodstnum_nonperiodstcount_with_periodtcount_without_periodRtf_lengtht f_periodst f_penaltyR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs2     cCsl|jjd|jD}xE|j|D]4\}}}||jkr0|jjj|q0q0WdS(u Recalculates abbreviations given type frequencies, despite no prior determination of abbreviations. This fails to include abbreviations otherwise found as "rare". css*|] }|r|jdr|VqdS(u.N(RL(RlR=((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pys sN(RvR9RRRR3R(R RRqRR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytfind_abbrev_typess  cCs|js|j rtS|j}|j||j|d }||jjks]||jkratS|jd |j j kr~t S|j r|j}|jj |}|t@r|t@ rt SndS(u A word type is counted as a rare abbreviation if... - it's not already marked as an abbreviation - it occurs fewer than ABBREV_BACKOFF times - either it is followed by a sentence-internal punctuation mark, *or* it is followed by a lower-case word that sometimes appears with upper case, but never occurs with lower case at the beginning of sentences. iiN(RqRXR}RYRRvR3tABBREV_BACKOFFRIRwR'RR]R7R@RA(R tcur_toktnext_tokR=RRttyp2ortho_context((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs  !    c Cst||}d}t|tj|||tjd|}t|tj|||tjd|}||}d|S(u A function that calculates the modified Dunning log-likelihood ratio scores for abbreviation candidates. The details of how this works is available in the paper. gGz?g?g(tfloatRtlog( tcount_atcount_btcount_abRtp1tp2t null_hypotalt_hypot likelihood((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs c CsJddl}d||}d||}d||||}||j||||jd|}|||j||||||jd|} ||krd} n,||j||||jd|} ||krd} n8|||j||||||jd|} || | | } d| S(u= A function that will just compute log-likelihood estimate, in the original paper it's described in algorithm 6 and 7. This *should* be the original Dunning log-likelihood values, unlike the previous log_l function where it used modified Dunning log-likelihood values iNg?ig(RR( RRRRRRQRRtsummand1tsummand2tsummand3tsummand4R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt_col_log_likelihoods$ $    $cCsF|js6|jr|js6|joE|js6|joE|joE|jS(ut Returns True if the pair of tokens may form a collocation given log-likelihood statistics. (tINCLUDE_ALL_COLLOCStINCLUDE_ABBREV_COLLOCSRqRXRcReRj(R RR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyREs    ccsLxE|jD]:}y|\}}Wntk r6q nX||jjkrOq n|j|}|j||j|d}|j||j|d}|dkr |dkr |j|kot||knr |j||||jj}||j krDt |jj|t ||krD||f|fVqDq q WdS(uI Generates likely collocations and their log-likelihood. u.iN( Rt TypeErrorRvR5RtMIN_COLLOC_FREQtminRRt COLLOCATIONR(R RRRt col_countt typ1_countt typ2_countR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRQs&  " cCs#|jo"|jp|j o"|jS(u Returns True given a token and the token that preceds it if it seems clear that the token is beginning a sentence. (RXRcReRg(R Rtprev_tok((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRqs ccsx|jD]}|sq n|j|}|j||j|d}||krWq n|j|j|||jj}||jkr t|jj|jt||kr ||fVq q WdS(u~ Uses collocation heuristics for each candidate token to determine if it frequently starts sentences. u.N(RRRRRt SENT_STARTERR(R R=ttyp_at_break_countt typ_countR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR}s  cCstd|DS(uj Returns the number of sentence breaks marked in a given set of augmented tokens. css|]}|jrdVqdS(iN(RX(RlR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pys s(tsum(R R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRsN(%R"R#R$R,R}RRHR8RRRRRRRRRRRRRRRRRRRRRt staticmethodRRRRRRR(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR_s>(    >      0 @  /* tPunktSentenceTokenizercBseZdZdeeedZedZe dZ dZ e dZ e dZ dZdZd Zd Zd Zd Zd ZdZedZdZdZdZRS(u' A sentence tokenizer which uses an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences; and then uses that model to find sentence boundaries. This approach has been shown to work well for many European languages. cCs;tj|d|d||r7|j|||_ndS(u train_text can either be the sole training text for this sentence boundary detector, or can be a PunktParameters object. RyRzN(RuR8RRv(R RRRyRz((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR8scCs5t|ts|St|d|jd|jjS(u Derives parameters from a given training text, or uses the parameters given. Repeated calls to this method destroy previous parameters. For incremental training, instantiate a separate PunktTrainer instance. RyRz(t isinstanceRRRwRxR(R RR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRscCst|j||S(uM Given a text, returns a list of the sentences in that text. (Rtsentences_from_text(R Rtrealign_boundaries((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyttokenizesccsxz|jjj|D]`}|j|jd}|j|}t|j|}x|djs||jdq_Wt d|j dd|d|dj d|dj dt |dj d t |djd |dj|jjkd |j|dd t|jj|djd |dj|djf|jjkd|j|d|dpgtd|dj VqWdS(u Classifies candidate periods as sentence breaks, yielding a dict for each that may be used to understand why the decision was made. See format_debug_decision() to help make this output readable. u after_tokit period_indexiRttype1ttype2ttype1_in_abbrsttype1_is_initialttype2_is_sent_starterttype2_ortho_heuristicttype2_ortho_contextst collocationtreasontbreak_decisionN(RwR!tfinditertgroupRRRRMtpoptdicttendRKtboolRqReRYRvR5t_ortho_heuristicR2RGR4t_second_pass_annotationtREASON_DEFAULT_DECISIONRX(R RR`t decision_textR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytdebug_decisionss$  & cCsM|j|}|r*|j||}ng|D]}|j|jf^q1S(uj Given a text, returns a list of the (start, end) spans of sentences in the text. (t_slices_from_textt_realign_boundarieststarttstop(R RRtslicestsl((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt span_tokenizescCs0g|j||D]\}}|||!^qS(u Given a text, generates the sentences in that text by only testing candidate sentence breaks. If realign_boundaries is True, includes in the sentence closing punctuation that follows the period. (R(R RRRte((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRsccsd}x|jjj|D]r}|j|jd}|j|rt||jV|jdr|jd}q|j}qqWt|t|VdS(Niu after_tokunext_tok( RwR!RRttext_contains_sentbreaktsliceRR RV(R Rt last_breakR`R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR sccsd}xt|D]\}}t|j||j}|sV||r|Vqqn|jjj||}|rt|j|jt|jdj V|j }qd}||r|VqqWdS(u@ Attempts to realign punctuation that falls after the period but should otherwise be included in the same sentence. For example: "(Sent1.) Sent2." will otherwise be split as:: ["(Sent1.", ") Sent1."]. This method will produce:: ["(Sent1.)", "Sent2."]. iN( R0RR R RwR)R`RVRtrstripR(R RRtrealigntsl1tsl2tm((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR  s  - cCsIt}x<|j|j|D]"}|r/tS|jrt}qqWtS(uK Returns True if the given text includes a sentence break. (R}t_annotate_tokensRRRX(R RtfoundR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR(s  cCs(|j|j|}|j||S(u Given a text, generates the sentences in that text. Annotates all tokens, rather than just those with possible sentence breaks. Should produce the same results as ``sentences_from_text``. (RRt_build_sentence_list(R RR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytsentences_from_text_legacy4sc#sutjfd|D}g}x5|D]-}|j|j|jr2|Vg}q2q2W|rq|VndS(uw Given a sequence of tokens, generates lists of tokens, each list corresponding to a sentence. c3s|]}j|VqdS(N(Rx(RlR(R (se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pys BsN(R*RtappendRIRX(R RtsentenceR((R se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytsentences_from_tokens=s%   cCs"|j|}|j|}|S(u Given a set of tokens augmented with markers for line-start and paragraph-start, returns an iterator through those tokens with full annotation including predicted sentence breaks. (Rt_annotate_second_pass(R R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRLsc cs?d}tjd}d}x|D]}|j}|j||j}|t|7}|||t|!|krdjd|D} tj| j||} | r| j}qn|||t|!|kst|t|7}|r||7}n||7}|jr"|Vd}q"q"W|r;|VndS(u Given the original text and the list of augmented word tokens, construct and return a tokenized list of sentence strings. iu\s*ucss|]}tj|VqdS(N(R R(RlRF((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pys ~sN( R RRIR`RRVRtAssertionErrorRX( R RRtpost WS_REGEXPRRRItwstpatR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRas,   #    cCstdtddj}x`|D]X}|jrB|jdn&|jr[|jdn |jd|jt|q#WWdQXdS(Nuwriting to /tmp/punkt.new...u/tmp/punkt.newuwu u u (RtopenR{twriteR|tstr(R RtoutfileR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytdumps     u;:,.!?ccs6x/t|D]!\}}|j|||Vq WdS(u Performs a token-based classification (section 4) over the given tokens, making use of the orthographic heuristic (4.1.1), collocation heuristic (4.1.2) and frequent sentence starter heuristic (4.1.3). N(R0R(R Rtt1tt2((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR!sc Cs{|s dS|j}|js dS|j}|j}|j}|j}||f|jjkrrt|_t |_ t S|j s|j r| r|j |}|t krt |_tS|jr||jjkrt |_tSn|s|dkrw|j |}|tkr.t|_t |_ |r'tStSn|dkrw|rw|jrw|jj|t@ rwt|_t |_ tSndS(ur Performs token-based classification over a pair of contiguous tokens updating the first. Nu ##number##uunknown(RIRMRWRYReRvR4R}RXRRqtREASON_KNOWN_COLLOCATIONRrRt'REASON_ABBR_WITH_ORTHOGRAPHIC_HEURISTICR[R5t!REASON_ABBR_WITH_SENTENCE_STARTERt*REASON_INITIAL_WITH_ORTHOGRAPHIC_HEURISTICt)REASON_NUMBER_WITH_ORTHOGRAPHIC_HEURISTICR7t _ORTHO_LCt2REASON_INITIAL_WITH_SPECIAL_ORTHOGRAPHIC_HEURISTIC( R RRRIR=Rtnext_typttok_is_initialtis_sent_starter((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRsL                  cCsq|j|jkrtS|jj|j}|jrK|t@rK|t@ rKt S|j rm|t @si|t @ rmtSdS(uR Decide whether the given token is the first token in a sentence. uunknown( RIt PUNCTUATIONR}RvR7RYR[R3RARR]t _ORTHO_UCRC(R RR7((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyR s      N(R"R#R$R,R}RRHR8RRRR RRR R RRR RRR+ttupleR8R!RR(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyRs(        6   RuText: %(text)r (at offset %(period_index)d) Sentence break? %(break_decision)s (%(reason)s) Collocation? %(collocation)s %(type1)r: known abbreviation: %(type1_in_abbrs)s is initial: %(type1_is_initial)s %(type2)r: known sentence starter: %(type2_is_sent_starter)s orthographic heuristic suggests is a sentence starter? %(type2_ortho_heuristic)s orthographic contexts in training: %(type2_ortho_contexts)s cCst|S(N(tDEBUG_DECISION_FMT(td((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytformat_debug_decision4scCshd}|}t|_|j|||j}x'|j|D]}t||qJWdS(u4Builds a punkt model and applies it to the same textcSs+tjdtjjd|jddS(Nu (?:\r|^\s+)uu u (R RR(RTtreplace(R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pyt9sN(RRRRRR(Rttok_clst train_clstcleanupttrainertsbdtl((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytdemo7s    (.R$t __future__RRR Rt collectionsRt nltk.compatRRRtnltk.probabilityRtnltk.tokenize.apiRR@RARBRCRDRER9R3RRR.R/R0R1R2R4tobjectRRRRhR0R1RHRuRRR;R=RF(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/tokenize/punkt.pytesX               n 9[F