'log' rule is put with the stem, so that short stems like 'geo' 'theo' etc work like 'archaeo' 'philo' etc. This follows a suggestion of Barry Wilkins, research student at Birmingham. February 2000 the cvc test for not dropping final -e now looks after vc at the beginning of a word, so are, eve, ice, ore, use keep final -e. In this test c is any consonant, including w, x and y. This extension was suggested by Chris Emerson. -fully -> -ful treated like -fulness -> -ful, and -tionally -> -tion treated like -tional -> -tion both in Step 2. These were suggested by Hiranmay Ghosh, of New Delhi. Invariants proceed, succeed, exceed. Also suggested by Hiranmay Ghosh. Additional modifications were made to incorperate this module into nltk. All such modifications are marked with "--NLTK--". i(tprint_functiontunicode_literalsu plaintextN(tStemmerI(tpython_2_unicode_compatiblet PorterStemmercBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ddZdZdZdZRS(up A word stemmer based on the Porter stemming algorithm. Porter, M. "An algorithm for suffix stripping." Program 14.3 (1980): 130-137. A few minor modifications have been made to Porter's basic algorithm. See the source code of this module for more information. The Porter Stemmer requires that all tokens have string types. cCsi ddgd6dgd6dgd6dgd6d gd 6d d gd 6d d gd 6ddgd6dgd6dgd6dgd6dgd6}i|_x0|D](}x||D]}||j| b[i] is a consonant.uyii(RtFalsetTruet_cons(Rtwordti((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyRs cCsd}d}x:trH||kr%|S|j||s;Pn|d}qW|d}xtrx:tr||kru|S|j||rPn|d}q_W|d}|d}x:tr||kr|S|j||sPn|d}qW|d}qVWdS(uFm() measures the number of consonant sequences between k0 and j. if c is a consonant sequence and v a vowel sequence, and <..> indicates arbitrary presence, gives 0 vc gives 1 vcvc gives 2 vcvcvc gives 3 .... iiN(RR(RRtjtnR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_ms2           cCs7x0tt|D]}|j||stSqWtS(u3vowelinstem(stem) is TRUE <=> stem contains a vowel(trangetlenRRR (RtstemR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt _vowelinstemscCsHt|dkrtS|d|dkr.tS|j|t|dS(u;doublec(word) is TRUE <=> word ends with a double consonantiiii(RR R(RR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_doublecs cCs|dkrtS|dkr?|j|d o>|j|dS|j|| s|j||ds|j||d rtS||}|dks|dks|dkrtStS(ucvc(i) is TRUE <=> a) ( --NEW--) i == 1, and word[0] word[1] is vowel consonant, or b) word[i - 2], word[i - 1], word[i] has the form consonant - vowel - consonant and also if the second c is not w, x or y. this is used when trying to restore an e at the end of a short word. e.g. cav(e), lov(e), hop(e), crim(e), but snow, box, tray. iiiuwuxuy(R RR(RRRtch((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_cvcs  #@ $cCs4|ddkr|jdr,|d }q|jdrgt|dkrZ|d }q|d }q|ddkr|d }qnt}|jdrt|dkr|d }qs|d }n|jdr |j|t|dd krs|d }qsnj|jd r>|j|d r>|d }t}n5|jd rs|j|d rs|d }t}n|r0|jd s|jds|jdr|d7}q0|j|r|ddkr-|d }q-q0|j|t|ddkr0|j|t|dr0|d7}q0n|S(u"step1ab() gets rid of plurals and -ed or -ing. e.g. caresses -> caress ponies -> poni sties -> sti tie -> tie (--NEW--: see below) caress -> caress cats -> cat feed -> feed agreed -> agree disabled -> disable matting -> mat mating -> mate meeting -> meet milling -> mill messing -> mess meetings -> meet iusussesiuiesiuiedueediueduingiuatubluizueuluzi(ulusuz(tendswithRR RRRRR(RRted_or_ing_trimmed((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step1abs@     ""  "  - >cCsR|ddkrJt|dkrJ|j|t|drJ|d dS|SdS(ustep1c() turns terminal y to i when there is another vowel in the stem. --NEW--: This has been modified from the original Porter algorithm so that y->i is only done when y is preceded by a consonant, but not if the stem is only a single consonant, i.e. (*c and not c) Y -> I So 'happy' -> 'happi', but 'enjoy' -> 'enjoy' etc This is a much better rule. Formerly 'enjoy'->'enjoi' and 'enjoyment'-> 'enjoy'. Step 1c is perhaps done too soon; but with this modification that no longer really matters. Also, the removal of the vowelinstem(z) condition means that 'spy', 'fly', 'try' ... stem to 'spi', 'fli', 'tri' and conflate with 'spied', 'tried', 'flies' ... iuyiuiN(RR(RR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step1c?s> cCst|dkr|S|d}|dkr|jdrm|j|t|ddkri|d dS|S|jd r|j|t|d dkr|d S|S|Sn;|d krF|jd r|j|t|d dkr|d dS|S|jdr?|j|t|d dkr;|d dS|S|Sn|dkr|jdr|j|t|d dkr|d S|S|SnV|dkr|jdr|j|t|ddkr|d dS|S|jdr2|j|t|d dkr+|d }|j|S|Sq|jdri|j|t|dre|d S|S|jdr|j|t|dr|d S|S|jdr|j|t|dr|d S|S|jdr|j|t|dr |d S|S|Sn|d kr|jd!r\|j|t|drX|d d"S|S|jd#r|j|t|dr|d$ dS|S|jd%r|j|t|d r|d dS|S|Sn|d&kr|jd'r|j|t|dr|d S|S|jd(r|jd)rb|j|t|dr^|d S|S|jd*r|j|t|dr|d S|S|jd+r|j|t|dr|d S|S|Sq|Sn|d,kr|jd-r!|j|t|dr|d S|S|jd.r\|j|t|drX|d$ d/S|S|jd0r|j|t|d r|d1 dS|S|SnN|d2kr|jd3r|j|t|dr|d S|S|Sn|Sd4S(5ustep2() maps double suffices to single ones. so -ization ( = -ize plus -ation) maps to -ize etc. note that the string before the suffix must give m() > 0. iiuauationaliiiuateutionaliucuenciiiuenceuanciuanceueuizeriulubliiiubleualliufulliiuentliueliuousliuouizationuizeuationiuatorusualismunessuivenessufulnessuousnessutualitiuivitiuiveubilitiiugulogiN(RRRt_step2(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyR Ws  2. 22 . 2"  (((( ,,, (((( (,, (cCs|d}|dkr|jdrM|j|t|drI|d S|S|jdr|j|t|dr|d S|S|jdr|j|t|dr|d S|S|Sn|d kr |jd r|j|t|dr|d S|S|Sn|d kr|jd rO|j|t|d rK|d S|S|jdr|j|t|dr|d S|S|SnN|dkr|jdr|j|t|d r|d S|S|Sn|SdS(uEstep3() deals with -ic-, -full, -ness etc. similar strategy to step2.iueuicateiiuativeiualizeuiuicitiuluicaliiufuliusunessiN(RRR(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step3s0  ((( ( (( (cCst|dkr|S|d}|dkrp|jdri|j|t|ddkre|d S|S|Sn.|dkr|jdr|j|t|ddkr|d S|S|jd r|j|t|ddkr|d S|S|Sn|d krM|jd rF|j|t|ddkrB|d S|S|SnQ|d kr|jdr|j|t|ddkr|d S|S|Sn|dkr*|jdr|j|t|ddkr|d S|S|jdr#|j|t|ddkr|d S|S|Snt|dkr1|jdrs|j|t|ddkro|d S|S|jdr|j|t|ddkr|d S|S|jdr|j|t|ddkr|d S|S|jdr*|j|t|ddkr&|d S|S|Snm|dkr|jds[|jdr|j|t|ddkr|d S|S|jdr|j|t|ddkr|d S|S|Sn|dkr|jd r|j|t|ddkr|d S|S|Sn|d!kr|jd"rf|j|t|ddkrb|d S|S|jd#r|j|t|ddkr|d S|S|Sn|d$kr|jd%r|j|t|ddkr|d S|S|Sn|d&krJ|jd'rC|j|t|ddkr?|d S|S|SnT|d(kr|jd)r|j|t|ddkr|d S|S|Sn|Sd*S(+u:step4() takes off -ant, -ence etc., in context vcvc.iiuaualiucuanceiiuenceueueruiuiculuableuibleunuantiiuementiiumentuentuousionutionuouusuismutuateuitiuuuousuvuiveuzuizeN(RRR(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step4s  . .. . . .. .... .. . .. . . .cCs|ddkrq|j|t|d}|dksa|dkrq|j|t|d rq|d }qqn|jdr|j|t|ddkr|d }n|S(uYstep5() removes a final -e if m() > 1, and changes -ll to -l if m() > 1. iueiiull(RRRR(RRta((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step5+s51 icCs|dkr!|dkr!|}n0|dkr@t|d}n|||d!}||jkrk|j|St|dkr|S|j|}|j|}|j|}|j|}|j|}|j|}|S(uX Returns the stem of p, or, if i and j are given, the stem of p[i:j+1]. iiiN( tNoneRRRRR R!R"R$(RtpRRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt stem_word8s    cCsf|j}d}xMtt|D]9}||||krP|||7}q%|||7}q%W|S(Nu(tlowerRR(RRRR(trettx((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt _adjust_caseVs cCs5|j|jdt|d}|j||S(Nii(R'R(RR+(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyRws%cCsdS(Nu((R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt__repr__}sN(t__name__t __module__t__doc__R RRRRRRRR R!R"R$R%R'R+RR,(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyRbs" , '    C  \ # U  ! c Csnddlm}ddlm}|j}g}g}x[|jd D]I}x@|j|D]/\}}|j||j|j|q_WqIWdj|}t j dd|dj }dj|} t j dd| dj } t dj d jdd jd dt | t d j d jdd jd dt |t d d d S(u^ A demonstration of the porter stemmer on a sample from the Penn Treebank corpus. i(ttreebank(Riu u (.{,70})\su\1\nu -Original-iFu*u-u -Results-N(t nltk.corpusR0tnltkRRtfilest tagged_wordstappendtjointretsubtrstriptprinttcentertreplace( R0RtstemmertorigtstemmedtitemRttagtresultstoriginal((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pytdemos$  + + ( R/t __future__RRt __docformat__R7t nltk.stem.apiRt nltk.compatRRRD(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pytUs 0