ó <¿CVc@swdZddlmZmZdZddlZddlmZddlm Z e defd„ƒYƒZ d „Z dS( u8 Porter Stemmer This is the Porter stemming algorithm, ported to Python from the version coded up in ANSI C by the author. It follows the algorithm presented in Porter, M. "An algorithm for suffix stripping." Program 14.3 (1980): 130-137. only differing from it at the points marked --DEPARTURE-- and --NEW-- below. For a more faithful version of the Porter algorithm, see http://www.tartarus.org/~martin/PorterStemmer/ Later additions: June 2000 The 'l' of the 'logi' -> 'log' rule is put with the stem, so that short stems like 'geo' 'theo' etc work like 'archaeo' 'philo' etc. This follows a suggestion of Barry Wilkins, research student at Birmingham. February 2000 the cvc test for not dropping final -e now looks after vc at the beginning of a word, so are, eve, ice, ore, use keep final -e. In this test c is any consonant, including w, x and y. This extension was suggested by Chris Emerson. -fully -> -ful treated like -fulness -> -ful, and -tionally -> -tion treated like -tional -> -tion both in Step 2. These were suggested by Hiranmay Ghosh, of New Delhi. Invariants proceed, succeed, exceed. Also suggested by Hiranmay Ghosh. Additional modifications were made to incorperate this module into nltk. All such modifications are marked with "--NLTK--". iÿÿÿÿ(tprint_functiontunicode_literalsu plaintextN(tStemmerI(tpython_2_unicode_compatiblet PorterStemmercBs¤eZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z d „Z d „Z d „Zd dd„Zd„Zd„Zd„ZRS(up A word stemmer based on the Porter stemming algorithm. Porter, M. "An algorithm for suffix stripping." Program 14.3 (1980): 130-137. A few minor modifications have been made to Porter's basic algorithm. See the source code of this module for more information. The Porter Stemmer requires that all tokens have string types. cCsèi ddgd6dgd6dgd6dgd6d gd 6d d gd 6d d gd 6ddgd6dgd6dgd6dgd6dgd6}i|_x0|D](}x||D]}||j| b[i] is a consonant.uyii(RtFalsetTruet_cons(Rtwordti((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyR¤s cCsüd}d}x:trH||kr%|S|j||ƒs;Pn|d}qW|d}x¢tr÷x:tr˜||kru|S|j||ƒr‹Pn|d}q_W|d}|d}x:tré||krÆ|S|j||ƒsÜPn|d}q°W|d}qVWdS(uFm() measures the number of consonant sequences between k0 and j. if c is a consonant sequence and v a vowel sequence, and <..> indicates arbitrary presence, gives 0 vc gives 1 vcvc gives 2 vcvcvc gives 3 .... iiN(RR(RRtjtnR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_m¯s2           cCs7x0tt|ƒƒD]}|j||ƒstSqWtS(u3vowelinstem(stem) is TRUE <=> stem contains a vowel(trangetlenRRR (RtstemR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt _vowelinstemÖscCsHt|ƒdkrtS|d|dkr.tS|j|t|ƒdƒS(u;doublec(word) is TRUE <=> word ends with a double consonantiiÿÿÿÿiþÿÿÿi(RR R(RR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_doublecÝs cCs¹|dkrtS|dkr?|j|dƒ o>|j|dƒS|j||ƒ s|j||dƒs|j||dƒ rƒtS||}|dks±|dks±|dkrµtStS(u©cvc(i) is TRUE <=> a) ( --NEW--) i == 1, and word[0] word[1] is vowel consonant, or b) word[i - 2], word[i - 1], word[i] has the form consonant - vowel - consonant and also if the second c is not w, x or y. this is used when trying to restore an e at the end of a short word. e.g. cav(e), lov(e), hop(e), crim(e), but snow, box, tray. iiiuwuxuy(R RR(RRRtch((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_cvcås  #@ $cCs4|ddkr‡|jdƒr,|d }q‡|jdƒrgt|ƒdkrZ|d }q„|d }q‡|ddkr‡|d }q‡nt}|jdƒrÈt|ƒdkr»|d }qs|d }n«|jdƒr |j|t|ƒdƒd krs|d }qsnj|jd ƒr>|j|d ƒr>|d }t}n5|jd ƒrs|j|d ƒrs|d }t}n|r0|jd ƒs¦|jdƒs¦|jdƒr³|d7}q0|j|ƒrâ|ddkr-|d }q-q0|j|t|ƒdƒdkr0|j|t|ƒdƒr0|d7}q0n|S(u"step1ab() gets rid of plurals and -ed or -ing. e.g. caresses -> caress ponies -> poni sties -> sti tie -> tie (--NEW--: see below) caress -> caress cats -> cat feed -> feed agreed -> agree disabled -> disable matting -> mat mating -> mate meeting -> meet milling -> mill messing -> mess meetings -> meet iÿÿÿÿusussesiþÿÿÿuiesiuiedueediueduingiýÿÿÿuatubluizueuluzi(ulusuz(tendswithRR RRRRR(RRted_or_ing_trimmed((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step1abüs@     ""  "  - >cCsR|ddkrJt|ƒdkrJ|j|t|ƒdƒrJ|d dS|SdS(ustep1c() turns terminal y to i when there is another vowel in the stem. --NEW--: This has been modified from the original Porter algorithm so that y->i is only done when y is preceded by a consonant, but not if the stem is only a single consonant, i.e. (*c and not c) Y -> I So 'happy' -> 'happi', but 'enjoy' -> 'enjoy' etc This is a much better rule. Formerly 'enjoy'->'enjoi' and 'enjoyment'-> 'enjoy'. Step 1c is perhaps done too soon; but with this modification that no longer really matters. Also, the removal of the vowelinstem(z) condition means that 'spy', 'fly', 'try' ... stem to 'spi', 'fli', 'tri' and conflate with 'spied', 'tried', 'flies' ... iÿÿÿÿuyiuiN(RR(RR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step1c?s> cCsðt|ƒdkr|S|d}|dkr±|jdƒrm|j|t|ƒdƒdkri|d dS|S|jd ƒrª|j|t|ƒd ƒdkr¦|d S|S|Sn;|d krF|jd ƒrþ|j|t|ƒd ƒdkrú|d dS|S|jdƒr?|j|t|ƒd ƒdkr;|d dS|S|Sn¦|dkr–|jdƒr�|j|t|ƒd ƒdkr‹|d S|S|SnV|dkr|jdƒrã|j|t|ƒdƒdkrß|d dS|S|jdƒr2|j|t|ƒd ƒdkr+|d }|j|ƒS|Sqì|jdƒri|j|t|ƒdƒre|d S|S|jdƒr |j|t|ƒdƒrœ|d S|S|jdƒr×|j|t|ƒdƒrÓ|d S|S|jdƒr|j|t|ƒdƒr |d S|S|Sn×|d krÙ|jd!ƒr\|j|t|ƒdƒrX|d d"S|S|jd#ƒr—|j|t|ƒdƒr“|d$ dS|S|jd%ƒrÒ|j|t|ƒd ƒrÎ|d dS|S|Sn|d&krÞ|jd'ƒr|j|t|ƒdƒr|d S|S|jd(ƒr×|jd)ƒrb|j|t|ƒdƒr^|d S|S|jd*ƒr™|j|t|ƒdƒr•|d S|S|jd+ƒrÐ|j|t|ƒdƒrÌ|d S|S|Sqì|Sn|d,krž|jd-ƒr!|j|t|ƒdƒr|d S|S|jd.ƒr\|j|t|ƒdƒrX|d$ d/S|S|jd0ƒr—|j|t|ƒd ƒr“|d1 dS|S|SnN|d2krè|jd3ƒrá|j|t|ƒdƒrÝ|d S|S|Sn|Sd4S(5u³step2() maps double suffices to single ones. so -ization ( = -ize plus -ation) maps to -ize etc. note that the string before the suffix must give m() > 0. iiþÿÿÿuauationaliiiùÿÿÿuateutionaliucuenciiiüÿÿÿuenceuanciuanceueuizeriÿÿÿÿulubliiiýÿÿÿubleualliufulliiuentliueliuousliuouizationuizeuationiûÿÿÿuatorusualismunessuivenessufulnessuousnessutualitiuivitiuiveubilitiiúÿÿÿugulogiN(RRRt_step2(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyR WsŠ  2. 22 . 2"  (((( ,,, (((( (,, (cCsß|d}|dkrÂ|jdƒrM|j|t|ƒdƒrI|d S|S|jdƒr„|j|t|ƒdƒr€|d S|S|jdƒr»|j|t|ƒdƒr·|d S|S|Sn|d kr |jd ƒr|j|t|ƒdƒr|d S|S|SnÏ|d kr�|jd ƒrO|j|t|ƒd ƒrK|d S|S|jdƒr†|j|t|ƒdƒr‚|d S|S|SnN|dkr×|jdƒrÐ|j|t|ƒd ƒrÌ|d S|S|Sn|SdS(uEstep3() deals with -ic-, -full, -ness etc. similar strategy to step2.iÿÿÿÿueuicateiiýÿÿÿuativeiûÿÿÿualizeuiuicitiuluicaliiþÿÿÿufuliusunessiüÿÿÿN(RRR(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step3³s0  ((( ( (( (cCs¢t|ƒdkr|S|d}|dkrp|jdƒri|j|t|ƒdƒdkre|d S|S|Sn.|dkrý|jdƒr¹|j|t|ƒdƒdkrµ|d S|S|jd ƒrö|j|t|ƒdƒdkrò|d S|S|Sn¡|d krM|jd ƒrF|j|t|ƒdƒdkrB|d S|S|SnQ|d kr�|jdƒr–|j|t|ƒdƒdkr’|d S|S|Sn|dkr*|jdƒræ|j|t|ƒdƒdkrâ|d S|S|jdƒr#|j|t|ƒdƒdkr|d S|S|Snt|dkr1|jdƒrs|j|t|ƒdƒdkro|d S|S|jdƒr°|j|t|ƒdƒdkr¬|d S|S|jdƒrí|j|t|ƒdƒdkré|d S|S|jdƒr*|j|t|ƒdƒdkr&|d S|S|Snm|dkrÍ|jdƒs[|jdƒr‰|j|t|ƒdƒdkr…|d S|S|jdƒrÆ|j|t|ƒdƒdkrÂ|d S|S|SnÑ|dkr|jd ƒr|j|t|ƒdƒdkr|d S|S|Sn�|d!krª|jd"ƒrf|j|t|ƒdƒdkrb|d S|S|jd#ƒr£|j|t|ƒdƒdkrŸ|d S|S|Snô|d$krú|jd%ƒró|j|t|ƒdƒdkrï|d S|S|Sn¤|d&krJ|jd'ƒrC|j|t|ƒdƒdkr?|d S|S|SnT|d(krš|jd)ƒr“|j|t|ƒdƒdkr�|d S|S|Sn|Sd*S(+u:step4() takes off -ant, -ence etc., in context vcvc.iiþÿÿÿuaualiucuanceiiüÿÿÿuenceueueruiuiculuableuibleunuantiiýÿÿÿuementiiûÿÿÿumentuentuousionutionuouusuismutuateuitiuuuousuvuiveuzuizeN(RRR(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step4Ös„  . .. . . .. .... .. . .. . . .cCs³|ddkrq|j|t|ƒdƒ}|dksa|dkrq|j|t|ƒdƒ rq|d }qqn|jdƒr¯|j|t|ƒdƒdkr¯|d }n|S(uYstep5() removes a final -e if m() > 1, and changes -ll to -l if m() > 1. iÿÿÿÿueiiull(RRRR(RRta((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt_step5+s51 icCsß|dkr!|dkr!|}n0|dkr@t|ƒd}n|||d!}||jkrk|j|St|ƒdkr�|S|j|ƒ}|j|ƒ}|j|ƒ}|j|ƒ}|j|ƒ}|j|ƒ}|S(uX Returns the stem of p, or, if i and j are given, the stem of p[i:j+1]. iiiN( tNoneRRRRR R!R"R$(RtpRRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt stem_word8s    cCsf|jƒ}d}xMtt|ƒƒD]9}||||krP|||7}q%|||7}q%W|S(Nu(tlowerRR(RRRR(trettx((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt _adjust_caseVs cCs5|j|jƒdt|ƒdƒ}|j||ƒS(Nii(R'R(RR+(RRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyRws%cCsdS(Nu((R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyt__repr__}sN(t__name__t __module__t__doc__R RRRRRRRR R!R"R$R%R'R+RR,(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pyRbs" , '    C  \ # U  ! c Csnddlm}ddlm}|jƒ}g}g}x[|jƒd D]I}x@|j|ƒD]/\}}|j|ƒ|j|j|ƒƒq_WqIWdj|ƒ}t j dd|dƒj ƒ}dj|ƒ} t j dd| dƒj ƒ} t dj d ƒjdd ƒjd dƒƒt | ƒt d j d ƒjdd ƒjd dƒƒt |ƒt d d ƒd S(u^ A demonstration of the porter stemmer on a sample from the Penn Treebank corpus. iÿÿÿÿ(ttreebank(Riu u (.{,70})\su\1\nu -Original-iFu*u-u -Results-N(t nltk.corpusR0tnltkRRtfilest tagged_wordstappendtjointretsubtrstriptprinttcentertreplace( R0RtstemmertorigtstemmedtitemRttagtresultstoriginal((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pytdemo‘s$  + + ( R/t __future__RRt __docformat__R7t nltk.stem.apiRt nltk.compatRRRD(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/stem/porter.pytUs ÿÿ0