U en@s8dZdZddlZddlmZGdddeZddZdS) a Porter Stemmer This is the Porter stemming algorithm. It follows the algorithm presented in Porter, M. "An algorithm for suffix stripping." Program 14.3 (1980): 130-137. with some optional deviations that can be turned on or off with the `mode` argument to the constructor. Martin Porter, the algorithm's inventor, maintains a web page about the algorithm at https://www.tartarus.org/~martin/PorterStemmer/ which includes another Python implementation and other implementations in many languages. Z plaintextN)StemmerIc@seZdZdZdZdZdZefddZddZd d Z d d Z d dZ ddZ ddZ ddZddZddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd-d(d)Zd*d+Zd,S). PorterStemmeraY A word stemmer based on the Porter stemming algorithm. Porter, M. "An algorithm for suffix stripping." Program 14.3 (1980): 130-137. See https://www.tartarus.org/~martin/PorterStemmer/ for the homepage of the algorithm. Martin Porter has endorsed several modifications to the Porter algorithm since writing his original paper, and those extensions are included in the implementations on his website. Additionally, others have proposed further improvements to the algorithm, including NLTK contributors. There are thus three modes that can be selected by passing the appropriate constant to the class constructor's `mode` attribute: - PorterStemmer.ORIGINAL_ALGORITHM An implementation that is faithful to the original paper. Note that Martin Porter has deprecated this version of the algorithm. Martin distributes implementations of the Porter Stemmer in many languages, hosted at: https://www.tartarus.org/~martin/PorterStemmer/ and all of these implementations include his extensions. He strongly recommends against using the original, published version of the algorithm; only use this mode if you clearly understand why you are choosing to do so. - PorterStemmer.MARTIN_EXTENSIONS An implementation that only uses the modifications to the algorithm that are included in the implementations on Martin Porter's website. He has declared Porter frozen, so the behaviour of those implementations should never change. - PorterStemmer.NLTK_EXTENSIONS (default) An implementation that includes further improvements devised by NLTK contributors or taken from other modified implementations found on the web. For the best stemming, you should use the default NLTK_EXTENSIONS version. However, if you need to get the same results as either the original algorithm or one of Martin Porter's hosted versions for compatibility with an existing implementation or dataset, you can use one of the other modes instead. NLTK_EXTENSIONSMARTIN_EXTENSIONSORIGINAL_ALGORITHMc Cs||j|j|jfkrtd||_|j|jkrddgdgdgdgdgdd gd d gd d gdgdgdgdgd }i|_|D]}||D]}||j|<qqvtdddddg|_dS)NzwMode must be one of PorterStemmer.NLTK_EXTENSIONS, PorterStemmer.MARTIN_EXTENSIONS, or PorterStemmer.ORIGINAL_ALGORITHMskyZskiesZdyingZlyingZtyingnewsZinningsinningZoutingsoutingZcanningscanninghoweproceedexceedsucceed) rZdieZlieZtierr r r r r rraeiou)rrr ValueErrormodepool frozensetvowels)selfrZirregular_formskeyvalrQ/var/www/html/assets/scripts/venv/lib/python3.8/site-packages/nltk/stem/porter.py__init__Vs8  zPorterStemmer.__init__cCs@|||jkrdS||dkr<|dkr*dS|||d SdS)aReturns True if word[i] is a consonant, False otherwise A consonant is defined in the paper as follows: A consonant in a word is a letter other than A, E, I, O or U, and other than Y preceded by a consonant. (The fact that the term `consonant' is defined to some extent in terms of itself does not make it ambiguous.) So in TOY the consonants are T and Y, and in SYZYGY they are S, Z and G. If a letter is not a consonant it is a vowel. FyrT)r _is_consonant)rwordrrrrr"~s  zPorterStemmer._is_consonantcCs>d}tt|D]"}|||r*|d7}q|d7}q|dS)aReturns the 'measure' of stem, per definition in the paper From the paper: A consonant will be denoted by c, a vowel by v. A list ccc... of length greater than 0 will be denoted by C, and a list vvv... of length greater than 0 will be denoted by V. Any word, or part of a word, therefore has one of the four forms: CVCV ... C CVCV ... V VCVC ... C VCVC ... V These may all be represented by the single form [C]VCVC ... [V] where the square brackets denote arbitrary presence of their contents. Using (VC){m} to denote VC repeated m times, this may again be written as [C](VC){m}[V]. m will be called the \measure\ of any word or word part when represented in this form. The case m = 0 covers the null word. Here are some examples: m=0 TR, EE, TREE, Y, BY. m=1 TROUBLE, OATS, TREES, IVY. m=2 TROUBLES, PRIVATE, OATEN, ORRERY. cvvc)rangelenr"count)rstemZ cv_sequencerrrr_measures "   zPorterStemmer._measurecCs||dkS)Nrr,)rr+rrr_has_positive_measuresz#PorterStemmer._has_positive_measurecCs(tt|D]}|||s dSq dS)z1Returns True if stem contains a vowel, else FalseTF)r(r)r")rr+rrrr_contains_vowels zPorterStemmer._contains_vowelcCs0t|dko.|d|dko.||t|dS)zjImplements condition *d from the paper Returns True if word ends with a double consonant r!r)r"rr#rrr_ends_double_consonants  z$PorterStemmer._ends_double_consonantcCst|dkrV||t|drV||t|d rV||t|drV|ddkp|j|jkot|dko||d o||dS)zImplements condition *o from the paper From the paper: *o - the stem ends cvc, where the second c is not W, X or Y (e.g. -WIL, -HOP). r0r!r1)wxr r)r)r"rrr4rrr _ends_cvcs      zPorterStemmer._ends_cvccCs<||std|dkr"||S|dt| |SdS)z-Replaces `suffix` of `word` with `replacementz(Given word doesn't end with given suffixr$N)endswithAssertionErrorr))rr#suffix replacementrrr_replace_suffixszPorterStemmer._replace_suffixcCs|D]}|\}}}|dkrT||rT|dd}|dks@||rL||S|S||r|||d}|dks|||r||S|Sq|S)aApplies the first applicable suffix-removal rule to the word Takes a word and a list of suffix-removal rules represented as 3-tuples, with the first element being the suffix to remove, the second element being the string to replace it with, and the final element being the condition for the rule to be applicable, or None if the rule is unconditional. *dNr2r$)r5r:r>)rr#rulesruler<r= conditionr+rrr_apply_rule_lists       zPorterStemmer._apply_rule_listcCsD|j|jkr0|dr0t|dkr0||ddS||ddddgS)aImplements Step 1a from "An algorithm for suffix stripping" From the paper: SSES -> SS caresses -> caress IES -> I ponies -> poni ties -> ti SS -> SS caress -> caress S -> cats -> cat iesie)ZssesssN)rDrN)rGrGN)sr$N)rrr:r)r>rCr4rrr_step1as zPorterStemmer._step1ac sjjkr>|dr>t|dkr0|ddS|ddS|drp|dd}|dkrl|dS|Sd }d D].}||rx||drxd }qqx|s|Sd d dddfddfddfddfgS)a=Implements Step 1b from "An algorithm for suffix stripping" From the paper: (m>0) EED -> EE feed -> feed agreed -> agree (*v*) ED -> plastered -> plaster bled -> bled (*v*) ING -> motoring -> motor sing -> sing If the second or third of the rules in Step 1b is successful, the following is done: AT -> ATE conflat(ed) -> conflate BL -> BLE troubl(ed) -> trouble IZ -> IZE siz(ed) -> size (*d and not (*L or *S or *Z)) -> single letter hopp(ing) -> hop tann(ed) -> tan fall(ing) -> fall hiss(ing) -> hiss fizz(ed) -> fizz (m=1 and *o) -> E fail(ing) -> fail fil(ing) -> file The rule to map to a single letter causes the removal of one of the double letter pair. The -E is put back on -AT, -BL and -IZ, so that the suffixes -ATE, -BLE and -IZE can be recognised later. This E may be removed in step 4. ZiedrErFrZeedr$reeF)ZedZingT)atateN)blbleN)ZizizeNr?r1cs ddkS)Nr1)lrHzrr+)intermediate_stemrrxz'PorterStemmer._step1b..rcs|dko|SNr!)r,r9rRrrrrT~rU)rrr:r)r>r,r/rC)rr#r+Zrule_2_or_3_succeededr<r)rSrr_step1b/sD#        zPorterStemmer._step1bcs<fdd}fdd}|ddjjkr2|n|fgS)zImplements Step 1c from "An algorithm for suffix stripping" From the paper: Step 1c (*v*) Y -> I happy -> happi sky -> sky cs t|dko|t|dS)a This has been modified from the original Porter algorithm so that y->i is only done when y is preceded by a consonant, but not if the stem is only a single consonant, i.e. (*c and not c) Y -> I So 'happy' -> 'happi', but 'enjoy' -> 'enjoy' etc This is a much better rule. Formerly 'enjoy'->'enjoi' and 'enjoyment'->'enjoy'. Step 1c is perhaps done too soon; but with this modification that no longer really matters. Also, the removal of the contains_vowel(z) condition means that 'spy', 'fly', 'try' ... stem to 'spi', 'fli', 'tri' and conflate with 'spied', 'tried', 'flies' ... r!r3rRrWrrnltk_conditionsz-PorterStemmer._step1c..nltk_conditioncs |S)N)r/rRrWrroriginal_conditionsz1PorterStemmer._step1c..original_conditionr r)rCrr)rr#rYrZrrWr_step1cs   zPorterStemmer._step1ccsjjkr>dr>ddr>ddSddjf}ddjf}dd jfd d jfd d jfddjfddjfjjkr|n|ddjfddjfddjfddjfddjfdd jfdd jfddjfddjfddjfd djfd!djfd"djfd#djfg}jjkrd|d$djf|d%d&fd'd(fjjkr|d%d&jf |S))aImplements Step 2 from "An algorithm for suffix stripping" From the paper: Step 2 (m>0) ATIONAL -> ATE relational -> relate (m>0) TIONAL -> TION conditional -> condition rational -> rational (m>0) ENCI -> ENCE valenci -> valence (m>0) ANCI -> ANCE hesitanci -> hesitance (m>0) IZER -> IZE digitizer -> digitize (m>0) ABLI -> ABLE conformabli -> conformable (m>0) ALLI -> AL radicalli -> radical (m>0) ENTLI -> ENT differentli -> different (m>0) ELI -> E vileli - > vile (m>0) OUSLI -> OUS analogousli -> analogous (m>0) IZATION -> IZE vietnamization -> vietnamize (m>0) ATION -> ATE predication -> predicate (m>0) ATOR -> ATE operator -> operate (m>0) ALISM -> AL feudalism -> feudal (m>0) IVENESS -> IVE decisiveness -> decisive (m>0) FULNESS -> FUL hopefulness -> hopeful (m>0) OUSNESS -> OUS callousness -> callous (m>0) ALITI -> AL formaliti -> formal (m>0) IVITI -> IVE sensitiviti -> sensitive (m>0) BILITI -> BLE sensibiliti -> sensible Zallir$alZblirNZabliableZationalrLZtionalZtionZencienceZancianceZizerrOZentlientZelirZousliousZizationZationZatorZalismZivenessiveZfulnessfulZousnessZalitiZivitiZbilitiZfulliZlogilogcsddS)N)r.rRr4rrrTrUz&PorterStemmer._step2..) rrr:r.r>_step2rappendrrC)rr#Zbli_ruleZ abli_ruler@rr4rrfsH                       zPorterStemmer._step2c CsR||dd|jfdd|jfdd|jfdd|jfdd|jfd d|jfd d|jfgS) aVImplements Step 3 from "An algorithm for suffix stripping" From the paper: Step 3 (m>0) ICATE -> IC triplicate -> triplic (m>0) ATIVE -> formative -> form (m>0) ALIZE -> AL formalize -> formal (m>0) ICITI -> IC electriciti -> electric (m>0) ICAL -> IC electrical -> electric (m>0) FUL -> hopeful -> hope (m>0) NESS -> goodness -> good ZicateicZativer$Zalizer\ZicitiZicalrcZness)rCr.r4rrr_step3s       zPorterStemmer._step3csfdd}|dd|fdd|fdd|fdd|fdd|fd d|fd d|fd d|fd d|fd d|fdd|fddfddfdd|fdd|fdd|fdd|fdd|fdd|fdd|fgS)aImplements Step 4 from "An algorithm for suffix stripping" Step 4 (m>1) AL -> revival -> reviv (m>1) ANCE -> allowance -> allow (m>1) ENCE -> inference -> infer (m>1) ER -> airliner -> airlin (m>1) IC -> gyroscopic -> gyroscop (m>1) ABLE -> adjustable -> adjust (m>1) IBLE -> defensible -> defens (m>1) ANT -> irritant -> irrit (m>1) EMENT -> replacement -> replac (m>1) MENT -> adjustment -> adjust (m>1) ENT -> dependent -> depend (m>1 and (*S or *T)) ION -> adoption -> adopt (m>1) OU -> homologou -> homolog (m>1) ISM -> communism -> commun (m>1) ATE -> activate -> activ (m>1) ITI -> angulariti -> angular (m>1) OUS -> homologous -> homolog (m>1) IVE -> effective -> effect (m>1) IZE -> bowdlerize -> bowdler The suffixes are now removed. All that remains is a little tidying up. cs|dkSrVr-rRrWrrrT;rUz&PorterStemmer._step4..r\r$r_r^Zerrhr]ZibleantZementZmentr`Zioncs|dko|ddkS)Nr!r1)rHtr-rRrWrrrTOrUZouZismrLZitirarbrOrC)rr#Z measure_gt_1rrWr_step4s6  zPorterStemmer._step4cCsJ|drF||dd}||dkr*|S||dkrF||sF|S|S)a=Implements Step 5a from "An algorithm for suffix stripping" From the paper: Step 5a (m>1) E -> probate -> probat rate -> rate (m=1 and not *o) E -> cease -> ceas rr$r!)r:r>r,r9)rr#r+rrr_step5a[s zPorterStemmer._step5acsddfddfgS)aImplements Step 5a from "An algorithm for suffix stripping" From the paper: Step 5b (m > 1 and *d and *L) -> single letter controll -> control roll -> roll llrPcsdddkS)Nr1r!r-rRr4rrrTrUz'PorterStemmer._step5b..rlr4rr4r_step5bs zPorterStemmer._step5bTcCs|r |n|}|j|jkr0||jkr0|j|S|j|jkrLt|dkrL|S||}||}||}| |}| |}| |}| |}| |}|S)zW :param to_lowercase: if `to_lowercase=True` the word always lowercase r0)lowerrrrrr)rIrXr[rfrirmrnrp)rr#Z to_lowercaser+rrrr+s         zPorterStemmer.stemcCsdS)NzrrWrrr__repr__szPorterStemmer.__repr__N)T)__name__ __module__ __qualname____doc__rrrrr"r,r.r/r5r9r>rCrIrXr[rfrirmrnrpr+rrrrrrrs.5 (3 T0P<$ rc Csddlm}ddlm}|}g}g}|ddD]2}||D]"\}}|||||qFq8d|}t dd|d }d|} t dd| d } t d d dd d dt | t d d dd d dt |t ddS)z^ A demonstration of the porter stemmer on a sample from the Penn Treebank corpus. rrR)treebankNr6 z (.{,70})\sz\1\nz -Original-F*-z -Results-zF**********************************************************************)Znltkr+Z nltk.corpusrwrZfileidsZ tagged_wordsrgjoinresubrstripprintcenterreplace) r+rwZstemmerorigZstemmeditemr#tagresultsoriginalrrrdemos$     r)rv __docformat__r}Z nltk.stem.apirrrrrrrs