ó <¿CVc@sødZddlmZmZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZed ƒZejd efd „ƒYƒZd efd„ƒYZejdefd„ƒYƒZejdefd„ƒYƒZejdefd„ƒYƒZejdefd„ƒYƒZejdefd„ƒYƒZejdefd„ƒYƒZejdefd„ƒYƒZ ejdefd„ƒYƒZ!ejdefd „ƒYƒZ"ejd!efd"„ƒYƒZ#ejd#efd$„ƒYƒZ$d%efd&„ƒYZ%ejd'efd(„ƒYƒZ&d)„Z'd*„Z(ejd+e fd,„ƒYƒZ)ejd-e*fd.„ƒYƒZ+d/e+fd0„ƒYZ,d1e+fd2„ƒYZ-ej.d3d4ƒZ/d5„Z0d6„Z1d7efd8„ƒYZ2d9e2fd:„ƒYZ3d;„Z4d<„Z5d=„Z6d>d?d@„Z7dA„Z8e9dBkr¦e7d>dCƒe7dDdEƒe8ƒndFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]gZ:dS(^uý Classes for representing and processing probabilistic information. The ``FreqDist`` class is used to encode "frequency distributions", which count the number of times that each outcome of an experiment occurs. The ``ProbDistI`` class defines a standard interface for "probability distributions", which encode the probability of each outcome for an experiment. There are two types of probability distribution: - "derived probability distributions" are created from frequency distributions. They attempt to model the probability distribution that generated the frequency distribution. - "analytic probability distributions" are created directly from parameters (such as variance). The ``ConditionalFreqDist`` class and ``ConditionalProbDistI`` interface are used to encode conditional distributions. Conditional probability distributions can be derived or analytic; but currently the only implementation of the ``ConditionalProbDistI`` interface is ``ConditionalProbDist``, a derived distribution. iÿÿÿÿ(tprint_functiontunicode_literalsN(t itemgetter(t defaultdict(treduce(tcompat(tCounter(traise_unorderable_typesu-1e300tFreqDistcBsÔeZdZdd„Zd„Zd„Zd„Zdd„Zdd„Z d„Z d„Z d „Z d „Z d „Zd „Zd „Zd„Zd„Zd„Zd„Zddd„Zdd„Zd„ZRS(u‘ A frequency distribution for the outcomes of an experiment. A frequency distribution records the number of times each outcome of an experiment has occurred. For example, a frequency distribution could be used to record the frequency of each word type in a document. Formally, a frequency distribution can be defined as a function mapping from each sample to the number of times that sample occurred as an outcome. Frequency distributions are generally constructed by running a number of experiments, and incrementing the count for a sample every time it is an outcome of an experiment. For example, the following code will produce a frequency distribution that encodes how often each word occurs in a text: >>> from nltk.tokenize import word_tokenize >>> from nltk.probability import FreqDist >>> sent = 'This is an example sentence' >>> fdist = FreqDist() >>> for word in word_tokenize(sent): ... fdist[word.lower()] += 1 An equivalent way to do this is with the initializer: >>> fdist = FreqDist(word.lower() for word in word_tokenize(sent)) cCstj||ƒdS(ub Construct a new frequency distribution. If ``samples`` is given, then the frequency distribution will be initialized with the count of each object in ``samples``; otherwise, it will be initialized to be empty. In particular, ``FreqDist()`` returns an empty frequency distribution; and ``FreqDist(samples)`` first creates an empty frequency distribution, and then calls ``update`` with the list ``samples``. :param samples: The samples to initialize the frequency distribution with. :type samples: Sequence N(Rt__init__(tselftsamples((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ZscCst|jƒƒS(uú Return the total number of sample outcomes that have been recorded by this FreqDist. For the number of unique sample values (or bins) with counts greater than zero, use ``FreqDist.B()``. :rtype: int (tsumtvalues(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytNls cCs t|ƒS(u  Return the total number of sample values (or "bins") that have counts greater than zero. For the total number of sample outcomes recorded, use ``FreqDist.N()``. (FreqDist.B() is the same as len(FreqDist).) :rtype: int (tlen(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytBws cCs'g|D]}||dkr|^qS(ue Return a list of all samples that occur once (hapax legomena) :rtype: list i((R titem((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pythapaxes‚scCs|j|ƒ|S(N(tr_Nr(R trtbins((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytNr‹scCs]ttƒ}x$|jƒD]}||cd7 0. :type bins: int :param bins: The number of possible sample outcomes. ``bins`` is used to calculate Nr(0). In particular, Nr(0) is ``bins-self.B()``. If ``bins`` is not specified, it defaults to ``self.B()`` (so Nr(0) will be 0). :rtype: int iiN(RtintR tNoneR(R Rt_r_Nrtcount((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRŽs &ccs.d}x!|D]}|||7}|Vq WdS(u0 Return the cumulative frequencies of the specified samples. If no samples are specified, all counts are returned, starting with the largest. :param samples: the samples whose frequencies should be returned. :type samples: any :rtype: list(float) gN((R R tcftsample((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt_cumulative_frequencies£s  cCs.|jƒdkrdSt||ƒ|jƒS(u0 Return the frequency of a given sample. The frequency of a sample is defined as the count of that sample divided by the total number of sample outcomes that have been recorded by this FreqDist. The count of a sample is defined as the number of times that sample outcome was recorded by this FreqDist. Frequencies are always real numbers in the range [0, 1]. :param sample: the sample whose frequency should be returned. :type sample: any :rtype: float i(Rtfloat(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytfreq´scCs6t|ƒdkr!tdƒ‚n|jdƒddS(uÕ Return the sample with the greatest number of outcomes in this frequency distribution. If two or more samples have the same number of outcomes, return one of them; which sample is returned is undefined. If no outcomes have occurred in this frequency distribution, return None. :return: The sample with the maximum number of outcomes in this frequency distribution. :rtype: any or None iu?A FreqDist must have at least one sample before max is defined.i(Rt ValueErrort most_common(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytmaxÇs c Os¢yddlm}Wntk r3tdƒ‚nXt|ƒdkrXt|ƒg}ng|j|ŒD]\}}|^qh}t|dtƒ}|r¶t|j |ƒƒ}d} n#g|D]} || ^q½}d} |j t dd ƒd |krd |d   c Ost|ƒdkr$t|ƒg}ng|j|ŒD]\}}|^q4}t|dtƒ}|r|t|j|ƒƒ}ng|D]}||^qƒ}x2tt|ƒƒD]} td|| ddƒq¬Wtƒx2tt|ƒƒD]} td|| ddƒqèWtƒdS(uU Tabulate the given samples from the frequency distribution (cumulative), displaying the most frequent sample first. If an integer parameter is supplied, stop after this many samples have been plotted. :param samples: The samples to plot (default is all samples) :type samples: list iu cumulativeu%4stendu u%4dN(RR!R(R)R*RR0tprint( R R5R6RR7R R8R9Rti((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyttabulates (cCs |j|ƒS(uY Create a copy of this frequency distribution. :rtype: FreqDist (t __class__(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytcopyscsTtˆtƒs"tdˆˆƒntˆƒjˆƒoSt‡‡fd†ˆDƒƒS(Nu<=c3s#|]}ˆ|ˆ|kVqdS(N((t.0tkey(totherR (sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys *s(t isinstanceRRtsettissubsettall(R RC((RCR sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt__le__'scCs||k p||kS(N((R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt-scCs||ko||k S(N((R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRI.scCs ||k S(N((R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRI/scCs |jƒS(uZ Return a string representation of this FreqDist. :rtype: string (tpformat(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt__repr__1si cCs t|jd|ƒd|ƒdS(uä Print a string representation of this FreqDist to 'stream' :param maxlen: The maximum number of items to print :type maxlen: int :param stream: The stream to print to. stdout by default tmaxlentfileN(R<RJ(R RLtstream((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytpprint9scCscg|j|ƒD]}dj|Œ^q}t|ƒ|krM|jdƒndjdj|ƒƒS(u² Return a string representation of this FreqDist. :param maxlen: The maximum number of items to display :type maxlen: int :rtype: string u {0!r}: {1!r}u...uFreqDist({{{0}}})u, (R!tformatRtappendtjoin(R RLRtitems((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRJCs+cCsdt|ƒ|jƒfS(uZ Return a string representation of this FreqDist. :rtype: string u*(RR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt__str__PsN(t__name__t __module__t__doc__RR RRRRRRRR"R.R>R@RHt__ge__t__lt__t__gt__RKRORJRT(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR<s*       ,        t ProbDistIcBsSeZdZeZd„Zd„Zd„Zd„Zd„Z d„Z d„Z RS(u A probability distribution for the outcomes of an experiment. A probability distribution specifies how likely it is that an experiment will have any given outcome. For example, a probability distribution could be used to predict the probability that a token in a document will have a given type. Formally, a probability distribution can be defined as a function mapping from samples to nonnegative real numbers, such that the sum of every number in the function's range is 1.0. A ``ProbDist`` is often used to model the probability distribution of the experiment used to generate a frequency distribution. cCs"|jtkrtdƒ‚ndS(Nu Interfaces can't be instantiated(R?R[tNotImplementedError(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR nscCs tƒ‚dS(u Return the probability for a given sample. Probabilities are always real numbers in the range [0, 1]. :param sample: The sample whose probability should be returned. :type sample: any :rtype: float N(R\(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytprobrs cCs/|j|ƒ}|dkr+tj|dƒStS(uÜ Return the base 2 logarithm of the probability for a given sample. :param sample: The sample whose probability should be returned. :type sample: any :rtype: float ii(R]tmathtlogt_NINF(R Rtp((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytlogprob~s cCs tƒ‚dS(uÕ Return the sample with the greatest probability. If two or more samples have the same probability, return one of them; which sample is returned is undefined. :rtype: any N(R\(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"‹scCs tƒ‚dS(u¢ Return a list of all samples that have nonzero probabilities. Use ``prob`` to find the probability of each sample. :rtype: list N(R\(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR •scCsdS(ui Return the ratio by which counts are discounted on average: c*/c :rtype: float g((R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytdiscountŸscCsœtjƒ}|}x7|jƒD])}||j|ƒ8}|dkr|SqW|dkr\|S|jrƒtjd|||fƒntjt|jƒƒƒS(uº Return a randomly selected sample from this probability distribution. The probability of returning each sample ``samp`` is equal to ``self.prob(samp)``. ig-Cëâ6?uTProbability distribution %r sums to %r; generate() is returning an arbitrary sample.(trandomR R]t SUM_TO_ONEtwarningstwarntchoiceR*(R Ratp_initR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytgenerate©s     ( RURVRWR,ReR R]RbR"R RcRj(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR[]s   tUniformProbDistcBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(u— A probability distribution that assigns equal probability to each sample in a given set; and a zero probability to all other samples. cCs`t|ƒdkr%tddƒ‚nt|ƒ|_dt|jƒ|_t|jƒ|_dS(u4 Construct a new uniform probability distribution, that assigns equal probability to each sample in ``samples``. :param samples: The samples that should be given uniform probability. :type samples: list :raise ValueError: If ``samples`` is empty. iu(A Uniform probability distribution must uhave at least one sample.gð?N(RR REt _samplesett_probR*t_samples(R R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR Ås  cCs||jkr|jSdS(Ni(RlRm(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]ÖscCs |jdS(Ni(Rn(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"ÙscCs|jS(N(Rn(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ÜscCsdt|jƒS(Nu!(RRl(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKßs(RURVRWR R]R"R RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRk¾s     tRandomProbDistcBsAeZdZd„Zed„ƒZd„Zd„Zd„ZRS(uÎ Generates a random probability distribution whereby each sample will be between 0 and 1 with equal probability (uniform random distribution. Also called a continuous uniform distribution). cCsSt|ƒdkr%tddƒ‚n|j|ƒ|_t|jjƒƒ|_dS(Niu A probability distribution must uhave at least one sample.(RR tunirandt_probsR*tkeysRn(R R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ês  cs±gtt|ƒƒD]}tjƒ^q‰tˆƒ}x(tˆƒD]\}}||ˆ|s(R0RRdR t enumeratetdict(tclsR R=ttotaltx((Rssb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRpñs+   cCs|jj|dƒS(Ni(Rqtget(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]scCs|jS(N(Rn(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR scCsdt|jƒS(Nu'(RRq(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK s( RURVRWR t classmethodRpR]R RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRoãs    tDictionaryProbDistcBsMeZdZdeed„Zd„Zd„Zd„Zd„Z d„Z RS(u£ A probability distribution whose probabilities are directly specified by a given dictionary. The given dictionary maps samples to probabilities. c Cs‹|dk r|jƒni|_||_|r‡t|ƒdkrUtddƒ‚n|rõtt|jjƒƒƒ}|t kr¿t j dt|ƒdƒ}xQ|D]}||j|Osi(thasattrR"R|RSt_max(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"Ms)cCs |jjƒS(N(R|Rr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR QscCsdt|jƒS(Nu(RR|(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKSsN( RURVRWRR)R R]RbR"R RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR{s(    t MLEProbDistcBsGeZdZdd„Zd„Zd„Zd„Zd„Zd„Z RS(u% The maximum likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The "maximum likelihood estimate" approximates the probability of each sample as the frequency of that sample in the frequency distribution. cCs ||_dS(u) Use the maximum likelihood estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. N(t _freqdist(R tfreqdistR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR `s cCs|jS(u… Return the frequency distribution that this probability distribution is based on. :rtype: FreqDist (Rˆ(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR‰kscCs|jj|ƒS(N(RˆR(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]tscCs |jjƒS(N(RˆR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"wscCs |jjƒS(N(RˆRr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR zscCsd|jjƒS(u\ :rtype: str :return: A string representation of this ``ProbDist``. u!(RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK}sN( RURVRWRR R‰R]R"R RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR‡Ws   tLidstoneProbDistcBsVeZdZeZdd„Zd„Zd„Zd„Z d„Z d„Z d„Z RS( u8 The Lidstone estimate for the probability distribution of the experiment used to generate a frequency distribution. The "Lidstone estimate" is parameterized by a real number *gamma*, which typically ranges from 0 to 1. The Lidstone estimate approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as ``c+gamma)/(N+B*gamma)``. This is equivalent to adding *gamma* to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCs6|dks*|d krQ|jƒdkrQ|jjd }td|dƒ‚n|d k r¬||jƒkr¬|jjd }td|d|dd|jƒƒ‚n||_t|ƒ|_|jjƒ|_ |d krñ|jƒ}n||_ |j |||_ |j d kr2d|_d |_ nd S( uÇ Use the Lidstone estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type gamma: float :param gamma: A real number used to parameterize the estimate. The Lidstone estimate is equivalent to adding *gamma* to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. iiøÿÿÿuA %s probability distribution umust have at least one bin.u) The number of bins in a %s distribution u&(%d) must be greater than or equal to u(the number of bins in the FreqDist used uto create it (%d).giN( RRR?RUR RRˆRt_gammat_Nt_binst_divisor(R R‰tgammaRtname((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR “s&*      cCs|jS(u… Return the frequency distribution that this probability distribution is based on. :rtype: FreqDist (Rˆ(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR‰ÂscCs|j|}||j|jS(N(RˆR‹RŽ(R Rtc((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]Ës cCs |jjƒS(N(RˆR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"ÏscCs |jjƒS(N(RˆRr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ÕscCs|j|j}||j|S(N(R‹RRŒ(R tgb((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRcØscCsd|jjƒS(u[ Return a string representation of this ``ProbDist``. :rtype: str u&(RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKÜsN( RURVRWR)ReRR R‰R]R"R RcRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRŠ…s  /    tLaplaceProbDistcBs#eZdZdd„Zd„ZRS(u¹ The Laplace estimate for the probability distribution of the experiment used to generate a frequency distribution. The "Laplace estimate" approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as *(c+1)/(N+B)*. This is equivalent to adding one to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCstj||d|ƒdS(uŽ Use the Laplace estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. iN(RŠR (R R‰R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ðscCsd|jjƒS(u\ :rtype: str :return: A string representation of this ``ProbDist``. u%(RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKsN(RURVRWRR RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR“ås  t ELEProbDistcBs#eZdZdd„Zd„ZRS(uÕ The expected likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The "expected likelihood estimate" approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as *(c+0.5)/(N+B/2)*. This is equivalent to adding 0.5 to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCstj||d|ƒdS(uš Use the expected likelihood estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. gà?N(RŠR (R R‰R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR scCsd|jjƒS(u[ Return a string representation of this ``ProbDist``. :rtype: str u!(RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK%sN(RURVRWRR RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR” s  tHeldoutProbDistcBsqeZdZeZd d„Zd„Zd„Zd„Z d„Z d„Z d„Z d„Z d „Zd „ZRS( ug The heldout estimate for the probability distribution of the experiment used to generate two frequency distributions. These two frequency distributions are called the "heldout frequency distribution" and the "base frequency distribution." The "heldout estimate" uses uses the "heldout frequency distribution" to predict the probability of each sample, given its frequency in the "base frequency distribution". In particular, the heldout estimate approximates the probability for a sample that occurs *r* times in the base distribution as the average frequency in the heldout distribution of all samples that occur *r* times in the base distribution. This average frequency is *Tr[r]/(Nr[r].N)*, where: - *Tr[r]* is the total count in the heldout distribution for all samples that occur *r* times in the base distribution. - *Nr[r]* is the number of samples that occur *r* times in the base distribution. - *N* is the number of outcomes recorded by the heldout frequency distribution. In order to increase the efficiency of the ``prob`` member function, *Tr[r]/(Nr[r].N)* is precomputed for each value of *r* when the ``HeldoutProbDist`` is created. :type _estimate: list(float) :ivar _estimate: A list mapping from *r*, the number of times that a sample occurs in the base distribution, to the probability estimate for that sample. ``_estimate[r]`` is calculated by finding the average frequency in the heldout distribution of all samples that occur *r* times in the base distribution. In particular, ``_estimate[r]`` = *Tr[r]/(Nr[r].N)*. :type _max_r: int :ivar _max_r: The maximum number of times that any sample occurs in the base distribution. ``_max_r`` is used to decide how large ``_estimate`` must be. c Cs’||_||_||jƒ|_|jƒ}|j|ƒ}gt|jdƒD]}||^qT}|jƒ}|j|||ƒ|_ dS(uá Use the heldout estimate to create a probability distribution for the experiment used to generate ``base_fdist`` and ``heldout_fdist``. :type base_fdist: FreqDist :param base_fdist: The base frequency distribution. :type heldout_fdist: FreqDist :param heldout_fdist: The heldout frequency distribution. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. iN( t _base_fdistt_heldout_fdistR"t_max_rt _calculate_TrRR0Rt_calculate_estimatet _estimate( R t base_fdistt heldout_fdistRtTrRRRR((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR Ys   * cCsPdg|jd}x5|jD]*}|j|}||c|j|7(R–RR—(R R:((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKÆsN(RURVRWR)ReRR R™RšRœRR R]R"RcRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR•.s) "      tCrossValidationProbDistcBsJeZdZeZd„Zd„Zd„Zd„Zd„Z d„Z RS(uA The cross-validation estimate for the probability distribution of the experiment used to generate a set of frequency distribution. The "cross-validation estimate" for the probability of a sample is found by averaging the held-out estimates for the sample in each pair of frequency distributions. cCsi||_g|_xP|D]H}x?|D]7}||k r&t|||ƒ}|jj|ƒq&q&WqWdS(u Use the cross-validation estimate to create a probability distribution for the experiment used to generate ``freqdists``. :type freqdists: list(FreqDist) :param freqdists: A list of the frequency distributions generated by the experiment. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. N(t _freqdistst_heldout_probdistsR•RQ(R t freqdistsRtfdist1tfdist2tprobdist((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR Ùs     cCs|jS(u€ Return the list of frequency distributions that this ``ProbDist`` is based on. :rtype: list(FreqDist) (R¡(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR£ôscCs/ttg|jD]}t|ƒ^qgƒƒS(N(RER R¡R*(R tfd((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR üscCs>d}x$|jD]}||j|ƒ7}qW|t|jƒS(Ng(R¢R]R(R RR]theldout_probdist((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]scCs tƒ‚dS(N(R\(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRcscCsdt|jƒS(u[ Return a string representation of this ``ProbDist``. :rtype: str u!(RR¡(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK s( RURVRWR)ReR R£R R]RcRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR Ïs     tWittenBellProbDistcBsPeZdZdd„Zd„Zd„Zd„Zd„Zd„Z d„Z RS( u› The Witten-Bell estimate of a probability distribution. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to *T / (N + T)* where *T* is the number of observed event types and *N* is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occurring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - *p = T / Z (N + T)*, if count = 0 - *p = c / (N + T)*, otherwise cCsß|dks4||jƒks4td|jƒƒ‚|dkrO|jƒ}n||_|jjƒ|_||jjƒ|_|jjƒ|_|jdkr´d|j|_n'|jt |j|j|jƒ|_dS(u) Creates a distribution of Witten-Bell probability estimates. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to *T / (N + T)* where *T* is the number of observed event types and *N* is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occurring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - *p = T / Z (N + T)*, if count = 0 - *p = c / (N + T)*, otherwise The parameters *T* and *N* are taken from the ``freqdist`` parameter (the ``B()`` and ``N()`` values). The normalizing factor *Z* is calculated using these values along with the ``bins`` parameter. :param freqdist: The frequency counts upon which to base the estimation. :type freqdist: FreqDist :param bins: The number of possible event types. This must be at least as large as the number of bins in the ``freqdist``. If None, then it's assumed to be equal to that of the ``freqdist`` :type bins: int u4bins parameter must not be less than %d=freqdist.B()igð?N( RRtAssertionErrorRˆt_Tt_ZRRŒt_P0R(R R‰R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR $s!  cCs8|j|}|dkr1|t|j|jƒS|jS(Ni(RˆRRŒR«R­(R RR‘((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]Ns cCs |jjƒS(N(RˆR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"SscCs |jjƒS(N(RˆRr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR VscCs|jS(N(Rˆ(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR‰YscCs tƒ‚dS(N(R\(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRc\scCsd|jjƒS(u[ Return a string representation of this ``ProbDist``. :rtype: str u((RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK_sN( RURVRWRR R]R"R R‰RcRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR©s *     tSimpleGoodTuringProbDistcBs§eZdZeZdd„Zd„Zd„Zd„Z d„Z d„Z d„Z d„Z d „Zd „Zd „Zd „Zd „Zd„Zd„Zd„ZRS(u< SimpleGoodTuring ProbDist approximates from frequency to frequency of frequency into a linear line under log space by linear regression. Details of Simple Good-Turing algorithm can be found in: - Good Turing smoothing without tears" (Gale & Sampson 1995), Journal of Quantitative Linguistics, vol. 2 pp. 217-237. - "Speech and Language Processing (Jurafsky & Martin), 2nd Edition, Chapter 4.5 p103 (log(Nc) = a + b*log(c)) - http://www.grsampson.net/RGoodTur.html Given a set of pair (xi, yi), where the xi denotes the frequency and yi denotes the frequency of frequency, we want to minimize their square variation. E(x) and E(y) represent the mean of xi and yi. - slope: b = sigma ((xi-E(x)(yi-E(y))) / sigma ((xi-E(x))(xi-E(x))) - intercept: a = E(y) - b.E(x) cCs¯|dks8||jƒks8td|jƒdƒ‚|dkrW|jƒd}n||_||_|jƒ\}}|j||ƒ|j||ƒ|j||ƒdS(up :param freqdist: The frequency counts upon which to base the estimation. :type freqdist: FreqDist :param bins: The number of possible event types. This must be larger than the number of bins in the ``freqdist``. If None, then it's assumed to be equal to ``freqdist``.B() + 1 :type bins: int u6bins parameter must not be less than %d=freqdist.B()+1iN( RRRªRˆRRt find_best_fitt_switcht _renormalize(R R‰RRtnr((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ·s !   cCs|jjƒ}|d=|S(Ni(RˆR(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt_r_Nr_non_zeroÌscCs2|jƒ}|sggfStt|jƒƒŒS(uW Split the frequency distribution in two list (r, Nr), where Nr(r) > 0 (R³tziptsortedRS(R tnonzero((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRÑs  cCsÎ| s| rdSg}x“tt|ƒƒD]}|dkrK||dnd}|t|ƒdkryd|||n ||d}d||||}|j|ƒq+Wg|D]}tj|ƒ^qµ}g|D]}tj|ƒ^q×} d} } dt|ƒt|ƒ} dt| ƒt| ƒ} xEt|| ƒD]4\}}| || || 7} | || d7} q@W| dkrŽ| | nd|_|jdkr¶tj d ƒn| |j| |_ dS( uÚ Use simple linear regression to tune parameters self._slope and self._intercept in the log-log space based on count and Nr(count) (Work in log space to avoid floating point underflow.) Niiig@ggð?iÿÿÿÿu—SimpleGoodTuring did not find a proper best fit line for smoothing probabilities of occurrences. The probability estimates are likely to be unreliable.( R0RRQR^R_R R´t_slopeRfRgt _intercept(R RR²tzrtjR=tktzr_tlog_rtlog_zrtxy_covtx_vartx_meanty_meanRxty((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR¯Ûs(  6"" c Csúxót|ƒD]å\}}t|ƒ|dksG||d|dkrT||_Pn|j}|d||dƒ||ƒ}d|d||d||}tj|j|||||dƒƒ}t||ƒd|kr ||_Pq q WdS(ul Calculate the r frontier where we must switch from Nr to Sr when estimating E[Nr]. igð?g\Âõ(\ÿ?N(RtRt _switch_att smoothedNrR^tsqrtt _variancetabs( R RR²R=tr_tSrt smooth_r_startunsmooth_r_startstd((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR°s.  ""* cCsHt|ƒ}t|ƒ}t|ƒ}|dd||dd||S(Ngð?i(R(R RR²tnr_1((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRÇs   cCsdd}x4t||ƒD]#\}}|||j|ƒ7}qW|r`d|jdƒ||_ndS(uy It is necessary to renormalize all the probability estimates to ensure a proper probability distribution results. This can be done by keeping the estimate of the probability mass for unseen items as N(1)/N and renormalizing all the estimates for previously seen items (as Gale and Sampson (1995) propose). (See M&S P.213, 1999) giiN(R´t _prob_measuret _renormal(R RR²tprob_covRÉtnr_((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR±s cCs$tj|j|jtj|ƒƒS(u“ Return the number of samples with count r. :param r: The amount of frequency. :type r: int :rtype: float (R^texpR¸R·R_(R R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRÅ*scCs{|j|}|j|ƒ}|dkrj|j|jjƒkrId}qw|d|j|jjƒ}n ||j}|S(uŽ Return the sample's probability. :param sample: sample of the event :type sample: str :rtype: float iggð?(RˆRÏRRRÐ(R RRRa((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]:s   ! cCsî|dkr%|jjƒdkr%dS|dkrg|jjƒdkrgd|jjdƒ|jjƒS|j|kr©d|jj|dƒ}d|jj|ƒ}n"|j|dƒ}|j|ƒ}|d||}||jjƒS(Nigð?i(RˆRRRÄRÅ(R RtEr_1tErtr_star((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRÏMs!!!cCs_d}xEtdt|jƒƒD]+}||j||j|ƒ|j7}qWtd|ƒdS(NgiuProbability Sum:(R0Rt_NrRÏRÐR<(R tprob_sumR=((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytcheck]s)cCsd|jdƒ|jjƒS(u„ This function returns the total mass of probability transfers from the seen samples to the unseen samples. gð?i(RÅRˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRcdscCs |jjƒS(N(RˆR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"kscCs |jjƒS(N(RˆRr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR nscCs|jS(N(Rˆ(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR‰qscCsd|jjƒS(u[ Return a string representation of this ``ProbDist``. :rtype: str u.(RˆR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKtsN(RURVRWR)ReRR R³RR¯R°RÇR±RÅR]RÏRÙRcR"R R‰RK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR®¢s$   '           tMutableProbDistcBsAeZdZed„Zd„Zd„Zd„Zed„ZRS(uÕ An mutable probdist where the probabilities may be easily modified. This simply copies an existing probdist, storing the probability values in a mutable dictionary and providing an update method. csÈ|_t‡fd†ttˆƒƒDƒƒ|_tjtdƒdgƒtˆƒ|_xWttˆƒƒD]C}|r˜|jˆ|ƒ|j|”sudgN( RnRuR0Rt _sample_dicttarraytstrt_dataRbR]t_logs(R t prob_distR t store_logsR=((R sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR …s +(cCs|jS(N(Rn(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR scCsE|jj|ƒ}|dkr"dS|jr:d|j|S|j|S(Ngi(RÛRyRRßRÞ(R RR=((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]¡s cCsS|jj|ƒ}|dkr(tdƒS|jr<|j|Stj|j|dƒS(Nu-infi(RÛRyRRRßRÞR^R_(R RR=((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRb¨s  cCsv|jj|ƒ}|dk s$t‚|jrU|r9|ntj|dƒ|j|t|ƒdkr!tdƒ‚nt|ƒ}|\}}}||jkrV|j|S||jkr|j||jƒ|j||f}n™||f|jkr#||f|jkr#|j||f}|j||f}||jƒ|j||f}||j ||} || }nd}||j|<|SdS(Niu$Expected an iterable with 3 members.g( RR ttupleRåRçRcRæRêRèRé( R ttrigramRëRìRíR]taftrtbfrt leftover_probtbeta((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]s&   !*   cCs|jS(uq Return the value by which counts are discounted. By default set to 0.75. :rtype: float (Rä(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRc9scCs ||_dS(uç Set the value by which counts are discounted to the value of discount. :param discount: the new value to discount counts by :type discount: float (preferred, but int possible) :rtype: None N(Rä(R Rc((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt set_discountAscCs |jjƒS(N(RçRr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR KscCs |jjƒS(N(RçR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR"NscCsdj|jjƒƒS(uV Return a string representation of this ProbDist :rtype: str u(bs(RDR[R R (RöRõ((RõRösb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytlog_likelihood]s cs1‡fd†ˆjƒDƒ}td„|Dƒƒ S(Nc3s|]}ˆj|ƒVqdS(N(R](RAR:(tpdist(sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys fscss%|]}|tj|dƒVqdS(iN(R^R_(RARa((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys gs(R R (Røtprobs((Røsb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytentropyestConditionalFreqDistcBsteZdZd d„Zd„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z d „Z d „ZRS( uº A collection of frequency distributions for a single experiment run under different conditions. Conditional frequency distributions are used to record the number of times each sample occurred, given the condition under which the experiment was run. For example, a conditional frequency distribution could be used to record the frequency of each word (type) in a document, given its length. Formally, a conditional frequency distribution can be defined as a function that maps from each condition to the FreqDist for the experiment under that condition. Conditional frequency distributions are typically constructed by repeatedly running an experiment under a variety of conditions, and incrementing the sample outcome counts for the appropriate conditions. For example, the following code will produce a conditional frequency distribution that encodes how often each word type occurs, given the length of that word type: >>> from nltk.probability import ConditionalFreqDist >>> from nltk.tokenize import word_tokenize >>> sent = "the the the dog dog some other words that we do not care about" >>> cfdist = ConditionalFreqDist() >>> for word in word_tokenize(sent): ... condition = len(word) ... cfdist[condition][word] += 1 An equivalent way to do this is with the initializer: >>> cfdist = ConditionalFreqDist((len(word), word) for word in word_tokenize(sent)) The frequency distribution for each condition is accessed using the indexing operator: >>> cfdist[3] FreqDist({'the': 3, 'dog': 2, 'not': 1}) >>> cfdist[3].freq('the') 0.5 >>> cfdist[3]['dog'] 2 When the indexing operator is used to access the frequency distribution for a condition that has not been accessed before, ``ConditionalFreqDist`` creates a new empty FreqDist for that condition. cCsHtj|tƒ|rDx+|D] \}}|||cd7­s((t conditionsR?R(R tkv_pairs((R sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt __reduce__¬scCst|jƒƒS(uT Return a list of the conditions that have been accessed for this ``ConditionalFreqDist``. Use the indexing operator to access the frequency distribution for a given condition. Note that the frequency distributions for some conditions may contain zero sample outcomes. :rtype: list (R*Rr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRþ°s cCstd„tj|ƒDƒƒS(u’ Return the total number of sample outcomes that have been recorded by this ``ConditionalFreqDist``. :rtype: int css|]}|jƒVqdS(N(R(RAtfdist((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys Ãs(R Rt itervalues(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR¼scséyddlm}Wntk r3tdƒ‚nXt|dtƒ}t|dtˆjƒƒƒ}t|ddƒ}t|dtt‡fd †|Dƒƒƒƒ}d |kr½d |d Üsu linewidthiuCumulative Countsu lower rightuCountsu upper rightu%sulabeltlocR$usilverR%iZuSamplesN(R&R#R'R R(R)RµRþRER*RR.tlegendR+R,R/R0RRR1R-R2R3R4(R R5R6R#R8RþR-R t conditionR9R3t legend_locRR:((R sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR.Ås:   %    !>  c sRt|dtƒ}t|dtˆjƒƒƒ}t|dtt‡fd†|Dƒƒƒƒ}td„|Dƒƒ}td|ddƒx"|D]}td|ddƒqWtƒx—|D]}td ||fddƒ|rýtˆ|j|ƒƒ} n!g|D]} ˆ|| ^q} x"| D]} td | ddƒq%Wtƒq»Wd S( uK Tabulate the given samples from the conditional frequency distribution. :param samples: The samples to plot :type samples: list :param title: The title for the graph :type title: str :param conditions: The conditions to plot (default is all) :type conditions: list u cumulativeu conditionsusamplesc3s&|]}ˆ|D] }|VqqdS(N((RAR‘R„(R (sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys scss|]}td|ƒVqdS(u%sN(R(RAR‘((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys su R;u%4su%*su%4dN( R(R)RµRþRER"R<R*R( R R5R6R8RþR tcondition_sizeR:R‘R9Rtf((R sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR>ös"  %  ! csftˆtƒs"tdˆˆƒntˆjƒƒjˆjƒƒoet‡‡fd†ˆjƒDƒƒS(Nu<=c3s#|]}ˆ|ˆ|kVqdS(N((RAR‘(RCR (sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys s(RDRûRRERþRFRG(R RC((RCR sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRHs!cCs8t|tƒs"td||ƒn||ko7||kS(Nu<(RDRûR(R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRYscCs,t|tƒs"td||ƒn||kS(Nu>=(RDRûR(R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRX!scCs,t|tƒs"td||ƒn||kS(Nu>(RDRûR(R RC((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRZ%scCsdt|ƒS(uf Return a string representation of this ``ConditionalFreqDist``. :rtype: str u((R(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRK*sN(RURVRWRR RRþRR.R>RHRYRXRZRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRûms/   1 "    tConditionalProbDistIcBs)eZdZd„Zd„Zd„ZRS(uo A collection of probability distributions for a single experiment run under different conditions. Conditional probability distributions are used to estimate the likelihood of each sample, given the condition under which the experiment was run. For example, a conditional probability distribution could be used to estimate the probability of each word type in a document, given the length of the word type. Formally, a conditional probability distribution can be defined as a function that maps from each condition to the ``ProbDist`` for the experiment under that condition. cCstdƒ‚dS(Nu Interfaces can't be instantiated(R\(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR AscCst|jƒƒS(uæ Return a list of the conditions that are represented by this ``ConditionalProbDist``. Use the indexing operator to access the probability distribution for a given condition. :rtype: list (R*Rr(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRþDscCsdt|ƒjt|ƒfS(uf Return a string representation of this ``ConditionalProbDist``. :rtype: str u<%s with %d conditions>(ttypeRUR(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRKNs(RURVRWR RþRK(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR 3s   tConditionalProbDistcBs eZdZd„Zd„ZRS(u² A conditional probability distribution modeling the experiments that were used to generate a conditional frequency distribution. A ConditionalProbDist is constructed from a ``ConditionalFreqDist`` and a ``ProbDist`` factory: - The ``ConditionalFreqDist`` specifies the frequency distribution for each condition. - The ``ProbDist`` factory is a function that takes a condition's frequency distribution, and returns its probability distribution. A ``ProbDist`` class's name (such as ``MLEProbDist`` or ``HeldoutProbDist``) can be used to specify that class's constructor. The first argument to the ``ProbDist`` factory is the frequency distribution that it should model; and the remaining arguments are specified by the ``factory_args`` parameter to the ``ConditionalProbDist`` constructor. For example, the following code constructs a ``ConditionalProbDist``, where the probability distribution for each condition is an ``ELEProbDist`` with 10 bins: >>> from nltk.corpus import brown >>> from nltk.probability import ConditionalFreqDist >>> from nltk.probability import ConditionalProbDist, ELEProbDist >>> cfdist = ConditionalFreqDist(brown.tagged_words()[:5000]) >>> cpdist = ConditionalProbDist(cfdist, ELEProbDist, 10) >>> cpdist['passed'].max() 'VBD' >>> cpdist['passed'].prob('VBD') 0.423... cOsJ||_||_||_x(|D] }|||||Ž|| probdist N(Râ(R t probdist_dict((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR ¥scCstƒ||<||S(N(R{(R RB((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR­s (RURVRWR R(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRŸs g ÂëþKH´9icCs_||tkr|S||tkr(|St||ƒ}|tjd||d||dƒS(u Given two numbers ``logx`` = *log(x)* and ``logy`` = *log(y)*, return *log(x+y)*. Conceptually, this is the same as returning ``log(2**(logx)+2**(logy))``, but the actual implementation avoids overflow errors that could result from direct computation. i(t_ADD_LOGS_MAX_DIFFtminR^R_(tlogxtlogytbase((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytadd_logs¸s cCs.t|ƒdkr*tt|d|dƒStS(Nii(RRRR`(tlogs((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR~ÆstProbabilisticMixIncBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(uÿ A mix-in class to associate probabilities with other classes (trees, rules, etc.). To use the ``ProbabilisticMixIn`` class, define a new class that derives from an existing class and from ProbabilisticMixIn. You will need to define a new constructor for the new class, which explicitly calls the constructors of both its parent classes. For example: >>> from nltk.probability import ProbabilisticMixIn >>> class A: ... def __init__(self, x, y): self.data = (x,y) ... >>> class ProbabilisticA(A, ProbabilisticMixIn): ... def __init__(self, x, y, **prob_kwarg): ... A.__init__(self, x, y) ... ProbabilisticMixIn.__init__(self, **prob_kwarg) See the documentation for the ProbabilisticMixIn ``constructor<__init__>`` for information about the arguments it expects. You should generally also redefine the string representation methods, the comparison methods, and the hashing method. cKsud|kr>d|kr'tdƒ‚qqtj||dƒn3d|kratj||dƒnd|_|_dS(u¡ Initialize this object's probability. This initializer should be called by subclass constructors. ``prob`` should generally be the first argument for those constructors. :param prob: The probability associated with the object. :type prob: float :param logprob: The log of the probability associated with the object. :type logprob: float uprobulogprobu.Must specify either prob or logprob (not both)N(t TypeErrorRtset_probt set_logprobRt_ProbabilisticMixIn__probt_ProbabilisticMixIn__logprob(R R6((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR æs   cCs||_d|_dS(u’ Set the probability associated with this object to ``prob``. :param prob: The new probability :type prob: float N(R!RR"(R R]((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRýs cCs||_d|_dS(uÿ Set the log probability associated with this object to ``logprob``. I.e., set the probability associated with this object to ``2**(logprob)``. :param logprob: The new log probability :type logprob: float N(R"RR!(R Rb((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR s cCs<|jdkr5|jdkr"dSd|j|_n|jS(u\ Return the probability associated with this object. :rtype: float iN(R!RR"(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR]s cCsD|jdkr=|jdkr"dStj|jdƒ|_n|jS(u Return ``log(p)``, where ``p`` is the probability associated with this object. :rtype: float iN(R"RR!R^R_(R ((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRbs (RURVRWR RR R]Rb(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRÍs   tImmutableProbabilisticMixIncBseZd„Zd„ZRS(cCstd|jjƒ‚dS(Nu%s is immutable(R R?RU(R R]((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR+scCstd|jjƒ‚dS(Nu%s is immutable(R R?RU(R R]((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR -s(RURVRR (((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR#*s cCs*||kr ||}||=n|}|S(N((R6RBtdefaulttarg((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyR(2s    cCsnddl}tƒ}xRt|ƒD]D}|jdd|dƒ|jd|dƒ}||cd7s  cCsotƒ}xYtdd|ddƒD]<}x3td|ddƒD]}|||cd7ˆsiÿÿÿÿu-u %3d %8.6f u%8.6f u| %8.6fuTotal u%siFu fdist1: %su fdist2: %su fdist3: %su Generating:c3s|]}ˆjƒVqdS(N(Rj(RAR=(Rø(sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys Ÿsiˆu%20s %sii7Nu =========u ---------u ---------u =========(R)R‡RŠR•R R®R*R0RQRîRR]R<RR*R´R RR?RU( R'R(R¤R¥tfdist3tpdiststvalstnt FORMATSTRtvaltzvalstsumsR((Røsb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytdemoWsP   +" #  "*cCs¬ddlm}|jjdƒ}t|ƒ}t|ƒ}tdd ƒd„t|jƒd d „d t ƒDƒ}x2|D]*}td ||||j |ƒfƒqzWdS(Niÿÿÿÿ(tcorpusuausten-emma.txtu%18s %8s %14suwordufreqencyuSimpleGoodTuringcss|]\}}|VqdS(N((RARBtvalue((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pys ªsRBcSs|dS(Ni((R((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyRIªstreverseu%18s %8d %14e(uwordufreqencyuSimpleGoodTuring( tnltkR5t gutenbergtwordsRR®R<RµRSR,R](R5t emma_wordsR§tsgttfd_keys_sortedRB((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pytgt_demo£s  + u__main__i iiˆuConditionalFreqDistuConditionalProbDistuConditionalProbDistIuCrossValidationProbDistuDictionaryConditionalProbDistuDictionaryProbDistu ELEProbDistuFreqDistuSimpleGoodTuringProbDistuHeldoutProbDistuImmutableProbabilisticMixInuLaplaceProbDistuLidstoneProbDistu MLEProbDistuMutableProbDistuKneserNeyProbDistu ProbDistIuProbabilisticMixInuUniformProbDistuWittenBellProbDistuadd_logsulog_likelihoodusum_logsuentropy(;RWt __future__RRR^RdRfRÜtoperatorRt collectionsRt functoolsRR8Rt nltk.compatRtnltk.internalsRRR`tpython_2_unicode_compatibleRtobjectR[RkRoR{R‡RŠR“R”R•R R©R®RÚRãR÷RúRûRuR R RR_RRR~RR#R(R)R*R4R>RUt__all__(((sb/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/probability.pyt'sŽ      ÿ!a $ , F - _ # $   D  Ûj t   Å #H  ]  L