ddl?Z?ddlZd5Z@d6d7ZAd8ZBd9ZCd:ZDd;ZEe8ZFeFjGZGd<ZHd=ZId>ZJeKd?kr=dd@lLmMZMeMZNeNjOdAdBdCdDdEdFdGdHeNjOdIdJdCdKdLdMdNe dEdOeNjOdPdQdCdRdLdMdNe dEdSeNjOdTdUdCdVdLdMdNe dEdWeNjOdXdYdCdZdNePdEd[eNjQ\ZRZSe8d\eRjTZUeSr x]eSD]UZVeUjGd]eVd^eRjWd_eRjXd`eRjYdaeRjZZ[e[e koeRjZrPnqWn.eUjGd^eRjWd_eRjXd`eRjYdaeRjZndS(bu$ The NLTK corpus and module downloader. This module defines several interfaces which can be used to download corpora, models, and other data packages that can be used with NLTK. Downloading Packages ==================== If called with no arguments, ``download()`` will display an interactive interface which can be used to download and install new packages. If Tkinter is available, then a graphical interface will be shown, otherwise a simple text interface will be provided. Individual packages can be downloaded by calling the ``download()`` function with a single argument, giving the package identifier for the package that should be downloaded: >>> download('treebank') # doctest: +SKIP [nltk_data] Downloading package 'treebank'... [nltk_data] Unzipping corpora/treebank.zip. NLTK also provides a number of "package collections", consisting of a group of related packages. To download all packages in a colleciton, simply call ``download()`` with the collection's identifier: >>> download('all-corpora') # doctest: +SKIP [nltk_data] Downloading package 'abc'... [nltk_data] Unzipping corpora/abc.zip. [nltk_data] Downloading package 'alpino'... [nltk_data] Unzipping corpora/alpino.zip. ... [nltk_data] Downloading package 'words'... [nltk_data] Unzipping corpora/words.zip. Download Directory ================== By default, packages are installed in either a system-wide directory (if Python has sufficient access to write to it); or in the current user's home directory. However, the ``download_dir`` argument may be used to specify a different installation target, if desired. See ``Downloader.default_download_dir()`` for more a detailed description of how the default download directory is chosen. NLTK Download Server ==================== Before downloading any packages, the corpus and module downloader contacts the NLTK download server, to retrieve an index file describing the available packages. By default, this index file is loaded from ``http://www.nltk.org/nltk_data/``. If necessary, it is possible to create a new ``Downloader`` object, specifying a different URL for the package index file. Usage:: python nltk/downloader.py [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS or:: python -m nltk.downloader [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS i(tprint_functiontdivisiontunicode_literalsN(tmd5( tTktFrametLabeltEntrytButtontCanvastMenutIntVartTclError(t showerror(tTable(tShowText(t ElementTree(tcompattPackagec B sYeZdZdddddddddded ZedZdZdZ RS(u4 A directory entry for a downloadable package. These entries are extracted from the XML index file that is downloaded by ``Downloader``. Each package consists of a single file; but if that file is a zip file, then it can be automatically decompressed when the package is installed. uuUnknowncK s||_|p||_||_||_t||_t||_||_||_| |_ | |_ | |_ | |_ t jj|jddd}t jj||||_tt| |_|jj|dS(Nu/ii(tidtnametsubdirturltinttsizet unzipped_sizetchecksumt svn_revisiont copyrighttcontacttlicensetauthortostpathtsplitexttsplittjointfilenametbooltunzipt__dict__tupdate(tselfRRRRRRRRRRRRR'tkwtext((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt__init__s          #cC sbt|tjr$tj|}nx.|jD]#}tj|j||j|(R(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt__repr__ sN( t__name__t __module__t__doc__tNonetTrueR-t staticmethodR5R7R8(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs 6 t CollectioncB s;eZdZddZedZdZdZRS(u A directory entry for a collection of downloadable packages. These entries are extracted from the XML index file that is downloaded by ``Downloader``. cK s>||_|p||_||_d|_|jj|dS(N(RRtchildrenR<tpackagesR(R)(R*RR@RR+((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-s    cC st|tjr$tj|}nx.|jD]#}tj|j||j|(R(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR81sN( R9R:R;R<R-R>R5R7R8(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR? s   tDownloaderMessagecB seZdZRS(uVA status message object, used by ``incr_download`` to communicate its progress.(R9R:R;(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRE8stStartCollectionMessagecB seZdZdZRS(u<Data server has started working on a collection of packages.cC s ||_dS(N(t collection(R*RG((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-=s(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRF;stFinishCollectionMessagecB seZdZdZRS(u=Data server has finished working on a collection of packages.cC s ||_dS(N(RG(R*RG((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-@s(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRH>stStartPackageMessagecB seZdZdZRS(u-Data server has started working on a package.cC s ||_dS(N(tpackage(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Cs(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRIAstFinishPackageMessagecB seZdZdZRS(u.Data server has finished working on a package.cC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Fs(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRKDstStartDownloadMessagecB seZdZdZRS(u.Data server has started downloading a package.cC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Is(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRLGstFinishDownloadMessagecB seZdZdZRS(u/Data server has finished downloading a package.cC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Ls(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRMJstStartUnzipMessagecB seZdZdZRS(u,Data server has started unzipping a package.cC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Os(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRNMstFinishUnzipMessagecB seZdZdZRS(u-Data server has finished unzipping a package.cC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Rs(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyROPstUpToDateMessagecB seZdZdZRS(u/The package download file is already up-to-datecC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Us(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRPSst StaleMessagecB seZdZdZRS(u3The package download file is out-of-date or corruptcC s ||_dS(N(RJ(R*RJ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-Xs(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRQVst ErrorMessagecB seZdZdZRS(u Data server encountered an errorcC s7||_t|tr*t||_n ||_dS(N(RJR.t Exceptiontstrtmessage(R*RJRU((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-[s (R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRRYstProgressMessagecB seZdZdZRS(u4Indicates how much progress the data server has madecC s ||_dS(N(tprogress(R*RW((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-ds(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRVbstSelectDownloadDirMessagecB seZdZdZRS(u:Indicates what download directory the data server is usingcC s ||_dS(N(t download_dir(R*RY((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-gs(R9R:R;R-(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRXest DownloadercB seZdZd$ZdZdZdZdZdZd%d%dZ d%e e e e e dZ d Zd Zd Zd Zd Zd%e dZdZdZdZd%d%e e de e dZd%dZd%dZd%dZd%dZdZe ddZd%dZdZdZ dZ!dZ"dZ#e$e"e#Z%d Z&d!Z'd"Z(e$e'e(Z)d#Z*RS(&uy A class used to access the NLTK data server, which can be used to download corpora and other data packages. i<uhttp://www.nltk.org/nltk_data/u installedu not installedu out of dateupartialcC sv|p |j|_i|_i|_||_d|_d|_i|_d|_ |jdkrr|j |_ndS(N( t DEFAULT_URLt_urlt _collectionst _packagest _download_dirR<t_indext_index_timestampt _status_cachet_errorstdefault_download_dir(R*tserver_index_urlRY((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-s       cC sd}|dkr,|j}td|n|rtddt|jtd|jtddt|j|d7}nt}} g} |r| jdn|r| jdnx| D]} td | j|d 7}xhtt || d t D]H} |j | |} | |j kr?|r?q n| |j krWt}n| |jkrot} nid |j 6d |j 6d|j6d|j6| }tjd d| jp| jddddd}td|| jjdd|f|t|jd7}|r |dkr tjd}|jd krKdSd}q q WtqWd}|r||d7}n| r|d7}nttj|ddddddS(!Niu!Using default data directory (%s)u=iu Data server index for <%s>iupackagesu collectionsu%s:iR4u*u-uPu iiKtsubsequent_indentu [%s] %s %siu.u uHit Enter to continue: uxuqu([*] marks installed packagesu+; [-] marks out-of-date or corrupt packagesu+; [P] marks partially installed collectionsu)twidthiL(uxuq(R<R_tprinttlenR\tFalsetappendt capitalizetsortedtgetattrRTtstatust INSTALLEDtSTALER=tPARTIALt NOT_INSTALLEDttextwraptfillRRtljustR#Rt raw_inputtlower(R*RYt show_packagestshow_collectionstheadert more_prompttskip_installedtlineststaletpartialt categoriestcategorytinfoRotprefixRt user_inputtmsg((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytlistsX      %  &    cC s|j|jjS(N(t _update_indexR^tvalues(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRAs cC s?|jg|jjD]!\}}|jdkr|^qS(Nucorpora(RR^titemsR(R*Rtpkg((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytcorporas cC s?|jg|jjD]!\}}|jdkr|^qS(Nucorpora(RR^RR(R*RR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytmodelss cC s|j|jjS(N(RR]R(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt collectionss cC s't|tjr|j|S|SdS(N(R.RR/R(R*t info_or_id((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt _info_or_ids cc s"|dkr#|j}t|Vnt|ttfrax"|j|||D] }|VqNWdSy|j|}Wn2tt fk r}t dd||fVdSXt|t rt |Vx%|j |j||D] }|VqWt|Vn%x"|j|||D] }|VqWdS(NuError loading %s: %s(R<R_RXR.Rttuplet_download_listRtIOErrort ValueErrorRRR?RFt incr_downloadR@RHt_download_package(R*RRYtforceRRte((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR s(      cC s$t|trdSt|jSdS(Ni(R.RRiRA(R*titem((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt _num_packages(sc # s:xdtt|D]P}yj||||5sig?id(trangeRiRRRRRtsumt enumerateR.RtfloatRARRVRW( R*RRYRtiRt num_packagesRWRtdeltaR((R*sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR,s"   c c st|VtdV|j||}| rc||jkrct|VtdVt|VdS|jj|jdt j j ||j }t j j|r||jkrt|Vnt j|nt j j|st j|nt j jt j j ||js8t jt j j ||jnt|VtdVytj|j}t|d}tdt|jd}xmtjD]_} |jd} |j| | sPn| ddkrtt d dd | |VqqWWdQX|j!Wn5t"k rI} t#|d |j|j| fVdSXt$|Vtd V|j j%d rt j j ||j} |j&st j jt j j | |jrt'|Vx+t(|| d t)D]} || _*| VqWt+|Vqnt|VdS(NiidiuwbiiiiiPiKu$Error downloading %r from <%s>: %su.ziptverbosei@i@(,RIRVRoRpRPRKRbtpopRR<R R!R$R%texistsRqRQtremovetmkdirRRLRturlopenRtopentmaxRRt itertoolstcounttreadtwritetmintcloseRRRRMtendswithR'RNt _unzip_iterRjRJRO(R*RRYRRotfilepathtinfiletoutfilet num_blockstblocktsRtzipdirR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRDs^     $"   -   -   u [nltk_data] c  sW|dkr2|dk r$||_n|jtSdfd}x |j|||D]} t| tr|| j|rt| jn|rt St|_ |st dt j j} | dkr|j| jj|||||st Sq| dkrt Sqn|sZt| trY|d| jjd 7t qOt| trt d |j r|d | jjqL|d | jjqOt| tr|d | jj|fqOt| tr|d| jjdqOt| tr1|d| jjdqOt| trO| j}qOqZqZWtS(Nuc  s/ttj|d|d|ddS(Ntinitial_indentRfu iu (RhRtRu(Rtprefix2(R(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytshowsu(Error installing package. Retry? [n/y/e]uyuYueuEuDownloading collection %ru | iu$Downloaded collection %r with errorsuDone downloading collection %suDownloading package %s to %s...u!Package %s is already up-to-date!u u Unzipping %s.(uyuY(ueuE(R<R_t_interactive_downloadR=RR.RRRURRjRcRhRRwtstriptdownloadRJRRFRGRHRIRPRNR%RXRY( R*RRYtquietRRt halt_on_errortraise_on_errorRRtchoice((Rsa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs`                cC s|j|||jkS(N(RoRq(R*RRY((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytis_stalescC s|j|||jkS(N(RoRp(R*RRY((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt is_installedscC s3|dkr|jjn|jj|ddS(N(R<RbtclearR(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytclear_status_caches cC sN|dkr|j}n|j|}t|trg|jD]}|j|j^q@}|j|krt|jS|j |kr|j S|j |kr|j |kr|j S|j |kr|j S|j Sn{t j j||j}||jkr |j||}n?|j|jkr<|j|||j|js (R R!RRsRtOSErrorRRRRqt md5_hexdigestRRRptisdirRtwalkR(R*RRtfilestattunzipdirR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs*    cC sY|jxH|jD]:}|j||jkr|j|d|d|qqWdS(uA Re-download any packages whose status is STALE. RRN(RRARoRqR(R*RRR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR)s c C sF|jdkp4|dk p4tj|j|jks;dS|pG|j|_tjjt j t j |jj |_tj|_g|jjdD]}tj|^q}td|D|_g|jjdD]}tj|^q}td|D|_x|jjD]}xt|jD]u\}}||jkrr|j||j|?sucollections/collectioncs s|]}|j|fVqdS(N(R(Rtc((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pys Dsu.removing collection member with no package: {}( R`R<ttimeRat INDEX_TIMEOUTR\tnltkt internalstElementWrapperRR0RRtgetrootRBRR5tdictR^R?R]RRR@RhtformatR.textendRRARbR( R*RRRARRRGRtchild_idtqueueRD((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR+s< '++  cC s|j|jS(u Return the XML index describing the packages available from the data server. If necessary, this index will be downloaded from the data server. (RR`(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytindex_s cC sR|j||jkr$|j|S||jkr>|j|Std|dS(uNReturn the ``Package`` or ``Collection`` record for the given item.uPackage %r not found in indexN(RR^R]R(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRhs    cC s|jx3|jjdD]}|jd|kr|SqWx3|jjdD]}|jd|krS|SqSWtd|dS(u-Return the XML info record for the given itemupackages/packageuiducollections/collectionuPackage %r not found in indexN(RR`RBRCR(R*RRJRG((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytxmlinfops cC s|jS(u)The URL for the data server's index file.(R\(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_get_urlscC s4|j}y|j|Wn||_nXdS(u Set a new URL for the data server. If we're unable to contact the given url, then the original url is kept. N(R\R(R*Rt original_url((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_set_urls   cC sdtjkrdSx<tjjD].}tjj|r tjj|r |Sq Wtj dkrdtjkrtjd}n-tjj d}|dkrt dntjj |dS(u Return the directory to which packages will be downloaded by default. This value can be overridden using the constructor, or on a case-by-case basis using the ``download_dir`` argument when calling ``download()``. On Windows, the default download directory is ``PYTHONHOME/lib/nltk``, where *PYTHONHOME* is the directory containing Python, e.g. ``C:\Python25``. On all other platforms, the default directory is the first of the following which exists or which can be created with write permission: ``/usr/share/nltk_data``, ``/usr/local/share/nltk_data``, ``/usr/lib/nltk_data``, ``/usr/local/lib/nltk_data``, ``~/nltk_data``. uAPPENGINE_RUNTIMENuwin32uAPPDATAu~/u+Could not find a default download directoryu nltk_data( R tenvironRtdataR!RRt is_writabletsystplatformt expanduserRR$(R*tnltkdirthomedir((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRds cC s|jS(u The default directory to which packages will be downloaded. This defaults to the value returned by ``default_download_dir()``. To override this default on a case-by-case basis, use the ``download_dir`` argument when calling ``download()``. (R_(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_get_download_dirscC s||_|jjdS(N(R_RbR(R*RY((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_set_download_dirs cC sUtrAyt|jWqQtk r=t|jqQXnt|jdS(N(tTKINTERt DownloaderGUItmainloopR tDownloaderShelltrun(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs  iN(+R9R:R;RR[RpRsRqRrR<R-R=RjRRARRRRRRRRRRRRRoRR)RRRRRRtpropertyRRdRRRYR(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRZmsR %  '        B   G    #  4     (  RcB sPeZdZdZdZdZdZdZdZdZ RS(cC s ||_dS(N(t_ds(R*t dataserver((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-scG sctdddtd|Dt|dd}td|j|tdddS(Nu-iKiDcs s|]}t|VqdS(N(Ri(Rto((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pys siu u (RhRRiR$(R*toptionstspc((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_simple_interactive_menus,cC stdxtr|jddddddtjdj}|sWtq n|jjd }|jd }y|d krt|jj |jj d t d tn|dkr|j nl|dkr|j nS|dkrdS|dkr|j|n'|dkr/|jntd|WnJtjk rd}td|n'tjk r}td|jnXtq WdS(NuNLTK Downloaderu d) Downloadul) Listu u) Updateu c) Configuh) Helpuq) Quitu Downloader> iiulR{R|uhucuquxuduuuCommand %r unrecognizeduError reading from server: %suError connecting to server: %s(uqux(RhR=RRRwRRxR#RRRYRjt_simple_interactive_helpt_simple_interactive_configt_simple_interactive_downloadt_simple_interactive_updatet HTTPErrortURLErrortreason(R*RtcommandtargsR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs>            cC sB|rZx5|D]F}y|jj|ddWq ttfk rR}t|q Xq Wnxtr=ttdtjd}|jdkr|jj |jj dt dtdtq]q]|jd krdS|r]xT|j D]F}y|jj|ddWqttfk r1}t|qXqWPq]q]WdS( NRu u*Download which package (l=list; x=cancel)?u Identifier> ulR{R|R}uxuqu(uxuqu( RRRRRhR=RRwRxRRYRjR#(R*RtargRRR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs.    c C sxtrg}t}}xctt|jddtD]@}|jj||jjkr;|j|j |j fq;q;Wt |rt dx\|D]T\}}t j dd|ddddd}t d |jd d |fqWt tjd }|jd krxxT|D]L\}}y|jj|ddWq$ttfk ro} t | q$Xq$WPq|jdkrdSqt ddSqWdS(NupackagesR4u/Will update following packages (o=ok; x=cancel)u-iiKRfu u [ ] %s %siu.u Identifier> uoRu uxuquuNothing to update.(uxuqu(R=RjRmRnRRTRoRqRkRRRhRtRuRvRRwRxRRR( R*tstale_packagesRRRtpidtpnameRRR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs4  (  $ cC s3ttdtdtdtddS(Nu Commands:uH d) Download a package or collection u) Update out of date packagesu1 l) List packages & collections h) Helpu1 c) View & Modify Configuration q) Quit(Rh(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR6s    cC sttdtd|jjtdt|jjtdt|jjttdtd|jjdS(Nu Data Server:u - URL: <%s>u$ - %d Package Collections Availableu$ - %d Individual Packages AvailableuLocal Machine:u - Data directory: %s(RhRRRiRRARY(R*((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt _show_config=s  cC s{|jxjtrvt|jddddtjdjj}|dkrd|jq |dkrtjdj}|dkrtdqstj j |r||j _ qstd|q |dkrctjdj}|dkr tdqs|j ds&d|}ny||j _Wqstk r_}td||fqsXq |dkr Pq q WdS(Nus) Show Configuu) Set Server URLud) Set Data Diru m) Main MenuuConfig> usudu New Directory> uuxuquXuQu Cancelled!u)Directory %r not found! Create it first.uuu New URL> uhttp://uError reading <%r>: %sum(uuxuquXuQ(uuxuquXuQ(R R=RhRRRwRRxR R!RRRYt startswithRRS(R*Rt new_dl_dirtnew_urlR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRIs:              ( R9R:R-RRRRRR R(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs  "    Rc B seZdZddddddddd d d d g Zid d6dd6d d6d d6Zidd6dd6dd6dd6dd6dd6ZdZdddddgZx eD]Zeekst qWx eD]Zeekst qWx eD]Zeekst qWdNZ idOe j 6dPe j 6dQe j6dRe j6ZdSZdTZdUZdVZd#Zed$Zd%Zd&Zd'Zd(Zd)Zd*ZdWd+Zd,Z d-Z!d.Z"d/Z#d0Z$d1Z%d2Z&d3Z'dWZ(d4Z)d5Z*d6Z+dZ,d7Z-d8Z.d9Z/d:Z0d;Z1d<Z2d=Z3d>Z4d?Z5d@Z6dAZ7e8j9dBZ:dCZ;dDZ<dZ=dEZ>dFZ?dGZ@dHZAdIZBdJeCjDfdKYZEdLZFdMZGRS(XuU Graphical interface for downloading packages from the NLTK data server. uu IdentifieruNameuSizeuStatusu Unzipped Sizeu CopyrightuContactuLicenseuAuthoruSubdiruChecksumiiiii-i i iu#000u#cccu#afau#080u#ffau#880u#faau#800u#fffu#888u#45cu#aaau#67au#f00uhelvetica -16 boldcC s||_||_tj|_g|_g|_t|_i|_ g|_ d|_ |j dt }|_|jd|jd|jd|jd|jd|j|jd|jt|_i|_|j|jy|jWnHtjk r$}td |n&tjk rI}td |jnX|j|j |j!j"d|j!jd |j#dS( NiuNLTK Downloader Started!u+50+50uNLTK Downloadert backgroundiu u uError reading from serveruError connecting to serveru ($Rt _use_threadst threadingtLockt_download_lockt_download_msg_queuet_download_abort_queueRjt _downloadingt_afteridt _log_messagest _log_indentt_logRttoptgeometryttitlet configuret_BACKDROP_COLORtbindtdestroyt _destroyedt _column_varst _init_widgetst _init_menut _fill_tableRRR RRt _show_infot_select_columnst_tabletselectt_destroy(R*Rt use_threadsRR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR-s<                 cC s.|jjdtjd|j|fdS(Nu%s %s%su | (RRkRtctimeR(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRsc  stjdddddddd}|jd d d td d |jddd|jdddt|ddjddddt|}|jddddddt|}|jddddddt|}|jddddddt|ddjddddt|}|jddddddt|ddjddddtjdddddjd}|jddd ddjd|jddd|dd?q%Wjj d'j%t&|d#d@dAjd.d_'j'jdd%t&|d#dBdAj(d.d_)j)jddCt |d#dDd-jddjd_*t+|d.dEdd+dj,dddFdd_-j.j-jddCj*jdd%dS(GNtreliefuraisedtborderitpadxitpadyitsidutoptexpandRuubothtweightitheighttcolumntrowtstickyunewsiiiiRtsideubottomuxuborderuhighlightthicknessu CollectionsuCorporauModelsu All Packagesttexttfontulefti u tcolumn_weightsthighlightthicknesstlistbox_heightitreprfunct foregroundRguuuuuu uurlu Server Index:u download_diruDownload Directory:ueucourierugroovetdisabledforegroundublackc s j|S(N(t _info_edit(RR4(R*(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt*suewuDownloadRuRefreshurightuiusunken(/RRtpackR=tgrid_rowconfiguretgrid_columnconfiguretgridRt _tab_namest_tabsRRt _TAB_FONTR t _select_tabRxtCOLUMNStCOLUMN_WEIGHTSRCRt_table_reprfuncR)t columnconfigt _MARK_COLORt COLUMN_WIDTHStDEFAULT_COLUMN_WIDTHtfocustbind_to_listboxest _downloadt _table_markt _prev_tabt _next_tabt _mark_allRRt_infoRt _info_saveRt_download_buttont_refresht_refresh_buttont_progresslabelR t_PROGRESS_COLORt _progressbart_init_progressbar(R*tf1ttabframet tableframet buttonframet infoframet progressframeRttabtlabelR6R<RgRR4tcallbacktentry((R*sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR$s'"   " "   "(     "(  !!   c  s%tj}t|dd}|jdddddjdd|j|jdd dd dfd |jdd dddfd |j|jdddddj|j|jdddddjdd|jddddd|t|dd}xjj dD]{}t j}|j ks[t |j |<|j kr|jdn|jd|ddd|djq1W|jddddd|t|dd}x>jj dD],}|jdd|d|fdqW|jx>jj dD],}|jdd|d|fdq:W|jddddd|t|dd}|jdddddj|jdddddjdd|jdd ddd|jjd!jjjd|dS("NttearoffiRjuDownloadt underlineRt acceleratoruReturnuChange Server Indexic s jdS(Nuurl(RB((R*(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRCLsuChange Download Directoryc s jdS(Nu download_dir(RB((R*(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRCNsuShow LogiuExitiuCtrl-xuFiletmenuitvariableuViewu Sort by %sc sjj|dS(Nu ascending(R)tsort_by(R(R*(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRCjsuReverse sort by %sc sjj|dS(Nu descending(R)Rr(R(R*(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRCpsuSortuAboutu InstructionsuF1uHelpu(R Rt add_commandRUt add_separatort _show_logR!t add_cascadeR)t column_namesR R#tAssertionErrortINITIAL_COLUMNStsettadd_checkbuttonR(taboutthelpR tconfig(R*tmenubartfilemenutviewmenuR6tvartsortmenuthelpmenu((R*sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR%DsV       cC sSxL|jjD];\}}|jr;|jj|q|jj|qWdS(N(R#RRCR)t show_columnt hide_column(R*R6R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR(~s cC sz|jjy|jWnHtjk r@}td|n&tjk re}td|jnX|jj ddS(NuError reading from serveruError connecting to serveri( RRR&RRR RRR)R*(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR]s cC s?|j|j|\}}d|d|jdq|dkr]|j|jq|dkr|j|jj|q|jjdd}|jt||dqW|S(u Given a package, return a list of values describing that package, one for each column in ``self.COLUMNS``. iuu IdentifieruStatusu u_un/a( RRLRkRRRoRxtreplaceRn(R*RR7t column_indext column_nametattr((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs   cG s'|jr dS|jjt|_dS(N(R"RR!R=(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR!s  cG sl|jdk r<x*|jjD]}|jj|qWn|jr[|jr[|jn|jj dS(N( RR<RRt after_cancelRRt_abort_downloadR#R(R*RR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR+s  cO s|jj||dS(N(RR(R*Rtkwargs((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRsun This tool can be used to download a variety of corpora and models that can be used with NLTK. Each corpus or model is distributed in a single zip file, known as a "package file." You can download packages individually, or you can download pre-defined collections of packages. When you download a package, it will be saved to the "download directory." A default download directory is chosen when you run the downloader; but you may also select a different download directory. On Windows, the default download directory is "package." The NLTK downloader can be used to download a variety of corpora, models, and other data packages. Keyboard shortcuts:: [return] Download [up] Select previous package [down] Select next package [left] Select previous tab [right] Select next tab cG s\y,t|jd|jjddddWn)t|jd|jjddnXdS(NuHelp: NLTK DowloaderRgiKR;ufixeduHelp: NLTK Downloader(RRtHELPR(R*R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR}s    cG sdd}d}y-ddlm}|d|d|jWnetk ry-ddlm}|d|d|jWqtk rt|j||qXnXdS(NuNLTK Downloader uWritten by Edward LoperuAbout: NLTK Downloaderi(tMessageRUR(t tkMessageBoxRRt ImportErrorttkinter.messageboxRR(R*RtABOUTtTITLER((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR|s  cC s |j}t|dt|d}}xtdt|dd|jD]c}|j||jdd||j|d|dd|jdd d t|d d d qOW|jd|jddd|jd|j ddddd|j ddS(NuwidthuheightiiiiRgRuu #%02x0000iPiii ugradienttstateuhiddenuredbox( RaRRt_gradient_widtht create_linetabst addtag_allt itemconfigtaddtag_withtagtcreate_rectangleR`(R*RRgR5R((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRbs !+ % cC s|j}|dkrD|jddddd|jdddnVt|dt|d}}|t|dd }|jddd||d dS( NuredboxiugradientRuhiddenuwidthuheightidi(RaR<tcoordsRR(R*tpercentRRgR5tx((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs  !cC s|j}|js(|jdddn|jddd|jd\}}}}|dkr|jd|jdddn|jdd d|jjd |j}||j d (su unzipped_sizeu%susizeuchecksumusubdiruurlu collectionsuiduDuplicate UID: %su nltk_datai(t_find_packagesR R!R$RR%R#RtinfolistRzRRRkRt_find_collectionsRCRRRtElementt _indent_xml(Rtbase_urlRAtpkg_xmlRRtzipstatRRRtuidsRttop_eltRJRG((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt build_indexs4+&!    ucC st|dkr|jpdjd|d|_x|D]}t||dq=Wx6|d D]*}|jpudjd|d|_qcW|djpdjd||d_ndS(u Helper for ``build_index()``: Given an XML ``ElementTree``, modify it (and its descendents) ``text`` and ``tail`` attributes to generate an indented tree, where each nested element is indented by 2 spaces with respect to its parent. iuu u iN(RiR:RRttail(R3RRD((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRIs$ (c stjjtjj|dd|jdkr]td|jdfntfd|jDrtdfndS(ur Helper for ``build_index()``: Perform some checks to make sure that the given package is consistent. iiuidu&package identifier mismatch (%s vs %s)c3 s/|]%}|ko&|jd VqdS(u/N(R (RR(tuid(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pys dsu;Zipfile %s.zip does not expand to a single subdirectory %s/N(R R!R"R#RCRRR(Rt zipfilenameR((Rsa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt_check_packageXs&cC stjddd|gdtjdtj}|j\}}|jdks[|s[| rtdtjj|dt j |fn|jd S( u Helper for ``build_index()``: Calculate the subversion revision number for a given file (by using ``subprocess`` to run ``svn``). usvnustatusu-vRtstderriu)Error determining svn_revision for %s: %sii( t subprocesstPopentPIPEt communicatet returncodeRR R!R#RtRu(R%RRR((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt _svn_revisionjs  ,cc syg}xltj|D][\}}}xI|D]A}|jdr,tjj||}tj|jVq,q,WqWdS(u Helper for ``build_index()``: Yield a list of ElementTree.Element objects, each holding the xml for a single package collection. u.xmlN(R RRR!R$RR0R(RRARtsubdirsRR%txmlfile((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRxs  c # sddlm}g}xtj|D]\}}}dj|||}xa|D]Y}|jdrTtjj||}|d d} ytj| } Wn)t k r} t d| | fnXyt j |j } Wn)t k r} t d|| fnXtjj|d d| jd krat d | jd fntfd | jDrt d fn| | |fVqTqTWy|jd Wq&t k rq&Xq&WdS(u Helper for ``build_index()``: Yield a list of tuples ``(pkg_xml, zf, subdir)``, where: - ``pkg_xml`` is an ``ElementTree.Element`` holding the xml for a package - ``zf`` is a ``zipfile.ZipFile`` for the package's contents. - ``subdir`` is the subdirectory (relative to ``root``) where the package was found (e.g. 'corpora' or 'grammars'). i(t _path_fromu/u.xmliu.zipuError reading file %r! %siuidu&package identifier mismatch (%s vs %s)c3 s/|]%}|ko&|jd VqdS(u/N(R (RR(R(sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pys su;Zipfile %s.zip does not expand to a single subdirectory %s/u.svnN(tnltk.corpus.reader.utilRR RR$RR!RRRSRRR0RR#RCRRR( RRRARRRtrelpathR%t xmlfilenameRRRR((Rsa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyRs>   cC sttjdS(N(Rt _downloaderR(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytdownload_shellscC sttjdS(N(RRR(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyt download_guiscC stjdS(N(RR)(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pyR)su__main__(t OptionParseru-du--dirtdestudirR}u!download package to directory DIRtmetavaruDIRu-qu--quietuquiettactionu store_truetdefaultu work quietlyu-fu--forceuforceu"download even if already installedu-eu--exit-on-erroru halt_on_erroruexit if an error occursu-uu--urluserver_index_urludownload server index urlReRRYRRR(\R;t __future__RRRRR RRRtRRthashlibRR=RttkinterRRRRRR R R R RR tnltk.draw.tableRtnltk.draw.utilRRjRt xml.etreeRRRtpython_2_unicode_compatibletobjectRR?RERFRHRIRKRLRMRNRORPRQRRRVRXRZRRRRR'RR RRRRRRRRRRR)R9toptparseRtparsert add_optionR<t parse_argsRRRet downloadertpkg_idtdirRRRtrv(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/downloader.pytDs\T@   Q + iE 9 D    5