U C^"@s~ddlmZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z eZGd d d eZGd d d eZdS) )unicode_literalsN) OrderedDict) BloomFilter)Errors)SimpleFrozenDict ensure_path) get_string_idc@s~eZdZdZddZddZddZedd Ze fd d Z e fd d Z ddZ ddZddZddZddZddZdS)LookupsaSContainer for large lookup tables and dictionaries, e.g. lemmatization data or tokenizer exception lists. Lookups are available via vocab.lookups, so they can be accessed before the pipeline components are applied (e.g. in the tokenizer and lemmatizer), as well as within the pipeline components via doc.vocab.lookups. cCs t|_dS)zInitialize the Lookups object. RETURNS (Lookups): The newly created object. DOCS: https://spacy.io/api/lookups#init N)r_tablesselfr0/tmp/pip-install-6_kvzl1k/spacy/spacy/lookups.py__init__szLookups.__init__cCs ||S)zCheck if the lookups contain a table of a given name. Delegates to Lookups.has_table. name (unicode): Name of the table. RETURNS (bool): Whether a table of that name is in the lookups. ) has_tabler namerrr __contains__!szLookups.__contains__cCs t|jS)z3RETURNS (int): The number of tables in the lookups.)lenr r rrr__len__*szLookups.__len__cCst|jS)z3RETURNS (list): Names of all tables in the lookups.)listr keysr rrrtables.szLookups.tablescCs6||jkrttjj|dt||d}||j|<|S)aAdd a new table to the lookups. Raises an error if the table exists. name (unicode): Unique name of table. data (dict): Optional data to add to the table. RETURNS (Table): The newly added table. DOCS: https://spacy.io/api/lookups#add_table r)rdata)r ValueErrorrZE158formatTabler )r rrtablerrr add_table3s   zLookups.add_tablecCs6||jkr,|tkr(ttjj||jd|S|j|S)a:Get a table. Raises an error if the table doesn't exist and no default value is provided. name (unicode): Name of the table. default: Optional default value to return if table doesn't exist. RETURNS (Table): The table. DOCS: https://spacy.io/api/lookups#get_table rr)r UNSETKeyErrorrE159rr)r rdefaultrrr get_tableBs zLookups.get_tablecCs,||jkr ttjj||jd|j|S)zRemove a table. Raises an error if the table doesn't exist. name (unicode): Name of the table to remove. RETURNS (Table): The removed table. DOCS: https://spacy.io/api/lookups#remove_table r!)r r#rr$rrpoprrrr remove_tableRs zLookups.remove_tablecCs ||jkS)zCheck if the lookups contain a table of a given name. name (unicode): Name of the table. RETURNS (bool): Whether a table of that name exists. DOCS: https://spacy.io/api/lookups#has_table )r rrrrr^szLookups.has_tablecKs t|jS)zSerialize the lookups to a bytestring. RETURNS (bytes): The serialized Lookups. DOCS: https://spacy.io/api/lookups#to_bytes )srsly msgpack_dumpsr )r kwargsrrrto_byteshszLookups.to_bytescKsBt|_t|D]&\}}t||j|<|j||q|S)zLoad the lookups from a bytestring. bytes_data (bytes): The data to load. RETURNS (Lookups): The loaded Lookups. DOCS: https://spacy.io/api/lookups#from_bytes )rr r) msgpack_loadsitemsrupdate)r bytes_datar+keyvaluerrr from_bytesqs zLookups.from_bytesc KsRt|jrNt|}|s"||d}|d}||W5QRXdS)zSave the lookups to a directory as lookups.bin. Expects a path to a directory, which will be created if it doesn't exist. path (unicode / Path): The file path. DOCS: https://spacy.io/api/lookups#to_disk lookups.binwbN)rr rexistsmkdiropenwriter,)r pathr+filepathfile_rrrto_disks  zLookups.to_diskc KsDt|}|d}|r@|d}|}W5QRX||S|S)aLoad lookups from a directory containing a lookups.bin. Will skip loading if the file doesn't exist. path (unicode / Path): The directory path. RETURNS (Lookups): The loaded lookups. DOCS: https://spacy.io/api/lookups#from_disk r4rb)rr6r8readr3)r r:r+r;r<rrrr from_disks   zLookups.from_diskN)__name__ __module__ __qualname____doc__rrrpropertyrrr r"r&r(rr,r3r=r@rrrrr s       r c@sbeZdZdZedddZdddZddZd d Zd d Z dd dZ ddZ ddZ ddZ dS)rzA table in the lookups. Subclass of builtin dict that implements a slightly more consistent and unified API. Includes a Bloom filter to speed up missed lookups. NcCs||d}|||S)zInitialize a new table from a dict. data (dict): The dictionary. name (unicode): Optional table name for reference. RETURNS (Table): The newly created object. DOCS: https://spacy.io/api/lookups#table.from_dict r)r/)clsrrr rrr from_dicts  zTable.from_dictcCsRt|||_d|_|r.t|dkr.t|n|j}t||_|rN||dS)aInitialize a new table. name (unicode): Optional table name for reference. data (dict): Initial data, used to hint Bloom Filter. RETURNS (Table): The newly created object. DOCS: https://spacy.io/api/lookups#table.init g.ArN) rrrZ default_sizerrZfrom_error_ratebloomr/)r rrsizerrrrs  zTable.__init__cCs&t|}t||||j|dS)zSet new key/value pair. String keys will be hashed. key (unicode / int): The key to set. value: The value to set. N)r r __setitem__rHaddr r1r2rrrrJszTable.__setitem__cCs |||<dS)zSet new key/value pair. String keys will be hashed. Same as table[key] = value. key (unicode / int): The key to set. value: The value to set. NrrLrrrsetsz Table.setcCst|}t||S)zGet the value for a given key. String keys will be hashed. key (unicode / int): The key to get. RETURNS: The value. )r r __getitem__r r1rrrrNszTable.__getitem__cCst|}t|||S)zGet the value for a given key. String keys will be hashed. key (unicode / int): The key to get. default: The default value to return. RETURNS: The value. )r rget)r r1r%rrrrPsz Table.getcCs"t|}||jkrdSt||S)zCheck whether a key is in the table. String keys will be hashed. key (unicode / int): The key to check. RETURNS (bool): Whether the key is in the table. F)r rHrrrOrrrrs zTable.__contains__cCs4d|jfdt|fd|jfg}tt|S)zSerialize table to a bytestring. RETURNS (bytes): The serialized table. DOCS: https://spacy.io/api/lookups#table.to_bytes rdictrH)rrQr.rHr,r)r*r)r rrrrr,s  zTable.to_bytescCsHt|}|di}|d|_t|d|_||||S)zLoad a table from a bytestring. bytes_data (bytes): The data to load. RETURNS (Table): The loaded table. DOCS: https://spacy.io/api/lookups#table.from_bytes rQrrH) r)r-rPrrr3rHclearr/)r r0Zloadedrrrrr3s    zTable.from_bytes)N)NN)N)rArBrCrD classmethodrGrrJrMrNrPrr,r3rrrrrs     r) __future__rr) collectionsrZ preshed.bloomrerrorsrutilrrstringsr objectr"r rrrrrs