U C^@sddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZddlmZGd d d eZd d Zd dZddZeeeeeZd gZdS))unicode_literalsN)NumpyOps)copy_reg)Doc)SPACYORTH intify_attr)Errorsc@sJeZdZdZdddZddZdd Zd d Zd d ZddZ ddZ dS)DocBinaaPack Doc objects for binary serialization. The DocBin class lets you efficiently serialize the information from a collection of Doc objects. You can control which information is serialized by passing a list of attribute IDs, and optionally also specify whether the user data is serialized. The DocBin is faster and produces smaller data sizes than pickle, and allows you to deserialize without executing arbitrary Python code. The serialization format is gzipped msgpack, where the msgpack object has the following structure: { "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] "tokens": bytes, # Serialized numpy uint64 array with the token data "spaces": bytes, # Serialized numpy boolean array with spaces data "lengths": bytes, # Serialized numpy int32 array with the doc lengths "strings": List[unicode] # List of unique strings in the token data } Strings for the words, tags, labels etc are represented by 64-bit hashes in the token data, and every string that occurs at least once is passed via the strings object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. A notable downside to this format is that you can't easily extract just one document from the DocBin. NFcCs\|pg}tdd|D}dd|D|_|jdtg|_g|_g|_t|_||_ dS)aCreate a DocBin object to hold serialized annotations. attrs (list): List of attributes to serialize. 'orth' and 'spacy' are always serialized, so they're not required. Defaults to None. store_user_data (bool): Whether to include the `Doc.user_data`. RETURNS (DocBin): The newly constructed object. DOCS: https://spacy.io/api/docbin#init cSsg|] }t|qS)r .0attrr r :/tmp/pip-install-6_kvzl1k/spacy/spacy/tokens/_serialize.py 8sz#DocBin.__init__..cSs g|]}|tkr|tkr|qSr )rrr r r rr9srN) sortedattrsinsertrtokensspaces user_datasetstringsstore_user_data)selfrrr r r__init__-s zDocBin.__init__cCs t|jS)z7RETURNS: The number of Doc objects added to the DocBin.)lenr)rr r r__len__AszDocBin.__len__cCs||j}t|jdkr.||jddf}|j||t}|jd|jdks\t||jddf}|j t j |t d|j dd|D|jr|jt|jdS)zAdd a Doc's annotations to the DocBin for serialization. doc (Doc): The Doc object to add. DOCS: https://spacy.io/api/docbin#add rZdtypecss|] }|jVqdSN)text)rwr r r TszDocBin.add..N)Zto_arrayrrshapereshaperappendrAssertionErrorrnumpyasarrayboolrupdaterrsrsly msgpack_dumps)rdocarrayrr r raddEs   z DocBin.addc #s|jD] }|q|jt}tt|jD]~}|j|}|j|}fdd|dd|fD}t||d}| |j|}|j rt j |j |dd} |j | |Vq.dS)zRecover Doc objects from the annotations, using the given vocab. vocab (Vocab): The shared vocab. YIELDS (Doc): The Doc objects. DOCS: https://spacy.io/api/docbin#get_docs csg|]}j|qSr )r)rZorthvocabr rrfsz#DocBin.get_docs..N)wordsrF)use_list)rrindexrrangerrrrZ from_arrayrr- msgpack_loadsrr,) rr3stringZorth_colirrr4r/rr r2rget_docsXs      zDocBin.get_docscCsf|j|jkr$ttjj|j|jd|j|j|j|j|j |j|j rb|j |j dS)a"Extend the annotations of this DocBin with the annotations from another. Will raise an error if the pre-defined attrs of the two DocBins don't match. other (DocBin): The DocBin to merge into the current bin. DOCS: https://spacy.io/api/docbin#merge )currentotherN) r ValueErrorr ZE166formatrextendrrr,rr)rr=r r rmergens z DocBin.mergecCs|jD]}t|jdkst|jqdd|jD}|jt|jdt|jdtj |dddt |j d}|j r|j |d<tt|S) zSerialize the DocBin's annotations to a bytestring. RETURNS (bytes): The serialized DocBin. DOCS: https://spacy.io/api/docbin#to_bytes rcSsg|] }t|qSr )r)rrr r rrsz#DocBin.to_bytes..Cint32r )rrrlengthsrr)rrr%r(rr)Zvstacktobytesrr*listrrrzlibcompressr-r.)rrrDmsgr r rto_bytess  zDocBin.to_bytescCstt|}|d|_t|d|_tj|ddd}tj|dt d}tj|ddd}|j t |jt |jf}| |}| |j d f}t |||_t |||_|jrd |krt|d |_|jD]}t |jd kst|jq|S) zDeserialize the DocBin's annotations from a bytestring. bytes_data (bytes): The data to load from. RETURNS (DocBin): The loaded DocBin. DOCS: https://spacy.io/api/docbin#from_bytes rrrDrCr rrZuint64rrr)r-r8rG decompressrrrr)Z frombufferr+sizerr&rZ unflattenrrrrFrr%r()r bytes_datarIrDZ flat_spacesZ flat_tokensr%rr r r from_bytess    zDocBin.from_bytes)NF) __name__ __module__ __qualname____doc__rrr1r;rArJrNr r r rr s r cCsVd}|D]4}|dk rtdd|}|dkr2|}q||q|dk rN|SdSdS)NT)r)r rNrArJ)ZbinsZmerged byte_stringdoc_binr r r merge_binss rVcCst|ffSr!) unpickle_binrJ)rUr r r pickle_binsrXcCs t|Sr!)r rN)rTr r rrWsrW) __future__rr)rGr-Zthinc.neural.opsrcompatrrrrrrr errorsr objectr rVrXrWpickleZBinder__all__r r r rs