U Dx`G@s4ddlZddlZddlZddlZddlmZmZm Z z ddl Z Wne k rXe Z YnXzddl m Z Wne k rddZ YnXddZdd Zd d Zd d ZddZddZddZddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Z d,d-Z!d.d/Z"dS)0N)SerializationContext py_bufferbuiltin_pickle)descr_to_dtypecCst|trt|St|trIsz!descr_to_dtype..)namesformatstitlesoffsetsitemsize) r strnpdtyper rlentypevoidrappendrzip)descrdtfieldsoffsetfieldnameZ descr_strshapeZis_padrrrZnametupsrrrrr%s0          rcCsd|}tj|tdddS)Nz'pyarrow.{}' is deprecated as of 2.0.0 and will be removed in a future version. Use pickle or the pyarrow IPC functionality instead.) stacklevel)formatwarningswarnDeprecationWarning)r#msgrrr_deprecate_serializationOs r,cCsT|jjdkr8|jjst|}|dtjj |jfS| tjj |jfSdSN|Ouint8) rrflags c_contiguousrascontiguousarrayviewlibr'dtype_to_descrtolistobjrrr_serialize_numpy_array_list\s   r9cCsV|ddkr6|djtjks t|dt|dStj|dt|ddSdS)Nrr.r)r)rrr/AssertionErrorr3rarraydatarrr_deserialize_numpy_array_listgs r>cCsZ|jjdkr<|jjs t|j}|jdtjj |jfS|j tjj |jfSdSr-) rrr0r1rr2Ar3r4r'r5r6r7rrr_serialize_numpy_matrixos   r@cCsb|ddkr@|djtjks ttj|dt|dddStj|dt|dddSdS)Nrr.rF)copy)rrA)rrr/r:matrixr3rr<rrr_deserialize_numpy_matrixzs  rCc Cs6t}tj||jd}||W5QRX|SN)schema)paBufferOutputStreamRecordBatchStreamWriterrEZ write_batchgetvalue)batch output_streamwrrrr_serialize_pyarrow_recordbatchsrMc Cs*t|}|W5QRSQRXdSr )rFRecordBatchStreamReaderZread_next_batchbufreaderrrr _deserialize_pyarrow_recordbatchs rRcCstj|gdg}t|S)Nr)rF RecordBatchZ from_arraysrM)r;rJrrr_serialize_pyarrow_arraysrTcCst|}|jdSNr)rRcolumns)rPrJrrr_deserialize_pyarrow_arraysrWc Cs6t}tj||jd}||W5QRX|SrD)rFrGrHrEZ write_tablerI)tablerKrLrrr_serialize_pyarrow_tablesrYc Cs*t|}|W5QRSQRXdSr )rFrNZread_allrOrrr_deserialize_pyarrow_tables rZcCstj|tjd}t|S)N)protocol)rdumpsHIGHEST_PROTOCOLr)xZpickledrrr_pickle_to_buffersr_cCst|}t|Sr ) memoryviewrloads)r=Z as_memoryviewrrr_load_pickle_from_buffersrbcs6z ddlWntk r"YdSXddlmdfddfddfdd}fd d }|jjd ||d |jjd ttd t j drt j j dr|jj j j j dttd t j j dr|jj j jjdttd t j j dr|jj j jjdttd |jjdd dS)Nrz{0} serialization is not supported. Note that {0} is planned to be deprecated in pandas future releases. See https://github.com/pandas-dev/pandas/issues/19239 for more information.cs,jjr"t|jr"td|S)NSparseDataFrame) _pandas_api has_sparser rcNotImplementedErrorr'Zdataframe_to_serialized_dictr7)pdpdcompatsparse_type_error_msgrr_serialize_pandas_dataframes zE_register_custom_pandas_handlers.._serialize_pandas_dataframecs |Sr )Zserialized_dict_to_dataframer<)rhrr_deserialize_pandas_dataframeszG_register_custom_pandas_handlers.._deserialize_pandas_dataframecs6jjr"t|jr"td|j|iS)N SparseSeries)rdrer rlrfr' DataFramer#r7)rjrgrhrirr_serialize_pandas_seriess zB_register_custom_pandas_handlers.._serialize_pandas_seriescs|}||jdSrU)rV)r=Z deserialized)rkrr_deserialize_pandas_seriesszD_register_custom_pandas_handlers.._deserialize_pandas_seriesz pd.SeriesZcustom_serializerZcustom_deserializerzpd.Indexarraysintervalz%pd.core.arrays.interval.IntervalArrayperiodz!pd.core.arrays.period.PeriodArray datetimesz&pd.core.arrays.datetimes.DatetimeArrayz pd.DataFrame)Zpandas ImportErrorZpyarrow.pandas_compatZ pandas_compat register_typeZSeriesZIndexr_rbhasattrcorerqrrZ IntervalArrayrsZ PeriodArrayrtZ DatetimeArrayrm)contextrnror)rkrjrgrhrir _register_custom_pandas_handlerssf       rzc stdzfddldd}fdd}jjjjjjjj j f D]}|j |d|j ||dqNWnt k rYnXdS) N%register_torch_serialization_handlersrcSsH|jr8tjj||jt |j dS|SdS)N)r$) Z is_sparserFSparseCOOTensor from_numpyZ_valuesdetachnumpyZ_indicesTlistr$r7rrr_serialize_torch_tensorszFregister_torch_serialization_handlers.._serialize_torch_tensorcsJt|tjr<j|dj|ddddf|jdS|SdS)Nrr)indicesvaluessize)r rFr|Zsparse_coo_tensorto_numpyrr$r}r<torchrr_deserialize_torch_tensors  zHregister_torch_serialization_handlers.._deserialize_torch_tensorztorch.rp)r,rZ FloatTensorZ DoubleTensorZ HalfTensorZ ByteTensorZ CharTensorZ ShortTensorZ IntTensorZ LongTensorZTensorrv__name__ru)serialization_contextrrtrrrr{s*  r{c Csdd}dd}|jtjd||ddd}d d }|jtjd ||dd d }dd}|jtjd||ddd}dd}|jtjd||ddS)NcSst|Sr )rr7rrr_serialize_deque5szF_register_collections_serialization_handlers.._serialize_dequecSs t|Sr ) collectionsdequer<rrr_deserialize_deque8szH_register_collections_serialization_handlers.._deserialize_dequezcollections.dequerpcSst|t|fSr rkeysrr7rrr_serialize_ordered_dict@szM_register_collections_serialization_handlers.._serialize_ordered_dictcSstt|d|dSNrr)r OrderedDictrr<rrr_deserialize_ordered_dictCszO_register_collections_serialization_handlers.._deserialize_ordered_dictzcollections.OrderedDictcSst|t||jfSr )rrrdefault_factoryr7rrr_serialize_default_dictKszM_register_collections_serialization_handlers.._serialize_default_dictcSst|dt|d|dS)Nrrr)r defaultdictrr<rrr_deserialize_default_dictNszO_register_collections_serialization_handlers.._deserialize_default_dictzcollections.defaultdictcSst|t|fSr rr7rrr_serialize_counterVszH_register_collections_serialization_handlers.._serialize_countercSsttt|d|dSr)rCounterdictrr<rrr_deserialize_counterYszJ_register_collections_serialization_handlers.._deserialize_counterzcollections.Counter)rvrrrrr) rrrrrrrrrrrr,_register_collections_serialization_handlers4s@rcszxddlm}m}m}mmmmfdd}dd}|j|d||d|j|d ||d|j|d ||dWnt k rYnXdS) Nr) csr_matrix csc_matrix coo_matrixisspmatrix_cooisspmatrix_csrisspmatrix_csc isspmatrixcsz|rdtj|fS|r0dtj|fS|rHdtj|fS|rddtj|fStd|ddS)Ncoocsrcsc%Serialization of {} is not supported.r)rFr|Z from_scipyZSparseCSRMatrixZSparseCSCMatrixZto_coorfr'r7rrrrrr_serialize_scipy_sparsels z9_register_scipy_handlers.._serialize_scipy_sparsecSsX|ddkr|dS|ddkr0|dS|ddkrH|dS|dSdS)Nrrrrr)Zto_scipyr<rrr_deserialize_scipy_sparse}s      z;_register_scipy_handlers.._deserialize_scipy_sparsezscipy.sparse.coo.coo_matrixrpzscipy.sparse.csr.csr_matrixzscipy.sparse.csc.csc_matrix) Z scipy.sparserrrrrrrrvru)rrrrrrrrr_register_scipy_handlersfs0$  rcsRz8ddlfdd}fdd}|jjd||dWntk rLYnXdS)Nrcs0t|jrdtj|fStdjdS)Nrr)r COOrFr|Zfrom_pydata_sparserfr'r7sparserr_serialize_pydata_sparses   zB_register_pydata_sparse_handlers.._serialize_pydata_sparsecsD|ddkr@|d\}}j|dddf|j|djdSdS)Nrrr)r=coordsr$)rrrr$)r=Z data_arrayrrrr_deserialize_pydata_sparses zD_register_pydata_sparse_handlers.._deserialize_pydata_sparsez sparse.COOrp)rrvrru)rrrrrr _register_pydata_sparse_handlerss   rcCs|jtdddddd|jtddddd |jtd dd |jtjd ttd|jtjd tt d|jt j d t t d|jt jdttd|jt jdttdt|t|t|t|dS)NintcSst|Sr )rr7rrrz:_register_default_serialization_handlers..cSst|Sr )rr<rrrrrrpcSsdSrUrrrrrrrfunctionT)picklerz np.matrixznp.arrayz pyarrow.Arrayzpyarrow.RecordBatchz pyarrow.Table)rvrrrrBr@rCZndarrayr9r>rFArrayrTrWrSrMrRZTablerYrZrrzrrrrrr(_register_default_serialization_handlerss\ rcCstdt|dS)N'register_default_serialization_handlers)r,rrrrrrsrcCstdt}t||S)Ndefault_serialization_context)r,rr)ryrrrrsr)#rr(rrZpyarrowrFZ pyarrow.librrrZ cloudpickleruZnumpy.lib.formatrr,r9r>r@rCrMrRrTrWrYrZr_rbrzr{rrrrrrrrrrsB  *     Q&2;2