U Dx`,#@sddlZddlmZddlmZmZmZmZmZddlm Z ddZ GdddZ dd Z d d d hZdddZdddZdddZdS)N) _pandas_api)Codec FeatherErrorTable concat_tablesschemacCstjdkrtddS)Nz0.17.0z!feather requires pandas >= 0.17.0)rZ loose_version ImportErrorr r 6/tmp/pip-target-oguziej0/lib/python/pyarrow/feather.py_check_pandas_versions r c@s6eZdZdZd ddZd ddZdd Zdd d ZdS)FeatherDataseta Encapsulates details of reading a list of Feather files. Parameters ---------- path_or_paths : List[str] A list of file names validate_schema : bool, default True Check that individual file schemas are all the same / compatible TcCs||_||_dSN)pathsvalidate_schema)selfZ path_or_pathsrr r r __init__,szFeatherDataset.__init__NcCsjt|jd|d}|g|_|j|_|jddD].}t||d}|jrR||||j|q0t|jS)a, Read multiple feather files as a single pyarrow.Table Parameters ---------- columns : List[str] Names of columns to read from the file Returns ------- pyarrow.Table Content of the file as a table (of columns) rcolumnsN) read_tablerZ_tablesrrvalidate_schemasappendr)rrZ_filpathtabler r r r0s  zFeatherDataset.read_tablecCs(|j|js$td||j|jdS)Nz-Schema in {!s} was different. {!s} vs {!s})requals ValueErrorformat)rZpiecerr r r rIs zFeatherDataset.validate_schemascCst|j|dj|dS)a Read multiple Parquet files as a single pandas DataFrame Parameters ---------- columns : List[str] Names of columns to read from the file use_threads : bool, default True Use multiple threads when converting to pandas Returns ------- pandas.DataFrame Content of the file as a pandas DataFrame (of columns) r use_threadsr rZ to_pandas)rrrr r r read_pandasPs zFeatherDataset.read_pandas)T)N)NT)__name__ __module__ __qualname____doc__rrrr r r r r r s   r cCsN|jdkrdS|jttfkr4td|ntd|t|jdS)NrzqColumn '{}' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurezkColumn '{}' of type {} was chunked on conversion to Arrow and cannot be currently written to Feather format)Z num_chunkstypeextbinarystringrrstr)namecolr r r check_chunked_overflowes r,lz4ZzstdZ uncompressedc Csdtjr(ttjr(t|tjjr(|}t|rrt j |dd}|dkrvt |j j D]\}}||} t|| qTn|}|dkrt|jtt|jkrtd|dk rtd|dk rtdn8|dkrtdrd }n |dk r|tkrtd |tztj||||||d WnLtk r^t|trXzt|Wntjk rVYnXYnXdS) a Write a pandas.DataFrame to Feather format. Parameters ---------- df : pandas.DataFrame or pyarrow.Table Data to write out as Feather format. dest : str Local destination path. compression : string, default None Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses LZ4 for V2 files if it is available, otherwise uncompressed. compression_level : int, default None Use a compression level particular to the chosen compressor. If None use the default compression level chunksize : int, default None For V2 files, the internal maximum size of Arrow RecordBatch chunks when writing the Arrow IPC file format. None means use the default, which is currently 64K version : int, default 2 Feather file version. Version 2 is the current. Version 1 is the more limited legacy format F)Zpreserve_indexrz'cannot serialize duplicate column namesNz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize optionZ lz4_framer-z1compression="{}" not supported, must be one of {}) compressioncompression_level chunksizeversion)rZ have_pandasr Z has_sparse isinstancepdZSparseDataFrameZto_denseZ is_data_framerZ from_pandas enumeraternamesr,lenZ column_namessetrrZ is_available_FEATHER_SUPPORTED_CODECSrr& write_feather Exceptionr)osremoveerror) Zdfdestr/r0r1r2rir*r+r r r r:wsT      r:TcCstt|||dj|dS)a) Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use feather.read_table. Parameters ---------- source : str file path, or file-like object columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. use_threads: bool, default True Whether to parallelize reading using multiple threads. memory_map : boolean, default True Use memory mapping when opening file on disk Returns ------- df : pandas.DataFrame )r memory_maprr)sourcerrrAr r r read_feathersrCcCst}|j||d|dkr&|Sdd|D}ttdd|rR||}nszread_table..cSs|tkSr )inttr r r zread_table..cSs|tkSr )r)rHr r r rJrKcSsg|] }|jqSr )r!)rDrIr r r rFszs  E  K