B @`@sdZddlmZddlZddlmZmZddlZddlZddl Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZddlZddlZddlmZmZddlmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+dd l,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGddlHmIZImJZJmKZKddlLmMmNZOddlPmQZQmRZRddlSmTZTddlUmVZVddlWmXZXddlYmZZZm[Z[e rddl\m]Z]m^Z^m_Z_dZ`dZaddZbddZcdd ZdeQZeefd!d"d#ZgGd$d%d%ehZiGd&d'd'ehZjGd(d)d)ekZld*ZmGd+d,d,ekZnd-ZoGd.d/d/ekZpd0Zqd1Zrd2d2d3d3d4Zse>dgiZtd5Zud6Zvewd78ejxd8d9euejyd:ejxd;devezd2d3dgd:WdQRXda{d9a|defe@e>dtdudvZeee~efdwdxdyZdeeje@feee~efeeeje?fdzd{d|Ze~e@e~e~ed}d~dZe~e~e~eeje@fdddZe~dddZeje~e~ejdddZeje~e~ejdddZeje~e~e~dddZe~e~e~dddZe~edddZe~eefe~dddZe~e~dddZe#dddZGdddZdS)zY High level interface to PyTables for reading and writing pandas data structures to disk )suppressN)datetzinfo)dedent) TYPE_CHECKINGAnyDictListOptionalSequenceTupleTypeUnion)config get_option)libwriters) timezones) ArrayLike FrameOrSeriesFrameOrSeriesUnionLabelShape)import_optional_dependency) patch_pickle)PerformanceWarning)cache_readonly) ensure_objectis_categorical_dtypeis_complex_dtypeis_datetime64_dtypeis_datetime64tz_dtypeis_extension_array_dtype is_list_likeis_string_dtypeis_timedelta64_dtypeneeds_i8_conversion)array_equivalent) DataFrame DatetimeIndexIndex Int64Index MultiIndex PeriodIndexSeriesTimedeltaIndexconcatisna) Categorical DatetimeArray PeriodArray) PyTablesExprmaybe_expression) extract_array) ensure_index)stringify_path)adjoin pprint_thing)ColFileNodez0.15.2UTF-8cCst|tjr|d}|S)z* if we have bytes, decode them to unicode zUTF-8) isinstancenpZbytes_decode)srD6/tmp/pip-unpacked-wheel-q9tj5l6a/pandas/io/pytables.py_ensure_decodedSs  rFcCs|dkr t}|S)N)_default_encoding)encodingrDrDrE_ensure_encodingZsrIcCst|trt|}|S)z Ensure that an index / column name is a str (python 3); otherwise they may be np.string dtype. Non-string dtypes are passed through unchanged. https://github.com/pandas-dev/pandas/issues/13492 )r@str)namerDrDrE _ensure_strbs rL) scope_levelcsV|dt|ttfr*fdd|D}nt|r>t|d}|dksNt|rR|SdS)z Ensure that the where is a Term or a list of Term. This makes sure that we are capturing the scope of variables that are passed create the terms here with a frame_level=2 (we are 2 levels down) cs0g|](}|dk rt|r(t|ddn|qS)NrN)rM)r6Term).0Zterm)levelrDrE }sz _ensure_term..)rMN)r@listtupler6rOlen)whererMrD)rQrE _ensure_termqs    rWc@s eZdZdS)PossibleDataLossErrorN)__name__ __module__ __qualname__rDrDrDrErXsrXc@s eZdZdS)ClosedFileErrorN)rYrZr[rDrDrDrEr\sr\c@s eZdZdS)IncompatibilityWarningN)rYrZr[rDrDrDrEr]sr]z where criteria is being ignored as this version [%s] is too old (or not-defined), read the file in and write it out to a new file to upgrade (with the copy_to method) c@s eZdZdS)AttributeConflictWarningN)rYrZr[rDrDrDrEr^sr^zu the [%s] attribute of the existing index is [%s] which conflicts with the new [%s], resetting the attribute to None c@s eZdZdS)DuplicateWarningN)rYrZr[rDrDrDrEr_sr_z; duplicate entries in table, taking most recently appended z your performance may suffer as PyTables will pickle object types that it cannot map directly to c-types [inferred_type->%s,key->%s] [items->%s] fixedtable)fr`traz; : boolean drop ALL nan rows when appending to a table z~ : format default format writing format, if None, then put will default to 'fixed' and append will default to 'table' zio.hdfZ dropna_tableF)Z validatordefault_formatc Cs8tdkr4ddl}|att|jjdkaWdQRXtS)Nrstrict) _table_modtablesrAttributeErrorfileZ_FILE_OPEN_POLICY!_table_file_open_policy_is_strict)rgrDrDrE_tabless  rkaTre) keyvaluemode complevelcomplibappendformatindex min_itemsizedropna data_columnserrorsrHc s|r$ f dd}n f dd}t|}t|trzt||||d}||WdQRXn||dS)z- store this object, close it if we opened it c s|j d S)N)rsrtrunan_reprvrwrxrH)rr)store) rwrvrHrxrsrtrmruryrnrDrEszto_hdf..c s|j d S)N)rsrtruryrwrxrHrv)put)rz) rwrvrHrxrsrtrmruryrnrDrEr{s)rorprqN)r9r@rJHDFStore) path_or_bufrmrnrorprqrrrsrtruryrvrwrxrHrbrzrD) rwrvrHrxrsrtrmruryrnrEto_hdfs   rr)rorxstartstop chunksizec  Ks|dkrtd|d|dk r,t|dd}t|trN|jsDtd|} d} nvt|}t|tshtd yt j |} Wnt tfk rd} YnX| st d |d t|f||d | } d } yz|dkr&| }t|dkrtd|d}x*|ddD]}t||stdqW|j}| j||||||| | dStt tfk rt|ts~tt| WdQRXYnXdS)a Read from the store, close it if we opened it. Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path_or_buf : str, path object, pandas.HDFStore or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.h5``. If you want to pass in a path object, pandas accepts any ``os.PathLike``. Alternatively, pandas accepts an open :class:`pandas.HDFStore` object. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. key : object, optional The group identifier in the store. Can be omitted if the HDF file contains a single pandas object. mode : {'r', 'r+', 'a'}, default 'r' Mode to use when opening the file. Ignored if path_or_buf is a :class:`pandas.HDFStore`. Default is 'r'. errors : str, default 'strict' Specifies how encoding and decoding errors are to be handled. See the errors argument for :func:`open` for a full list of options. where : list, optional A list of Term (or convertible) objects. start : int, optional Row number to start selection. stop : int, optional Row number to stop selection. columns : list, optional A list of columns names to return. iterator : bool, optional Return an iterator object. chunksize : int, optional Number of rows to include in an iteration when using an iterator. **kwargs Additional keyword arguments passed to HDFStore. Returns ------- item : object The selected object. Return type depends on the object stored. See Also -------- DataFrame.to_hdf : Write a HDF file from a DataFrame. HDFStore : Low-level access to HDF files. Examples -------- >>> df = pd.DataFrame([[1, 1.0, 'a']], columns=['x', 'y', 'z']) >>> df.to_hdf('./store.h5', 'data') >>> reread = pd.read_hdf('./store.h5') )rzr+rlzmode zG is not allowed while performing a read. Allowed modes are r, r+ and a.NrN)rMz&The HDFStore must be open for reading.Fz5Support for generic buffers has not been implemented.zFile z does not exist)rorxTrz]Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file.z?key must be provided when HDF5 file contains multiple datasets.)rVrrcolumnsiteratorr auto_close) ValueErrorrWr@r}is_openOSErrorr9rJNotImplementedErrorospathexists TypeErrorFileNotFoundErrorgroupsrU_is_metadata_of _v_pathnameselectKeyErrorrrhclose)r~rmrorxrVrrrrrkwargsrzrrrZcandidate_only_groupZgroup_to_checkrDrDrEread_hdfs`S          rr>)group parent_groupreturncCsJ|j|jkrdS|}x0|jdkrD|j}||kr<|jdkrd?Z!deeeeed@dAdBZ"deeeeeedCdDdEZ#dedFdGdHZ$dee%eeee&ee'eeffeeeeeedKdLdMZ(deddNdOZ)dee%eeee&ee'eeffeeeeeedPdQdRZ*de'dSdTdUZ+deeeeedVdWdXZ,dYdZZ-dd\d]Z.eed^dd_d`Z/ee&daddbdcZ0deeeededfdgZ1ed dhdiZ2djdkZ3eedldmdnZ4dee%eee&dadpdqdrZ5dee%eeee&ee'eeffeedsdtduZ6d^dvdwdxZ7eed^dydzd{Z8ed^dd|d}Z9dS)r}aa Dict-like IO interface for storing pandas objects in PyTables. Either Fixed or Table format. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path : str File path to HDF5 file. mode : {'a', 'w', 'r', 'r+'}, default 'a' ``'r'`` Read-only; no data can be modified. ``'w'`` Write; a new file is created (an existing file with the same name would be deleted). ``'a'`` Append; an existing file is opened for reading and writing, and if the file does not exist it is created. ``'r+'`` It is similar to ``'a'``, but the file must already exist. complevel : int, 0-9, default None Specifies a compression level for data. A value of 0 or None disables compression. complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib' Specifies the compression library to be used. As of v0.20.2 these additional compressors for Blosc are supported (default if no compressor specified: 'blosc:blosclz'): {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', 'blosc:zlib', 'blosc:zstd'}. Specifying a compression library which is not available issues a ValueError. fletcher32 : bool, default False If applying compression use the fletcher32 checksum. **kwargs These parameters will be passed to the PyTables open_file method. Examples -------- >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5') >>> store['foo'] = bar # write to HDF5 >>> bar = store['foo'] # retrieve >>> store.close() **Create or load HDF5 file in-memory** When passing the `driver` option to the PyTables open_file method through **kwargs, the HDF5 file is loaded or created in-memory and will only be written when closed: >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5', driver='H5FD_CORE') >>> store['foo'] = bar >>> store.close() # only now, data is written to disk r=_handle_mode _complevel _fletcher32rlNF)rorp fletcher32cKsd|krtdtd}|dk r@||jjkr@td|jjd|dkrX|dk rX|jj}t||_|dkrnd}||_d|_|r|nd|_ ||_ ||_ d|_ |j fd|i|dS) Nrsz-format is not a defined argument for HDFStorergzcomplib only supports z compression.rlrro)rrfiltersZ all_complibsZdefault_complibr9_pathrrr_complibr_filtersopen)selfrrorprqrrrgrDrDrE__init__s$  zHDFStore.__init__cCs|jS)N)r)rrDrDrE __fspath__3szHDFStore.__fspath__cCs||jdk st|jjS)z return the root node N)_check_if_openrAssertionErrorroot)rrDrDrEr6sz HDFStore.rootcCs|jS)N)r)rrDrDrEfilename=szHDFStore.filename)rmcCs ||S)N)get)rrmrDrDrE __getitem__AszHDFStore.__getitem__cCs|||dS)N)r|)rrmrnrDrDrE __setitem__DszHDFStore.__setitem__cCs ||S)N)remove)rrmrDrDrE __delitem__GszHDFStore.__delitem__)rKc CsDy ||Sttfk r"YnXtdt|jd|ddS)z& allow attribute access to get stores 'z' object has no attribute 'N)rrr\rhtyperY)rrKrDrDrE __getattr__Js  zHDFStore.__getattr__)rmrcCs8||}|dk r4|j}||ks0|dd|kr4dSdS)zx check for existence of this key can match the exact pathname or the pathnm w/o the leading '/' NrNTF)get_noder)rrmnoderKrDrDrE __contains__Ts  zHDFStore.__contains__)rcCs t|S)N)rUr)rrDrDrE__len__`szHDFStore.__len__cCst|j}t|d|dS)Nz File path:  )r;rr)rpstrrDrDrE__repr__cs zHDFStore.__repr__cCs|S)NrD)rrDrDrE __enter__gszHDFStore.__enter__cCs |dS)N)r)rexc_type exc_value tracebackrDrDrE__exit__jszHDFStore.__exit__pandas)includercCs^|dkrdd|DS|dkrJ|jdk s0tdd|jjddd DStd |d dS) a# Return a list of keys corresponding to objects stored in HDFStore. Parameters ---------- include : str, default 'pandas' When kind equals 'pandas' return pandas objects. When kind equals 'native' return native HDF5 Table objects. .. versionadded:: 1.1.0 Returns ------- list List of ABSOLUTE path-names (e.g. have the leading '/'). Raises ------ raises ValueError if kind has an illegal value rcSsg|] }|jqSrD)r)rPnrDrDrErRsz!HDFStore.keys..ZnativeNcSsg|] }|jqSrD)r)rPrrDrDrErRs/Table) classnamez8`include` should be either 'pandas' or 'native' but is 'r)rrrZ walk_nodesr)rrrDrDrEkeysmsz HDFStore.keyscCs t|S)N)iterr)rrDrDrE__iter__szHDFStore.__iter__ccs"x|D]}|j|fVq WdS)z' iterate on key->group N)rr)rgrDrDrEitemsszHDFStore.items)rocKst}|j|krR|jdkr$|dkr$n(|dkrL|jrLtd|jd|jd||_|jr`||jr|jdkrtj|j|j|j d|_ t r|jrd }t ||j |j|jf||_d S) a9 Open the file in the specified mode Parameters ---------- mode : {'a', 'w', 'r', 'r+'}, default 'a' See HDFStore docstring or tables.open_file for info about modes **kwargs These parameters will be passed to the PyTables open_file method. )rlw)rzr+)rzRe-opening the file [z ] with mode [z] will delete the current file!r)rzGCannot open HDF5 file, which is already opened, even in read-only mode.N)rkrrrXrrrFiltersrrrrjr open_filer)rrorrgmsgrDrDrErs$   z HDFStore.opencCs|jdk r|jd|_dS)z0 Close the PyTables file handle N)rr)rrDrDrErs  zHDFStore.closecCs|jdkrdSt|jjS)zF return a boolean indicating whether the file is open NF)rboolZisopen)rrDrDrErs zHDFStore.is_open)fsyncc Cs@|jdk r<|j|rt.||}|dkr*td|d||SQRXdS)z Retrieve pandas object stored in file. Parameters ---------- key : str Returns ------- object Same type as object stored in file. NzNo object named z in the file)rrr _read_group)rrmrrDrDrErs  z HDFStore.get)rmrc st||} | dkr"td|dt|dd}|| fdd} t|| |j|||||d } | S) a Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str Object being retrieved from file. where : list or None List of Term (or convertible) objects, optional. start : int or None Row number to start selection. stop : int, default None Row number to stop selection. columns : list or None A list of columns that if not None, will limit the return columns. iterator : bool or False Returns an iterator. chunksize : int or None Number or rows to include in iteration, return an iterator. auto_close : bool or False Should automatically close the store when finished. Returns ------- object Retrieved object from file. NzNo object named z in the filerN)rMcsj|||dS)N)rrrVr)read)_start_stop_where)rrCrDrEfunc<szHDFStore.select..func)rVnrowsrrrrr)rrrW_create_storer infer_axes TableIteratorr get_result) rrmrVrrrrrrrritrD)rrCrErs&.   zHDFStore.select)rmrrcCs8t|dd}||}t|ts(td|j|||dS)a return the selection as an Index .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection rN)rMz&can only read_coordinates with a table)rVrr)rW get_storerr@rrread_coordinates)rrmrVrrtblrDrDrEselect_as_coordinatesOs    zHDFStore.select_as_coordinates)rmcolumnrrcCs,||}t|tstd|j|||dS)a~ return a single column from the table. This is generally only useful to select an indexable .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str column : str The column of interest. start : int or None, default None stop : int or None, default None Raises ------ raises KeyError if the column is not found (or key is not a valid store) raises ValueError if the column can not be extracted individually (it is part of a data block) z!can only read_column with a table)rrr)rr@rr read_column)rrmrrrrrDrDrE select_columnos#  zHDFStore.select_column)rc  s~t|dd}t|ttfr.t|dkr.|d}t|trRj||||||| dSt|ttfshtdt|sxtd|dkr|d}fdd |D |} d} xxt | |fgt |D]\\} } | dkrt d | d | jstd | jd | dkr | j} q| j| krtdqWdd D}tdd|Ddfdd}t| ||| ||||| d }|jddS)a Retrieve pandas objects from multiple tables. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- keys : a list of the tables selector : the table to apply the where criteria (defaults to keys[0] if not supplied) columns : the columns I want back start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection iterator : boolean, return an iterator, default False chunksize : nrows to include in iteration, return an iterator auto_close : bool, default False Should automatically close the store when finished. Raises ------ raises KeyError if keys or selector is not found or keys is empty raises TypeError if keys is not a list or tuple raises ValueError if the tables are not ALL THE SAME DIMENSIONS rN)rMr)rmrVrrrrrrzkeys must be a list/tuplez keys must have a non-zero lengthNcsg|]}|qSrD)r)rPk)rrDrErRsz/HDFStore.select_as_multiple..zInvalid table []zobject [z>] is not a table, and cannot be used in all select as multiplez,all tables must have exactly the same nrows!cSsg|]}t|tr|qSrD)r@r)rPxrDrDrErRscSsh|]}|jddqS)r)non_index_axes)rPrcrDrDrE sz.HDFStore.select_as_multiple..cs*fddD}t|ddS)Ncsg|]}|jdqS))rVrrr)r)rPrc)rrrrrDrErRsz=HDFStore.select_as_multiple..func..F)axisverify_integrity)r0 _consolidate)rrrobjs)rrtbls)rrrrErsz)HDFStore.select_as_multiple..func)rVrrrrrrT) coordinates)rWr@rSrTrUrJrrrr itertoolschainzipris_tablepathnamerrr)rrrVselectorrrrrrrrCrrcrZ_tblsrrrD)rrrrrEselect_as_multiples^+   "    zHDFStore.select_as_multipleTre)rmrnrprurwrx track_timesrvcCsH|dkrtdpd}||}|j||||||||| | | | | |ddS)a< Store object in HDFStore. Parameters ---------- key : str value : {Series, DataFrame} format : 'fixed(f)|table(t)', default is 'fixed' Format to use when storing object in HDFStore. Value can be one of: ``'fixed'`` Fixed format. Fast writing/reading. Not-appendable, nor searchable. ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default False This will force Table format, append the input data to the existing. data_columns : list, default None List of columns to create as data columns, or True to use all columns. See `here `__. encoding : str, default None Provide an encoding for strings. track_times : bool, default True Parameter is propagated to 'create_table' method of 'PyTables'. If set to False it enables to have the same h5 files (same hashes) independent on creation time. .. versionadded:: 1.1.0 Nzio.hdf.default_formatr`) rsrtrrrqrpruryrwrHrxrrv)r_validate_format_write_to_group)rrmrnrsrtrrrqrpruryrwrHrxrrvrDrDrEr|s$0  z HDFStore.putc Cst|dd}y||}Wn~tk r0Ynjtk rDYnVtk r}z8|dk rftd|||}|dk r|jdddSWdd}~XYnXt |||r|j jddn|j std|j |||dSdS) a= Remove pandas object partially by specifying the where condition Parameters ---------- key : string Node to remove or delete rows from where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection Returns ------- number of rows removed (or None if not a Table) Raises ------ raises KeyError if key is not a valid store rN)rMNz5trying to remove a node with a non-None where clause!T) recursivez7can only remove with where on objects written as tables)rVrr) rWrrr ExceptionrrZ _f_removecomall_nonerrdelete)rrmrVrrrCerrrrDrDrErUs,   zHDFStore.remove)rmrnrprurvrwrxcCsl| dk rtd|dkr td}|dkr4tdp2d}||}|j||||||||| | | | ||||ddS)a6 Append to Table in file. Node must already exist and be Table format. Parameters ---------- key : str value : {Series, DataFrame} format : 'table' is the default Format to use when storing object in HDFStore. Value can be one of: ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default True Append the input data to the existing. data_columns : list of columns, or True, default None List of columns to create as indexed data columns for on-disk queries, or True to use all columns. By default only the axes of the object are indexed. See `here `__. min_itemsize : dict of columns that specify minimum str sizes nan_rep : str to use as str nan representation chunksize : size to chunk the writing expectedrows : expected TOTAL row size of this table encoding : default None, provide an encoding for str dropna : bool, default False Do not write an ALL nan row to the store settable by the option 'io.hdf.dropna_table'. Notes ----- Does *not* check if data being appended overlaps with existing data in the table, so be careful Nz>columns is not a supported keyword in append, try data_columnszio.hdf.dropna_tablezio.hdf.default_formatra)rsaxesrtrrrqrpruryr expectedrowsrvrwrHrx)rrrr)rrmrnrsrrtrrrqrprruryrrrvrwrHrxrDrDrErrs28  zHDFStore.append)dc s|dk rtdt|ts"td||kr2tdtttjttt d}d} g} x<| D]0\} dkr| dk rtd| } qj| qjW| dk rڈj |} | t| } t| | } | | || <|dkr||}|r2fdd|D}t|}x|D]}||}qWj||d d}xt| D]h\} | |kr^|nd}j|d }|dk rfd d | Dnd}|j| |f||d |qHWdS)a Append to multiple tables Parameters ---------- d : a dict of table_name to table_columns, None is acceptable as the values of one node (this will get all the remaining columns) value : a pandas object selector : a string that designates the indexable table; all of its columns will be designed as data_columns, unless data_columns is passed, in which case these are used data_columns : list of columns to create as data columns, or True to use all columns dropna : if evaluates to True, drop rows from all tables if any single row in each table has all NaN. Default False. Notes ----- axes parameter is currently not accepted Nztaxes is currently not accepted as a parameter to append_to_multiple; you can create the tables independently insteadzQappend_to_multiple must have a dictionary specified as the way to split the valuez=append_to_multiple requires a selector that is in passed dictrz.sz.HDFStore.append_to_multiple..ru)rcsi|]\}}|kr||qSrDrD)rPrmrn)vrDrE >sz/HDFStore.append_to_multiple..)rwru)rr@dictrrSsetrangendim _AXES_MAPrrextendr differencer*sorted get_indexertakevaluesnext intersectionlocpopreindexrr)rrrnrrwrrvrrZ remain_keyZ remain_valuesrorderedZorddZidxsZ valid_indexrtrudcvalfilteredrD)r rnrEappend_to_multiplesN &     zHDFStore.append_to_multiple)rmoptlevelkindcCsBt||}|dkrdSt|ts.td|j|||ddS)a Create a pytables index on the table. Parameters ---------- key : str columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError: raises if the node is not a table Nz1cannot create table index on a Fixed format store)rr#r$)rkrr@rr create_index)rrmrr#r$rCrDrDrEcreate_table_indexDs  zHDFStore.create_table_indexcCs<t||jdk sttdk s(tdd|jDS)z Return a list of all the top-level nodes. Each node returned is not a pandas storage object. Returns ------- list List of objects. NcSsPg|]H}t|tjjst|jddsHt|ddsHt|tjjr|jdkr|qS) pandas_typeNra) r@rflinkLinkgetattr_v_attrsrarr)rPrrDrDrErR|s  z#HDFStore.groups..)rkrrrrf walk_groups)rrDrDrErls  zHDFStore.groupsrccst||jdk sttdk s(tx|j|D]}t|jdddk rNq6g}g}xP|j D]B}t|jdd}|dkrt |tj j r| |jqb| |jqbW|jd||fVq6WdS)au Walk the pytables group hierarchy for pandas objects. This generator will yield the group path, subgroups and pandas object names for each group. Any non-pandas PyTables objects that are not a group will be ignored. The `where` group itself is listed first (preorder), then each of its child groups (following an alphanumerical order) is also traversed, following the same procedure. .. versionadded:: 0.24.0 Parameters ---------- where : str, default "/" Group where to start walking. Yields ------ path : str Full path to a group (without trailing '/'). groups : list Names (strings) of the groups contained in `path`. leaves : list Names (strings) of the pandas objects contained in `path`. Nr'r)rkrrrrfr,r*r+Z _v_childrenrr@rGrouprrrrrstrip)rrVrrZleaveschildr'rDrDrEwalks  z HDFStore.walkr>cCs~||dsd|}|jdk s(ttdk s4ty|j|j|}Wntjjk r`dSXt |tj sztt ||S)z; return the node with the key or None if it does not exist rN) r startswithrrrfrr exceptionsZNoSuchNodeErrorr@r>r)rrmrrDrDrErs  zHDFStore.get_node) GenericFixedrcCs8||}|dkr"td|d||}||S)z> return the storer object for a key, raise if not in the file NzNo object named z in the file)rrrr)rrmrrCrDrDrErs   zHDFStore.get_storerr) propindexesrprc  Cst|||||d} |dkr&t|}t|ttfs:|g}x|D]} || } | dk r@| | krl|rl| | || } t| trd} |rdd| j D} | j | | | t | dd| j dq@| j | | | j dq@W| S) a; Copy the existing store to a new file, updating in place. Parameters ---------- propindexes : bool, default True Restore indexes in copied file. keys : list, optional List of keys to include in the copy (defaults to all). overwrite : bool, default True Whether to overwrite (remove and replace) existing nodes in the new store. mode, complib, complevel, fletcher32 same as in HDFStore.__init__ Returns ------- open file handle of the new store )rorqrprNFcSsg|]}|jr|jqSrD) is_indexedrK)rPrlrDrDrErRsz!HDFStore.copy..rw)rtrwrH)rH)r}rSrr@rTrrrrrrrr*rHr|)rriror4rrqrpr overwriteZ new_storerrCdatartrDrDrEcopys2        z HDFStore.copyc Cst|j}t|d|d}|jrt|}t|rg}g}x|D]}y<||}|dk r|t|j pn||t|pdWqHt k rYqHt k r}z(||t|} |d| dWdd}~XYqHXqHW|t d||7}n|d7}n|d 7}|S) zg Print detailed information on the store. Returns ------- str z File path: rNzinvalid_HDFStore nodez[invalid_HDFStore node: r EmptyzFile is CLOSED) r;rrrrrrUrrrrrrr:) rroutputZlkeysrrrrCZdetailZdstrrDrDrEinfos.     ( z HDFStore.infocCs|jst|jddS)Nz file is not open!)rr\r)rrDrDrEr;szHDFStore._check_if_open)rsrc CsJyt|}Wn4tk rD}ztd|d|Wdd}~XYnX|S)z validate / deprecate formats z#invalid HDFStore format specified [rN) _FORMAT_MAPlowerrr)rrsrrDrDrEr?s $zHDFStore._validate_formatUTF-8)rnrHrxrc s"dk rtttfstdfdd}ttjdd}ttjdd}|dkrʈdkrttdk stt tddsttj j rd}d }qtd n(td td i} | t }dkr|d 7}d|kr*t td} y | |} Wn.tk r} z|d| Wdd} ~ XYnX| |||dS|dkrĈdk r|dkrtdd} | dk r| jdkrpd}n| jdkrd}nB|dkrtdd} | dk r| jdkrd}n| jdkrd}ttttttd}y ||} Wn.tk r} z|d| Wdd} ~ XYnX| |||dS)z$ return a suitable class to operate Nz(value must be None, Series, or DataFramec s$td|ddtdS)Nz(cannot properly create the storer for: [z ] [group->z,value->z ,format->)rr)rc)rsrrnrDrEerrorWsz&HDFStore._create_storer..errorr' table_typera frame_table generic_tablezKcannot create a storer if the object is not existing nor a value are passedseriesframeZ_table)rDrE _STORER_MAP)rHrx series_tablertrNappendable_seriesappendable_multiseriesappendable_frameappendable_multiframe)rCrHrIrJrKworm _TABLE_MAP)r@r.r(rrFr*r+rkrfrrarr SeriesFixed FrameFixedrnlevels GenericTableAppendableSeriesTableAppendableMultiSeriesTableAppendableFrameTableAppendableMultiFrameTable WORMTable)rrrsrnrHrxr@ptttZ _TYPE_MAPrFclsrrtrMrD)rsrrnrErIsj                     zHDFStore._create_storer)rmrnrprurxrcCst|ddr|dks|rdS|||}|j|||||d}|rr|jrZ|jrb|dkrb|jrbtd|jsz|n||js|rtd|j|||||| | | | | |||d t|t r|r|j |ddS) Nemptyra)rHrxr`zCan only append to Tablesz0Compression not supported on Fixed format stores) objrrrrqrprrurrrvryrwr)r) r*_identify_grouprr is_existsrset_object_infowriter@rr%)rrmrnrsrrtrrrqrprrurrrvryrwrHrxrrrCrDrDrErs8   zHDFStore._write_to_group)rcCs||}||S)N)rrr)rrrCrDrDrErs zHDFStore._read_group)rmrrrcCsN||}|jdk st|dk r8|s8|jj|ddd}|dkrJ||}|S)z@Identify HDF5 group based on key, delete/create group if needed.NT)r)rrr remove_node_create_nodes_and_group)rrmrrrrDrDrEr\s   zHDFStore._identify_groupcCsz|jdk st|d}d}xX|D]P}t|s0q"|}|dsF|d7}||7}||}|dkrn|j||}|}q"W|S)z,Create nodes from key and return group name.Nr)rrsplitrUendswithrZ create_group)rrmpathsrpnew_pathrrDrDrEras    z HDFStore._create_nodes_and_group)rlNNF)r)rl)F)NNNNFNF)NNN)NN)NNNNNFNF) NTFNNNNNNreTF)NNN)NNTTNNNNNNNNNNre)NNF)NNN)r)rTNNNFT)NNr?re)NTFNNNNNNFNNNreT):rYrZr[__doc__r __annotations__rJintrrrpropertyrrrrrrrrrrrr rrr iteritemsrrrrrrrrrrrrr|rrrr"r&rr0rrr8r<rrrrrr\rarDrDrDrEr}s @     "-E&t>5=@H]# 2 4*  X2+r}c@sreZdZUdZeeed<eed<eded<deede eee dd d Z d d Z d dZ de dddZ dS)raa Define the iteration interface on a table Parameters ---------- store : HDFStore s : the referred storer func : the function to execute the query where : the where of the query nrows : the rows to iterate on start : the passed start value (default is None) stop : the passed stop value (default is None) iterator : bool, default False Whether to use the default iterator. chunksize : the passed chunking value (default is 100000) auto_close : bool, default False Whether to automatically close the store at the end of iteration. rrz)r3rrCNF)rzrCrrrc Cs||_||_||_||_|jjrN|dkr,d}|dkr8d}|dkrD|}t||}||_||_||_d|_ |sr| dk r| dkr~d} t | |_ nd|_ | |_ dS)Nri) rzrCrrVrminrrrrrirr) rrzrCrrVrrrrrrrDrDrEr&s,    zTableIterator.__init__ccsz|j}|jdkrtdxT||jkrlt||j|j}|dd|j||}|}|dkst|sdq|VqW|dS)Nz*Cannot iterate until get_result is called.) rrrrrlrrrUr)rrrrnrDrDrErPs   zTableIterator.__iter__cCs|jr|jdS)N)rrzr)rrDrDrEr`szTableIterator.close)rcCs|jdk r4t|jtstd|jj|jd|_|S|rft|jtsLtd|jj|j|j|j d}n|j}| |j|j |}| |S)Nz0can only use an iterator or chunksize on a table)rVz$can only read_coordinates on a table)rVrr) rr@rCrrrrVrrrrr)rrrVresultsrDrDrErds   zTableIterator.get_result)NNFNF)F)rYrZr[rgr rirhr}rrrrrrrDrDrDrErs    rc @s|eZdZUdZdZdZdddgZeed<eed<dDee ed d d Z e e d d dZ e ed ddZe dddZed ddZeedddZed ddZe ed ddZejeedddZd d!Ze d"d#Ze d$d%Ze d&d'Ze d(d)Zd*d+ZdEd,d-Zd.d/Z d0ed1d2d3Z!dFd4d5Z"ed6d7d8Z#d9d:Z$d;d<Z%d=d>Z&d0d?d@dAZ'd0d?dBdCZ(dS)GIndexCola an index column description class Parameters ---------- axis : axis which I reference values : the ndarray like converted values kind : a string description of this type typ : the pytables type pos : the position in the pytables Tfreqtz index_namerKcnameN)rKrrcCst|tstd||_||_||_||_|p0||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_|dk r|||t|jtstt|jtstdS)Nz`name` must be a str.)r@rJrrr$typrKrrrposrorprqrrarmetadataset_posr)rrKrr$rsrrrrtrorprqrrarrurDrDrErs(   zIndexCol.__init__)rcCs|jjS)N)rsitemsize)rrDrDrErwszIndexCol.itemsizecCs |jdS)N_kind)rK)rrDrDrE kind_attrszIndexCol.kind_attr)rtcCs$||_|dk r |jdk r ||j_dS)z. set the position of this column in the Table N)rtrsZ_v_pos)rrtrDrDrErvszIndexCol.set_posc CsFttt|j|j|j|j|jf}dddt dddddg|DS) N,css |]\}}|d|VqdS)z->NrD)rPrmrnrDrDrEr sz$IndexCol.__repr__..rKrrrrtr$) rTmapr;rKrrrrtr$joinr)rtemprDrDrErs  zIndexCol.__repr__)otherrcstfdddDS)z compare 2 col items c3s&|]}t|dt|dkVqdS)N)r*)rPrl)r~rrDrEr sz"IndexCol.__eq__..)rKrrrrt)r)rr~rD)r~rrE__eq__s zIndexCol.__eq__cCs || S)N)r)rr~rDrDrE__ne__szIndexCol.__ne__cCs"t|jdsdSt|jj|jjS)z' return whether I am an indexed column r F)hasattrrar*r rrr5)rrDrDrEr5s zIndexCol.is_indexed)rrHrxcCst|tjstt||jjdk r.||j}t|j }t ||||}i}t|j |d<|j dk rpt|j |d<yt |f|}Wn0tk rd|krd|d<t |f|}YnXt||j}||fS)zV Convert the data from this selection to the appropriate pandas type. NrKro)r@rAndarrayrrdtypefieldsrrrFr$_maybe_convertrqror*r_set_tzrp)rrryrHrxval_kindrZ new_pd_indexrDrDrEconverts"     zIndexCol.convertcCs|jS)z return the values)r)rrDrDrE take_data szIndexCol.take_datacCs|jjS)N)rar+)rrDrDrEattrsszIndexCol.attrscCs|jjS)N)ra description)rrDrDrErszIndexCol.descriptioncCst|j|jdS)z# return my current col description N)r*rrr)rrDrDrEcolsz IndexCol.colcCs|jS)z return my cython values )r)rrDrDrEcvaluesszIndexCol.cvaluescCs t|jS)N)rr)rrDrDrEr!szIndexCol.__iter__cCsPt|jdkrLt|tr$||j}|dk rL|jj|krLtj ||j d|_dS)z maybe set a string col itemsize: min_itemsize can be an integer or a dict with this columns name with an integer size stringN)rwrt) rFr$r@rrrKrsrwrk StringColrt)rrurDrDrEmaybe_set_size$s   zIndexCol.maybe_set_sizecCsdS)NrD)rrDrDrEvalidate_names1szIndexCol.validate_namesAppendableTable)handlerrrcCs:|j|_||||||||dS)N)ra validate_col validate_attrvalidate_metadatawrite_metadataset_attr)rrrrrDrDrEvalidate_and_set4s    zIndexCol.validate_and_setcCs^t|jdkrZ|j}|dk rZ|dkr*|j}|j|krTtd|d|jd|jd|jSdS)z< validate this column: return the compared against itemsize rNz#Trying to store a string with len [z] in [z)] column but this column has a limit of [zC]! Consider using min_itemsize to preset the sizes on these columns)rFr$rrwrrr)rrwcrDrDrEr<s zIndexCol.validate_col)rrcCsB|r>t|j|jd}|dk r>||jkr>td|d|jddS)Nzincompatible kind in col [z - r)r*rryr$r)rrrZ existing_kindrDrDrErOs zIndexCol.validate_attrc Csx|jD]}t||d}||ji}||}||kr|dk r||kr|dkrt|||f}tj|tddd||<t ||dqt d|jd|d|d|d q|dk s|dk r|||<qWdS) z set/update the info for this indexable with the key/value if there is a conflict raise/warn as needed N)rorq) stacklevelzinvalid info for [z] for [z], existing_value [z] conflicts with new value [r) _info_fieldsr* setdefaultrKrattribute_conflict_docwarningswarnr^setattrr)rr<rmrnidxZexisting_valuewsrDrDrE update_infoXs   $zIndexCol.update_infocCs$||j}|dk r |j|dS)z# set my state from the passed info N)rrK__dict__update)rr<rrDrDrEset_infows zIndexCol.set_infocCst|j|j|jdS)z set the kind for this column N)rrryr$)rrDrDrEr}szIndexCol.set_attr)rcCsB|jdkr>|j}||j}|dk r>|dk r>t||s>tddS)z< validate that kind=category does not change the categories categoryNzEcannot append a categorical with different categories to the existing)rru read_metadatarrr'r)rrZ new_metadataZ cur_metadatarDrDrErs   zIndexCol.validate_metadatacCs|jdk r||j|jdS)z set the meta data N)rurrr)rrrDrDrErs zIndexCol.write_metadata) NNNNNNNNNNNNN)N)N))rYrZr[rgis_an_indexableis_data_indexablerrJrhr rrjrirwryrvrrrrrr5rArrrrrrrrrrrrrrrrrrrDrDrDrErn~s\           rnc@s>eZdZdZeedddZeje e dddZ dd Z d S) GenericIndexColz< an index which is not represented in the data of the table )rcCsdS)NFrD)rrDrDrEr5szGenericIndexCol.is_indexed)rrHrxcCs2t|tjstt|ttt|}||fS)z Convert the data from this selection to the appropriate pandas type. Parameters ---------- values : np.ndarray nan_rep : str encoding : str errors : str )r@rArrrr+arangerU)rrryrHrxrDrDrErs zGenericIndexCol.convertcCsdS)NrD)rrDrDrErszGenericIndexCol.set_attrN) rYrZr[rgrjrr5rArrJrrrDrDrDrErs rc s0eZdZdZdZdZddgZd2edfdd Ze ed d d Z e ed d d Z ed ddZ e edddZedddZddZeeddddZeddZeeedddd Zeeddd!d"Zed#d$Zed%d&Ze d'd(Ze d)d*Zd+d,Zejeed-d.d/Z d0d1Z!Z"S)3DataCola3 a data holding column, by definition this is not indexable Parameters ---------- data : the actual data cname : the column name in the table to hold the data (typically values) meta : a string description of the metadata metadata : the actual metadata FrprN)rKc s2tj||||||||| | | d | |_| |_dS)N) rKrr$rsrtrrrprrarru)superrrr7)rrKrr$rsrrrtrprrarrurr7) __class__rDrErszDataCol.__init__)rcCs |jdS)N_dtype)rK)rrDrDrE dtype_attrszDataCol.dtype_attrcCs |jdS)N_meta)rK)rrDrDrE meta_attrszDataCol.meta_attrc CsFttt|j|j|j|j|jf}dddt dddddg|DS) Nrzcss |]\}}|d|VqdS)z->NrD)rPrmrnrDrDrEr sz#DataCol.__repr__..rKrrrr$shape) rTr{r;rKrrrr$rr|r)rr}rDrDrErs zDataCol.__repr__)r~rcstfdddDS)z compare 2 col items c3s&|]}t|dt|dkVqdS)N)r*)rPrl)r~rrDrEr sz!DataCol.__eq__..)rKrrrrt)r)rr~rD)r~rrErs zDataCol.__eq__)r7cCs@|dk s t|jdkstt|\}}||_||_t||_dS)N)rr_get_data_and_dtype_namer7_dtype_to_kindr$)rr7 dtype_namerDrDrEset_datas   zDataCol.set_datacCs|jS)z return the data )r7)rrDrDrEr szDataCol.take_datar<)rrcCs|j}|j}|j}|jdkr&d|jf}t|trJ|j}|j||jj d}ntt |sZt |rf| |}nXt |rz||}nDt|rtj||dd}n&t|r|||}n|j||j d}|S)zW Get an appropriately typed and shaped pytables.Col object for values. rN)r$r)rwr)rrwrrsizer@r2codes get_atom_datarKr r!get_atom_datetime64r%get_atom_timedelta64rrkZ ComplexColr$get_atom_string)rYrrrwrratomrDrDrE _get_atom s$     zDataCol._get_atomcCstj||ddS)Nr)rwr)rkr)rYrrwrDrDrEr+ szDataCol.get_atom_string)r$rcCsR|dr$|dd}d|d}n"|dr4d}n|}|d}tt|S)z2 return the PyTables column class for this column uintNZUIntr<periodInt64Col)r1 capitalizer*rk)rYr$Zk4Zcol_nameZkcaprDrDrEget_atom_coltype/ s    zDataCol.get_atom_coltypecCs|j|d|ddS)N)r$r)r)r)rYrr$rDrDrEr> szDataCol.get_atom_datacCstj|ddS)Nr)r)rkr)rYrrDrDrErB szDataCol.get_atom_datetime64cCstj|ddS)Nr)r)rkr)rYrrDrDrErF szDataCol.get_atom_timedelta64cCst|jddS)Nr)r*r7)rrDrDrErJ sz DataCol.shapecCs|jS)z return my cython values )r7)rrDrDrErN szDataCol.cvaluescCs`|r\t|j|jd}|dk r2|t|jkr2tdt|j|jd}|dk r\||jkr\tddS)zAvalidate that we have the same order as the existing & same dtypeNz4appended items do not match existing items in table!z@appended items dtype do not match existing items dtype in table!)r*rryrSrrrr)rrrZexisting_fieldsZexisting_dtyperDrDrErS szDataCol.validate_attr)rrHrxcCst|tjstt||jjdk r.||j}|jdk s.cSsg|]}t|qSrD)r fromtimestamp)rPr rDrDrErR sr) categoriesrF)r8Or)ryrHrx)#r@rArrrrrrrrsrrr$rFrrurrprasarrayobjectrravelr*Zfloat64r1anyastyperiZcumsum_valuesr2Z from_codesr_unconvert_string_arrayr)rrryrHrx convertedrr$rrurrprrrmaskrDrDrEr` s^             zDataCol.convertcCsHt|j|j|jt|j|j|j|jdk s2tt|j|j|jdS)z set the data for this column N) rrryrrrrrr)rrDrDrEr szDataCol.set_attr) NNNNNNNNNNNN)#rYrZr[rgrrrrJrrjrrrrrrrrr classmethodrrr rrrrrrrrArrr __classcell__rDrD)rrErsL         erc@sTeZdZdZdZddZeddZeeddd d Z ed d Z ed dZ dS)DataIndexableColz- represent a data column that can be indexed TcCst|jstddS)Nz-cannot have non-object label DataIndexableCol)r*r is_objectr)rrDrDrEr szDataIndexableCol.validate_namescCstj|dS)N)rw)rkr)rYrrwrDrDrEr sz DataIndexableCol.get_atom_stringr<)r$rcCs|j|dS)N)r$)r)rYrr$rDrDrEr szDataIndexableCol.get_atom_datacCs tS)N)rkr)rYrrDrDrEr sz$DataIndexableCol.get_atom_datetime64cCs tS)N)rkr)rYrrDrDrEr sz%DataIndexableCol.get_atom_timedelta64N) rYrZr[rgrrrrrJrrrrDrDrDrEr s  rc@seZdZdZdS)GenericDataIndexableColz* represent a generic pytables data column N)rYrZr[rgrDrDrDrEr src@seZdZUdZeed<dZeed<eeed<e ed<eed<e ed<d ed <eed <d Z dEe d eedddZ e edddZe ee e e fdddZe ddZedddZddZddZe dd Ze d!d"Ze d#d$Ze d%d&Ze e dd'd(Ze edd)d*Ze d+d,Zd-d.Zd/d0Ze d1d2Ze edd3d4Z e d5d6Z!d7d8Z"dFd:d;Z#dd?d@Z&dAdBZ'dHe%e e%e d>dCdDZ(d9S)IFixedz represent an object in my store facilitate read/write of various types of objects this is an abstract base class Parameters ---------- parent : HDFStore group : Node The group node where the table resides. pandas_kindr` format_typeobj_typerrHrr>rrxFUTF-8re)rrrHrxcCsZt|tstt|tdk s"tt|tjs:tt|||_||_t||_ ||_ dS)N) r@r}rrrfr>rrrIrHrx)rrrrHrxrDrDrEr s  zFixed.__init__)rcCs*|jddko(|jddko(|jddkS)NrrN )version)rrDrDrEis_old_version szFixed.is_old_versioncCsbtt|jjdd}y0tdd|dD}t|dkrB|d}Wntk r\d}YnX|S) z compute and set our version pandas_versionNcss|]}t|VqdS)N)ri)rPrrDrDrEr  sz Fixed.version...r)r)rrr)rFr*rr+rTrbrUrh)rrrDrDrEr s   z Fixed.versioncCstt|jjddS)Nr')rFr*rr+)rrDrDrEr'$ szFixed.pandas_typecCs^||j}|dk rXt|ttfrDddd|D}d|d}|jdd|d S|jS) z* return a pretty representation of myself Nrzcss|]}t|VqdS)N)r;)rPrrDrDrEr . sz!Fixed.__repr__..[rz12.12z (shape->))rrr@rSrTr|r')rrCZjshaperDrDrEr( s zFixed.__repr__cCst|j|j_tt|j_dS)z set my pandas type & version N)rJrrr'_versionr)rrDrDrEr^3 szFixed.set_object_infocCst|}|S)N)r8)rZnew_selfrDrDrEr88 s z Fixed.copycCs|jS)N)r)rrDrDrEr< sz Fixed.shapecCs|jjS)N)rr)rrDrDrEr@ szFixed.pathnamecCs|jjS)N)rr)rrDrDrErD sz Fixed._handlecCs|jjS)N)rr)rrDrDrErH szFixed._filterscCs|jjS)N)rr)rrDrDrErL szFixed._complevelcCs|jjS)N)rr)rrDrDrErP szFixed._fletcher32cCs|jjS)N)rr+)rrDrDrErT sz Fixed.attrscCsdS)z set our object attributes NrD)rrDrDrE set_attrsX szFixed.set_attrscCsdS)z get our object attributes NrD)rrDrDrE get_attrs\ szFixed.get_attrscCs|jS)z return my storable )r)rrDrDrEstorable` szFixed.storablecCsdS)NFrD)rrDrDrEr]e szFixed.is_existscCst|jddS)Nr)r*r)rrDrDrEri sz Fixed.nrowscCs|dkr dSdS)z' validate against an existing storable NTrD)rr~rDrDrEvalidatem szFixed.validateNcCsdS)z- are we trying to operate on an old version? TrD)rrVrDrDrEvalidate_versions szFixed.validate_versioncCs|j}|dkrdS|dS)zr infer the axes of my storer return a boolean indicating if we have a valid storer or not NFT)rr)rrCrDrDrErw s zFixed.infer_axes)rrcCs tddS)Nz>cannot read on an abstract storer: subclasses should implement)r)rrVrrrrDrDrEr sz Fixed.readcKs tddS)Nz?cannot write on an abstract storer: subclasses should implement)r)rrrDrDrEr_ sz Fixed.writecCs0t|||r$|jj|jdddStddS)zs support fully deleting the node in its entirety (only) - where specification must be None T)rNz#cannot delete on an abstract storer)rrrr`rr)rrVrrrDrDrEr sz Fixed.delete)rre)N)NNNN)NNN))rYrZr[rgrJrhrr rrir}rrrjrrr rr'rr^r8rrrrrrrrrrr]rrrrr rr_rrDrDrDrEr sX               rc@sNeZdZUdZedediZddeDZgZ e e e d<e ddd Z d d Zd d ZddZeedddZddZddZddZd0e eeeedddZd1e eeeeedddZe edd d!Ze edd"d#Zd2e eeeeedd$d%Zd3d&eeeeed'd(d)Z e e!d*d+d,Z"d4e e#eed-d.d/Z$dS)5r3z a generified fixed version datetimercCsi|]\}}||qSrDrD)rPrr rDrDrEr  szGenericFixed. attributes)rcCs|j|dS)N)_index_type_mapr)rrYrDrDrE_class_to_alias szGenericFixed._class_to_aliascCst|tr|S|j|tS)N)r@r_reverse_index_maprr*)raliasrDrDrE_alias_to_class s zGenericFixed._alias_to_classcCs0|tkrddd}|S|tkr,ddd}|S|S)NcSs:tj|j|d}tj|dd}|dk r6|d|}|S)N)ro)rKUTC)r3 _simple_newrr) tz_localize tz_convert)rrorpZdtaresultrDrDrErb s z*GenericFixed._get_index_factory..fcSstj||d}tj|ddS)N)ro)rK)r4rr-)rrorpZparrrDrDrErb s)NN)NN)r)r-)rklassrbrDrDrE_get_index_factory s  zGenericFixed._get_index_factorycCs$|dk rtd|dk r tddS)zE raise if any keywords are passed which are not-None Nzqcannot pass a column specification when reading a Fixed format store. this store must be selected in its entiretyzucannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety)r)rrrVrDrDrE validate_read s zGenericFixed.validate_readcCsdS)NTrD)rrDrDrEr] szGenericFixed.is_existscCs|j|j_|j|j_dS)z set our object attributes N)rHrrx)rrDrDrEr s zGenericFixed.set_attrsc CsVtt|jdd|_tt|jdd|_x(|jD]}t||tt|j|dq0WdS)z retrieve our attributes rHNrxre)rIr*rrHrFrxrr)rrrDrDrEr s zGenericFixed.get_attrscKs |dS)N)r)rr[rrDrDrEr_ szGenericFixed.writeN)rmrrc Csddl}t|j|}|j}t|dd}t||jrD|d||}nztt|dd} t|dd} | dk rxtj| | d}n |||}| dkrt|d d} t || d d }n| d krtj |d d}|r|j S|SdS)z4 read an array for the specified node (off of group rN transposedF value_typer)rrrpT)rrzm8[ns]) rgr*rr+r@ZVLArrayrFrArZrrT) rrmrrrgrrrretrrrprDrDrE read_array s&      zGenericFixed.read_array)rmrrrcCshtt|j|d}|dkr.|j|||dS|dkrVt|j|}|j|||d}|Std|dS)N_varietymulti)rrregularzunrecognized index variety: )rFr*rread_multi_indexrread_index_noder)rrmrrZvarietyrrtrDrDrE read_index s zGenericFixed.read_index)rmrtcCst|tr,t|j|dd|||nt|j|ddtd||j|j}|||j t |j |}|j |j _ |j|j _t|ttfr|t||j _t|tttfr|j|j _t|tr|jdk rt|j|j _dS)Nrrr rt)r@r,rrwrite_multi_index_convert_indexrHrx write_arrayrr*rr$r+rKr)r-rr index_classr/rorp_get_tz)rrmrtrrrDrDrE write_index s     zGenericFixed.write_indexc Cst|j|d|jxtt|j|j|jD]\}\}}}t|rLt d|d|}t |||j |j }| ||jt|j|} |j| j_|| j_t| j|d|||d|} | | |q.WdS)N_nlevelsz=Saving a MultiIndex with an extension dtype is not supported._level_name_label)rrrP enumeraterlevelsrnamesr"rrrHrxrrr*rr$r+rK) rrmrtilev level_codesrK level_keyZ conv_levelr label_keyrDrDrEr 2 s"  zGenericFixed.write_multi_indexcCst|j|d}g}g}g}xxt|D]l}|d|} t|j| } |j| ||d} || || j|d|} |j| ||d} || q(Wt|||ddS)Nrr)rrrT)rrrr) r*rrrr rrrKrr,)rrmrrrPrrrrrrrrrrDrDrEr K s   zGenericFixed.read_multi_indexr>)rrrrc Cs>|||}d|jkr>t|jjdkr>tj|jj|jjd}t|jj}d}d|jkrlt|jj }t|}| tt |jdd}| |}i} d|jkr|jd| d<d|jkrt |jdtr|jdd | d<n|jd| d<|d kr|t|||j|jd fd ti| } n|t|||j|jd f| } || _ | S) Nrr)rrKrrrorpzutf-8r)rHrxr)r+rAprodrrZrrFr$rLrKrr*rr@bytesrB_unconvert_indexrHrxr) rrrrr7r$rKrfactoryrrtrDrDrEr b s<         zGenericFixed.read_index_node)rmrncCsJtd|j}|j|j||t|j|}t|j|j _ |j |j _ dS)z write a 0-len array )rNN) rArZrr create_arrayrr*rJrr+rr)rrmrnZarrrrDrDrEwrite_array_empty s  zGenericFixed.write_array_empty)rmr[rc Cs2t|dd}||jkr&|j|j||jdk}d}t|jrFtd|s^t|dr^|j }d}d}|j dk rt t t j|j}WdQRX|dk r|s|jj|j|||j|j d}||dd<n |||nH|jjtjkrHtj|dd} |rn*| d krnt| ||f} tj| td d |j|j|t } | |nt|jr~|j |j||!d d t"|j|j#_$nt%|jr|j |j||j&t"|j|} t'|j(| j#_(d | j#_$n\t)|jr|j |j||!d dt"|j|j#_$n&|r |||n|j |j|||t"|j|j#_*dS)NT)Z extract_numpyrFz]Cannot store a category dtype in a HDF5 dataset that uses format="fixed". Use format="table".r)r)skipnar)ri8rr)+r7rrr`rrrrrrrrrrkZAtomZ from_dtypeZ create_carrayrr$rrAZobject_r infer_dtypeperformance_docrrrZcreate_vlarray ObjectAtomrrr r#viewr*r+rr!asi8rrpr%r) rrmr[rrnZ empty_arrayrrca inferred_typerZvlarrrrDrDrEr s^              zGenericFixed.write_array)NN)NN)NN)NN)N)%rYrZr[rgr)r-rrrrr rJrhrrrrrjrr]rrr_r rirr*r rr,r r r rr$rrrDrDrDrEr3 s0  $ 2 r3csVeZdZUdZdgZeed<eddZd e e e e dddZ fd d Z Z S) rNrDrKc Cs,yt|jjfSttfk r&dSXdS)N)rUrrrrh)rrDrDrEr szSeriesFixed.shapeN)rrcCs<||||jd||d}|jd||d}t|||jdS)Nrt)rrr)rtrK)rr rr.rK)rrVrrrrtrrDrDrEr s zSeriesFixed.readc s8tj|f||d|j|d||j|j_dS)Nrtr)rr_rrtrrKr)rr[r)rrDrEr_ s zSeriesFixed.write)NNNN)rYrZr[rrrrhrjrr rirr_rrDrD)rrErN s  rNcs^eZdZUddgZeed<eeedddZ d eeeeddd Z fd d Z Z S) BlockManagerFixedrnblocks)rcCsy|j}d}xFt|jD]8}t|jd|d}t|dd}|dk r||d7}qW|jj}t|dd}|dk rt|d|d}ng}|||Stk rdSXdS)Nrblock_itemsrrN) rrr0r*rZ block0_valuesrSrrrh)rrrrrrrDrDrEr s"   zBlockManagerFixed.shapeN)rrcCs ||||d}g}xJt|jD]<}||kr>||fnd\}} |jd||| d} || q*W|d} g} xht|jD]Z}|d|d} |jd|d|| d}| | | }t |j ||dd }| |qWt | dkr t | dd }|j| d d }|St |d|dd S) Nr)NNr)rrr1r2rrN)rrt)rF)rr8)rrZ_get_block_manager_axisrrr rrr0rrr(rrUr0r)rrVrrrZ select_axisrrrraxrdfs blk_itemsrdfoutrDrDrEr1 s(  zBlockManagerFixed.readc stj|f||j}|s&|}|j|j_x>t|jD]0\}}|dkrZ|j sZt d| d||qd?d@Z,ee-j.dAdBdCZ/edDdEdFZ0dGdHZ1dIdJZ2dodKdLZ3dMdNZ4e5dOdPZ6dpe7edQdRdSZ8dqe7e e7e e ee9e9fdTdUdVZ:e;edWdXdYZe?d_d`Z@dsdadbdcddZAe7e ee7e eeefdedfdgZBdte7e e7e dhdidjZCduee7e e7e dkdldmZDZES)vra represent a table: facilitate read/write of various types of tables Attrs in Table Node ------------------- These are attributes that are store in the main table node, they are necessary to recreate these tables when read back in. index_axes : a list of tuples of the (original indexing axis and index column) non_index_axes: a list of tuples of the (original index axis and columns on a non-indexing axis) values_axes : a list of the columns which comprise the data of this table data_columns : a list of the columns that we are allowing indexing (these become single columns in values_axes), or True to force all columns nan_rep : the string to use for nan representations for string objects levels : the names of levels metadata : the names of the metadata columns Z wide_tablerarrArNrT index_axesr values_axesrwrur<Nrer>)rrrxc sPtj||||d|pg|_|p$g|_|p.g|_|p8g|_| pBi|_| |_dS)N)rHrx)rrr<rr=rwr<ry) rrrrHrxr<rr=rwr<ry)rrDrEr s      zTable.__init__)rcCs|jddS)N_r)rArb)rrDrDrEtable_type_short szTable.table_type_shortc Cs|t|jrd|jnd}d|d}d}|jrZddd|jD}d|d}dd d|jD}|jd |d |jd |j d |j d|d|d S)z* return a pretty representation of myself rzrz,dc->[rrcss|]}t|VqdS)N)rJ)rPrrDrDrEr  sz!Table.__repr__..rcss|] }|jVqdS)N)rK)rPrlrDrDrEr  sz12.12z (typ->z,nrows->z,ncols->z ,indexers->[r) rrUrwr|rrr<r'r?rncols)rZjdcrverZjverZ jindex_axesrDrDrEr s  zTable.__repr__)rcCs"x|jD]}||jkr|SqWdS)z return the axis for c N)rrK)rrrlrDrDrEr s  zTable.__getitem__c Cs|dkr dS|j|jkr2td|jd|jdxdD]}t||d}t||d}||kr8x@t|D]4\}}||}||krftd|d|d|dqfWtd|d|d|dq8WdS) z$ validate against an existing table Nz'incompatible table_type with existing [z - r)r<rr=zinvalid combination of [z] on appending data [z] vs current table [)rArr*rrr)rr~rsvovrsaxZoaxrDrDrEr s     zTable.validatecCs t|jtS)z@the levels attribute is 1 or a list in the case of a multi-index)r@rrS)rrDrDrEis_multi_index szTable.is_multi_index)r[rc Csfddt|jjD}y |}Wn,tk rN}ztd|Wdd}~XYnXt|ts^t||fS)ze validate that we can store the multi-index; reset and return the new object cSs&g|]\}}|dk r|nd|qS)NZlevel_rD)rPrlrDrDrErR sz-Table.validate_multiindex..zBduplicate names/columns in the multi-index when storing as a tableN)rrtrZ reset_indexrr@r(r)rr[rZ reset_objrrDrDrEvalidate_multiindex s zTable.validate_multiindexcCstdd|jDS)z/ based on our axes, compute the expected nrows cSsg|]}|jjdqS)r)rr)rPrrDrDrErR sz(Table.nrows_expected..)rArr<)rrDrDrEnrows_expected szTable.nrows_expectedcCs d|jkS)z has this table been created ra)r)rrDrDrEr] szTable.is_existscCst|jddS)Nra)r*r)rrDrDrEr szTable.storablecCs|jS)z. return the table group (this is my storable) )r)rrDrDrEra sz Table.tablecCs|jjS)N)rar)rrDrDrEr sz Table.dtypecCs|jjS)N)rar)rrDrDrEr szTable.descriptioncCst|j|jS)N)rrr<r=)rrDrDrEr sz Table.axescCstdd|jDS)z0 the number of total columns in the values axes css|]}t|jVqdS)N)rUr)rPrlrDrDrEr ! szTable.ncols..)sumr=)rrDrDrEr@ sz Table.ncolscCsdS)NFrD)rrDrDrE is_transposed# szTable.is_transposedcCs(ttdd|jDdd|jDS)z@return a tuple of my permutated axes, non_indexable at the frontcSsg|]}t|dqS)r)ri)rPrlrDrDrErR, sz*Table.data_orientation..cSsg|]}t|jqSrD)rir)rPrlrDrDrErR- s)rTrrrr<)rrDrDrEdata_orientation' szTable.data_orientationcsRdddddjD}fddjD}fddjD}t|||S)z> return a dict of the kinds allowable columns for this object rtr)rrNcSsg|]}|j|fqSrD)rr)rPrlrDrDrErR7 sz$Table.queryables..csg|]\}}|dfqS)NrD)rPrr) axis_namesrDrErR8 scs&g|]}|jtjkr|j|fqSrD)rKrrwrr)rPr )rrDrErR: s)r<rr=r)rd1Zd2Zd3rD)rLrrE queryables1 s  zTable.queryablescCsdd|jDS)z return a list of my index cols cSsg|]}|j|jfqSrD)rrr)rPrrDrDrErRD sz$Table.index_cols..)r<)rrDrDrE index_colsA szTable.index_colscCsdd|jDS)z! return a list of my values cols cSsg|] }|jqSrD)rr)rPrrDrDrErRH sz%Table.values_cols..)r=)rrDrDrE values_colsF szTable.values_cols)rmrcCs|jj}|d|dS)z+ return the metadata pathname for this key z/meta/z/meta)rr)rrmrrDrDrE_get_metadata_pathJ szTable._get_metadata_path)rmrcCs0t|}|jj|||d|j|j|jddS)z Write out a metadata array to the key as a fixed-format Series. Parameters ---------- key : str values : ndarray ra)rsrHrxryN)r.rr|rQrHrxry)rrmrrDrDrErO s zTable.write_metadata)rmcCs0tt|jdd|ddk r,|j||SdS)z) return the meta data array for this key rN)r*rrrrQ)rrmrDrDrErb szTable.read_metadatacCspt|j|j_||j_||j_|j|j_|j|j_|j|j_|j|j_|j |j_ |j |j_ |j |j_ dS)z! set our table type & indexables N) rJrArrOrPrrwryrHrxrr<)rrDrDrErh s        zTable.set_attrscCst|jddpg|_t|jddp$g|_t|jddp8i|_t|jdd|_tt|jdd|_tt|jdd|_ t|jd dpg|_ d d |j D|_ d d |j D|_ dS) z retrieve our attributes rNrwr<ryrHrxrercSsg|]}|jr|qSrD)r)rPrlrDrDrErR~ sz#Table.get_attrs..cSsg|]}|js|qSrD)r)rPrlrDrDrErR s)r*rrrwr<ryrIrHrFrxr indexablesr<r=)rrDrDrEru szTable.get_attrscCs\|dk rX|jddkrX|jddkrX|jddkrXtddd|jD}t|tdS) z- are we trying to operate on an old version? NrrNrrrcSsg|] }t|qSrD)rJ)rPrrDrDrErR sz*Table.validate_version..)rincompatibility_docr|rrr])rrVrrDrDrEr s*zTable.validate_versioncCs^|dkr dSt|tsdS|}x6|D]*\}}|dkr>q,||kr,td|dq,WdS)z validate the min_itemsize doesn't contain items that are not in the axes this needs data_columns to be defined Nrzmin_itemsize has the key [z%] which is not an axis or data_column)r@rrNrr)rruqrr rDrDrEvalidate_min_itemsize s zTable.validate_min_itemsizec sg}jjjxztjjD]j\}\}}t|}|}|dk rLdnd}|d}t|d} t|||| |j||d} || q Wt j t |fdd| fddtjj D|S) z1 create/cache the indexables if they don't exist Nrrx)rKrrtr$rsrarruc st|tstt}|krt}t|}t|j}t|dd}t|dd}t|} |}t|dd} |||||||j | ||d } | S)Nrxrr) rKrrrr$rtrsrarrur) r@rJrrrr*_maybe_adjust_namerrrra) rrrradj_namerrr$mdrr[)base_posrdescr table_attrsrDrErb s.   zTable.indexables..fcsg|]\}}||qSrDrD)rPrr)rbrDrErR sz$Table.indexables..)rrarrrOr*rrnrrrrwrUrrP) r _indexablesrrrKrrXrryr$ index_colrD)rYrrZrbrr[rErR s0     # zTable.indexables)r$c CsV|s dS|dkrdS|dks(|dkr8dd|jD}t|ttfsL|g}i}|dk r`||d<|dk rp||d<|j}x|D]}t|j|d}|dk r |jr|j }|j } |j } |dk r| |kr| n| |d<|dk r| |kr| n| |d<|jsN|j drtd |jf|q|||jd d kr|td |d |d|dq|WdS)aZ Create a pytables index on the specified columns. Parameters ---------- columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError if trying to create an index on a complex-type column. Notes ----- Cannot index Time64Col or ComplexCol. Pytables must be >= 3.0. NFTcSsg|]}|jr|jqSrD)rrr)rPrlrDrDrErR sz&Table.create_index..r#r$complexzColumns containing complex values can be stored but cannot be indexed when using table format. Either use fixed format, set index=False, or do not include the columns containing complex values to data_columns when initializing the table.rrNzcolumn z/ is not a data_column. In order to read column z: you must reload the dataframe into HDFStore and include z with the data_columns argument.)rrr@rTrSrar*r r5rtr#r$Z remove_indexrr1rr%rrh) rrr#r$kwrarr rtZ cur_optlevelZcur_kindrDrDrEr% sF    zTable.create_index)rrrc Cs^t||||d}|}g}x<|jD]2}||j|j||j|j|jd}| |q$W|S)a Create the axes sniffed from the table. Parameters ---------- where : ??? start : int or None, default None stop : int or None, default None Returns ------- List[Tuple[index_values, column_values]] )rVrr)ryrHrx) Selectionrrrr<rryrHrxrr) rrVrr selectionrrmrlresrDrDrE _read_axes<s   zTable._read_axes)rcCs|S)z return the data for this obj rD)rYr[rrDrDrE get_object^szTable.get_objectcst|s gS|d\}|j|i}|ddkrL|rLtd|d||dkr^t}n |dkrjg}t|trt|t|}|fdd | Dfd d |DS) zd take the input data_columns and min_itemize and create a data columns spec rrr,z"cannot use a multi-index on axis [z] with data_columns TNcs g|]}|dkr|kr|qS)rrD)rPr)existing_data_columnsrDrErRsz/Table.validate_data_columns..csg|]}|kr|qSrDrD)rPr) axis_labelsrDrErRs) rUr<rrrSr@rrrr)rrwrurrr<rD)rfrerEvalidate_data_columnscs$    zTable.validate_data_columns)r[rc0stts,|jj}td|dtddkr:dgfddD|rzd}d d|jDt|j }|j }nd }|j } |j d kst t|j d krtd g} |dkrd}fdddDd} j| } t| } |r|j|j|j|| ||.| |d }/t6|d rr|j?|/_?|/@||r|r|/A||/S)!a0 Create and return the axes. Parameters ---------- axes: list or None The names or numbers of the axes to create. obj : DataFrame The object to create axes on. validate: bool, default True Whether to validate the obj against an existing object already written. nan_rep : A value to use for string column nan_rep. data_columns : List[str], True, or None, default None Specify the columns that we want to create to allow indexing on. * True : Use all available columns. * None : Use no columns. * List[str] : Use the specified columns. min_itemsize: Dict[str, int] or None, default None The min itemsize for a column in bytes. z/cannot properly create the storer for: [group->z,value->rNrcsg|]}|qSrD)_get_axis_number)rPrl)r[rDrErRsz&Table._create_axes..TcSsg|] }|jqSrD)r)rPrlrDrDrErRsFrrNz.get_blk_items..rD)rjr9rD)rjrE get_blk_itemssz)Table._create_axes..get_blk_itemsz-cannot have non-object label DataIndexableColzIncompatible appended table [z]with existing table [Z values_block_) existing_colruryrHrxrpr)r8) rKrrrrsrtr$rprrrurr7cSsg|]}|jr|jqSrD)rrK)rPrrDrDrErRZs) rrrHrxr<rr=rwr<ryr)Br@r(rrrrrr<rSrwryr<rrrUrrrr'rAarrayrrrrYrrZ_get_axis_namerrHrxrrvrr _reindex_axisrgrdr_get_blocks_and_itemsr=rrrrrJ IndexErrorr_maybe_convert_for_string_atomrVrrrrrKrrrprrrrrrrrUr)0rrr[rryrwrur table_existsZnew_infonew_non_index_axesrrlZ append_axisindexerZ exist_axisr< axis_nameZ new_indexZnew_index_axesjrkr block_objr9r5Zvaxesrbb_itemsrrKrlrnew_namedata_convertedrWrsr$rprrurr7rrZdcsZ new_tablerD)rr[rE _create_axess             "              zTable._create_axesc Csddd}|jj}||j|}t|r|d\}} t| t|} |j| |dj} t| j}|| |}x<|D]4} |j| g|dj} || j||| | jqlW|r\ddt||D} g}g}x|D]}t |j }y&| |\}}| || |Wqt tfk rN}z*ddd |D}td |d |Wdd}~XYqXqW|}|}||fS) Ncsfdd|DS)Ncsg|]}j|jqSrD)rrr:)rPr;)rjrDrErRzszFTable._get_blocks_and_items..get_blk_items..rD)rjr9rD)rjrErkysz2Table._get_blocks_and_items..get_blk_itemsr)rcSs"i|]\}}||ft|qSrD)rTtolist)rPrxryrDrDrEr sz/Table._get_blocks_and_items..rzcss|]}t|VqdS)N)r;)rPitemrDrDrEr sz.Table._get_blocks_and_items..z+cannot match existing table structure for [z] on appending data)r8r9rUr*rrrSrrrTrrrrrprr|r)rwrrrsr=rwrkr9r5rrfZ new_labelsrjrZby_itemsZ new_blocksZ new_blk_itemsZearrxryrZjitemsrDrDrEross@          zTable._get_blocks_and_itemsr`)rac s|dk rt|}|dk rRjrRtjts.tx"jD]}||kr6|d|q6Wx jD]\}}t|||qZW|jdk rx2|j D]$\}}fdd} | ||qWS)z process axes filters NrcsxjD]}|}|}|dk s,t||krdjrJ|tj}||}j|d|S||krt t |j }t |}t t rd|}||}j|d|SqWtd|ddS)N)rrNzcannot find the field [z] for filtering!)Z _AXIS_ORDERSrh _get_axisrrEunionr*rrr8r*rr@r(r)fieldfiltruZ axis_numberZ axis_valuesZtakersr)r[oprrDrEprocess_filters"       z*Table.process_axes..process_filter) rSrEr@rrinsertrrnfilterrs) rr[rarrrlabelsrrrrD)r[rrrE process_axess  !zTable.process_axes)rprrrcCs|dkrt|jd}d|d}dd|jD|d<|rj|dkrH|jpFd}tj|||pZ|jd }||d <n|jdk r~|j|d <|S) z< create the description of the table from the axes & values Ni'ra)rKrcSsi|]}|j|jqSrD)rsrr)rPrlrDrDrEr sz,Table.create_description..r )rprqrr)maxrHrrrkrrr)rrqrprrrrrDrDrEcreate_descriptions       zTable.create_description)rrc Cs|||sdSt||||d}|}|jdk rxR|jD]D\}}}|j|||dd} ||| j |||j }qDWt |S)zf select coordinates (row numbers) from a table; return the coordinates object F)rVrrNrN)rr) rrr` select_coordsrrsrrlrilocrr*) rrVrrraZcoordsrrrr7rDrDrErs  "zTable.read_coordinates)rrrcCs||sdS|dk r$tdx|jD]v}||jkr,|jsPtd|dt|jj |}| |j |j ||||j |j|jd}tt|d|j|dSq,Wtd|d dS) zj return a single column from the table, generally only indexables are interesting FNz4read_column does not currently accept a where clausezcolumn [z=] can not be extracted individually; it is not data indexable)ryrHrxrN)rKz] not found in the table)rrrrrKrrr*rar rr<rryrHrxr.rrpr)rrrVrrrlrZ col_valuesrDrDrErs&      zTable.read_column)NreNNNNNN)N)NNN)NN)TNNN)N)NNN)NNN)FrYrZr[rgrrrJrhrrrir rrrnr rrrr}rrjr?rrrrrErr(rGrHr]rrarrrr@rJrKrNrOrPrQrArrrrrrrUrrRr r%rrcrrdrgr| staticmethodrorrrrrrDrD)rrErp s              JU *d 1=rc@s6eZdZdZdZd eeeedddZddZdS) rVz a write-once read-many table: this format DOES NOT ALLOW appending to a table. writing is a one-time operation the data are stored in a format that allows for searching the data on disk rLN)rrcCs tddS)z[ read the indices and the indexing array, calculate offset rows and return z!WORMTable needs to implement readN)r)rrVrrrrDrDrErLs zWORMTable.readcKs tddS)z write in a format that we can search later on (but cannot append to): write out the indices and the values using _write_array (e.g. a CArray) create an indexing table so that we can search z"WORMTable needs to implement writeN)r)rrrDrDrEr_XszWORMTable.write)NNNN) rYrZr[rgrAr rirr_rDrDrDrErVCsrVc @sveZdZdZdZdddZdeeedd d Z e j e e j ee j e e j d d d Z deeeedddZdS)rz* support the new appendable table formats Z appendableNFTcCs|s|jr|j|jd|j||||| | d}x|jD] }|q8W|js|j|||| d}|| |d<|jj |jf||j |j _ x|jD]}| ||qW|j || ddS)Nra)rr[rruryrw)rqrprrr)rv)r]rr`rr|rrrrZ create_tabler<rr write_data)rr[rrrrqrprrurrrvryrwrrarloptionsrDrDrEr_fs0     zAppendableTable.write)rrvcs|jj}|j}g}|rXx@|jD]6}t|jjdd}t|tj r| |j dddqWt |r|d}x|ddD] }||@}qvW| }nd}dd |jD} t | } | dkst| d d |jD} d d | D} g} xBt| D]6\} }|f|j|| | j}| | | |qW|dkr0d }tjt|||jd }||d}xt|D]t} | |t| d||krP|j|fdd | D|dk r|ndfdd | Ddq\WdS)z` we form the data into a 2-d including indexes,values,mask write chunk-by-chunk r)ru1F)r8rNNcSsg|] }|jqSrD)r)rPrlrDrDrErRsz.AppendableTable.write_data..cSsg|] }|qSrD)r)rPrlrDrDrErRsc Ss,g|]$}|tt|j|jdqS)rN)Z transposerAZrollrr)rPr rDrDrErRsi)rcsg|]}|qSrDrD)rPrl)end_istart_irDrErRscsg|]}|qSrDrD)rPr )rrrDrErRs)indexesrr)rrrHr=r1r7rr@rArrrrrUrr<rrrreshaperZrlrwrite_data_chunk)rrrvrrmasksrlrmrnindexesrZbvaluesrr Z new_shaperowschunksrD)rrrErsJ       zAppendableTable.write_data)rrrrc Csx|D]}t|jsdSqW|djd}|t|krHtj||jd}|jj}t|}x t|D]\} } | ||| <qbWx$t|D]\} }|||| |<qW|dk r|j t dd} | s|| }t|r|j ||j dS)z Parameters ---------- rows : an empty memory space where we are putting the chunk indexes : an array of the indexes mask : an array of the masks values : an array of the values Nr)rF)r8)rArrrUrZrrrrrrrrarrr) rrrrrr rrrrrrrDrDrErs&    z AppendableTable.write_data_chunk)rrcCsf|dkst|sf|dkr:|dkr:|j}|jj|jddn(|dkrH|j}|jj||d}|j|S|srdS|j}t ||||d}| }t | }t|} | rb| } t| | dkj} t| sdg} | d| kr| | | ddkr| dd| } xLt| D]@} |t| | }|j||jd||jddd| } qW|j| S)NT)r)rrrNrr)rUrrr`rraZ remove_rowsrrr`rr.Z sort_valuesdiffrSrtrrrrreversedrr)rrVrrrrararZ sorted_serieslnrrZpgrrrDrDrEr sB     "  zAppendableTable.delete) NFNNNNNNFNNT)F)NNN)rYrZr[rgrAr_r rirrrArr rrrDrDrDrEras( .=(rc@sleZdZUdZdZdZdZeZe e e d<e e dddZee d d d Zdeeeed ddZd S)rTz* support the new appendable table formats rBrJrr)rcCs|jdjdkS)NrrN)r<r)rrDrDrErJPsz"AppendableFrameTable.is_transposed)rcCs|r |j}|S)z these are written transposed )r)rYr[rrDrDrErdTszAppendableFrameTable.get_objectN)rrcs4|sdSj|||d}tjrHjjddini}fddtjD}t|dkstt |d}||d} g} x\tjD]L\} } | j krq|| \} }|ddkrt | }nt | }|d}|dk r|j|d d jr$|}|}t | t| d dd }n|j}t | t| d dd }|}|jdkrpt|tjrp|d|jdf}t|tjrt|j||d }n,t|t rt|||d }nt|g||d }|j|jkst |j|jf| |qWt| dkr| d}n t| dd}t|||d}j|||d}|S)N)rVrrrcs"g|]\}}|jdkr|qS)r)r<)rPrr3)rrDrErRrsz-AppendableFrameTable.read..rNrr,rT)inplacerK)rK)rrt)r)rar) rrrcrUrr<rrrrr=r, from_tuplesr* set_namesrJr*rrr@rArrrr(Zdtypesrrrrr0r`r)rrVrrrrr<ZindsindrtframesrrlZ index_valsrr rrZindex_Zcols_r6rarD)rrEr[sV "      "  zAppendableFrameTable.read)NNNN)rYrZr[rgrrArr(rr rrhrjrrJrrdr rirrDrDrDrErTHs rTcsveZdZdZdZdZdZeZe e dddZ e e dd d Z dfd d Zdeeeeedfdd ZZS)rRz* support the new appendable table formats rGrHr)rcCsdS)NFrD)rrDrDrErJsz#AppendableSeriesTable.is_transposed)rcCs|S)NrD)rYr[rrDrDrErdsz AppendableSeriesTable.get_objectNc s<t|ts|jpd}||}tjf||jd|S)z- we are going to write this as a frame table r)r[rw)r@r(rKZto_framerr_rr})rr[rwrrK)rrDrEr_s   zAppendableSeriesTable.write)rrrcs|j}|dk rF|rFt|jts"tx"|jD]}||kr*|d|q*Wtj||||d}|rn|j|jdd|j dddf}|j dkrd|_ |S)Nr)rVrrrT)rr) rEr@rrSrrrr set_indexrrK)rrVrrrrErrC)rrDrErs   zAppendableSeriesTable.read)N)NNNN)rYrZr[rgrrArr.rrjrrJrrdr_r rirrrDrD)rrErRs rRcs(eZdZdZdZdZfddZZS)rSz* support the new appendable table formats rGrIc s^|jpd}||\}|_t|jts*tt|j}||t||_t j fd|i|S)z- we are going to write this as a frame table rr[) rKrGrr@rSrrrr*rrr_)rr[rrKZnewobjr )rrDrEr_s    z AppendableMultiSeriesTable.write)rYrZr[rgrrAr_rrDrD)rrErSsrSc@sheZdZUdZdZdZdZeZe e e d<e e dddZe d d Zd d Zed dZddZdS)rQz< a table that read/writes the generic pytables table format rBrCrr)rcCs|jS)N)r)rrDrDrEr'szGenericTable.pandas_typecCst|jddp|jS)Nra)r*r)rrDrDrErszGenericTable.storablecCsLg|_d|_g|_dd|jD|_dd|jD|_dd|jD|_dS)z retrieve our attributes NcSsg|]}|jr|qSrD)r)rPrlrDrDrErRsz*GenericTable.get_attrs..cSsg|]}|js|qSrD)r)rPrlrDrDrErRscSsg|] }|jqSrD)rK)rPrlrDrDrErRs)rryrrRr<r=rw)rrDrDrErs zGenericTable.get_attrsc Cs|j}|d}|dk rdnd}tdd|j||d}|g}xlt|jD]^\}}t|ts\tt ||}||}|dk r|dnd}t |||g||j||d} | | qFW|S)z2 create the indexables from the table description rtNrr)rKrrarru)rKrtrrsrarru) rrrrarZ_v_namesr@rJrr*rrr) rrrXrr]r\rrrrrDrDrErR s*   zGenericTable.indexablescKs tddS)Nz cannot write on an generic table)r)rrrDrDrEr_,szGenericTable.writeN)rYrZr[rgrrArr(rr rrhrjrJr'rrrrRr_rDrDrDrErQs    #rQcsheZdZdZdZeZdZe dZ e e dddZ dfd d Zdeeeed fd d ZZS)rUz a frame with a multi-index rKrz ^level_\d+$)rcCsdS)NZappendable_multirD)rrDrDrEr?8sz*AppendableMultiFrameTable.table_type_shortNc s||dkrg}n|dkr |j}||\}|_t|jts@tx"|jD]}||krH|d|qHWtj f||d|S)NTr)r[rw) rr}rGrr@rSrrrr_)rr[rwrr)rrDrEr_<s  zAppendableMultiFrameTable.write)rrcsDtj||||d}|j}|jfdd|jjD|_|S)N)rVrrrcs g|]}j|rdn|qS)N) _re_levelssearch)rPrK)rrDrErRUsz2AppendableMultiFrameTable.read..)rrrrrtrr)rrVrrrr6)r)rrErHs  zAppendableMultiFrameTable.read)N)NNNN)rYrZr[rgrAr(rrrecompilerrjrJr?r_r rirrrDrD)rrErU0s rU)r[rrrcCs||}t|}|dk r"t|}|dks4||rB||rB|St|}|dk rlt|j|dd}||stddg|j}|||<|jt|}|S)NF)sort) rr8equalsuniquerslicerrrT)r[rrr~r3ZslicerrDrDrErn[s   rn)rprcCst|}|S)z- for a tz-aware type, return an encoded zone )rZ get_timezone)rpzonerDrDrErss r)rrprrcCst|tr"|jdks"|j|ks"t|dk rtt|trB|j}|j}n d}|}t|}t||d}|d |}n|rt j |dd}|S)a coerce the values to a DatetimeIndex if tz is set preserve the input shape if possible Parameters ---------- values : ndarray or Index tz : str or tzinfo coerce : if we do not have a passed timezone, coerce to M8[ns] ndarray N)rKrzM8[ns])r) r@r)rprrKr,rrFrrrAr)rrprrKrDrDrErys   r)rKrtrHrxrc Cstt|tst|j}t|\}}t|}t|}t|tsFt |j rlt ||||t |ddt |dd|dSt|t r~tdtj|dd} t|} | dkrtjdd | Dtjd }t ||dt|d S| d krt| ||}|j j} t ||d t| |d S| d kr$t |||||dSt|tjr>|j tksBt|dksTt|t}t |||||d SdS)Nrorp)rr$rsrorprqzMultiIndex not supported here!F)r%rcSsg|] }|qSrD) toordinal)rPr rDrDrErRsz"_convert_index..)r)rqr)integerZfloating)rr$rsrqr)r@rJrrKrrrrr+r&rrnr*r,rrr(rArZint32rkZ Time32Col_convert_string_arrayrwrrrr*) rKrtrHrxrqrrr$rr.rrwrDrDrErsL           r)r$rHrxrcCs|dkrt|}n|dkr$t|}n|dkrxytjdd|Dtd}Wqtk rttjdd|Dtd}YqXnT|dkrt|}n@|d krt|d||d }n&|d krt|d }ntd ||S)NrrrcSsg|]}t|qSrD)rr)rPr rDrDrErRsz$_unconvert_index..)rcSsg|]}t|qSrD)rr)rPr rDrDrErRs)rfloatr)ryrHrxrrzunrecognized index type )r)r/rArrrr)r7r$rHrxrtrDrDrEr!s$    r!)rKcCs|js |jS|jj}tj|jdd}|dkr6tdn(|dkrHtdn|dks^|dks^|jS|j|dd }t|t r~|d }|j} tj| dd}|dkrx\t t |j d D]F} | | } tj| dd}|dkr|jj| } td | d |d qWt| ||| j } | j |j ks*t| j |j f| j}t|tr\t||pX|dpXd }t|pfd |}|dk r||}||kr|}| jd|dd} | S)NF)r%rz+[date] is not implemented as a table columnrz>too many timezones in this block, create separate data columnsrr)ZdowncastrzCannot serialize the column [z!] because its data contents are [z] object dtyperz|S)r8)rrrrKrr(rZfillnar@rSrrUrigetr:rtrrrrwrrirrrr)rKr1rlruryrHrxrr.r7rrrr{rwZecirDrDrErqsH         rq)r7rHrxrcCs\t|r(t|j||j|j}t|}t dt |}t j |d|d}|S)a Take a string-like that is object dtype and coerce to a fixed size string type. Parameters ---------- data : np.ndarray[object] encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[fixed-length-string] rNS)r)rUr.rrJencoderrrrr libwritersmax_len_string_arrayrAr)r7rHrxensuredrwrDrDrEr5s  rcCs|j}tj|td}t|rvtt|}d|}t |dt r^t |j j ||dj}n|j|ddjtdd}|dkrd}t||}||S) a* Inverse of _convert_string_array. Parameters ---------- data : np.ndarray[fixed-length-string] nan_rep : the storage repr of NaN encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[object] Decoded data. )rUr)rxF)r8Nri)rrArrrrUrrrr@r r.rJrBrrZ!string_array_replace_from_nan_repr)r7ryrHrxrrwrrDrDrErTs  r)rrrHrxcCs6t|tstt|t|r2t|||}||}|S)N)r@rJrr _need_convert_get_converter)rrrHrxconvrDrDrEr{s  r)r$rHrxcs8|dkrddS|dkr&fddStd|dS)NrcSstj|ddS)NzM8[ns])r)rAr)rrDrDrEr{z _get_converter..rcst|ddS)N)ryrHrx)r)r)rHrxrDrEr{sz invalid kind )r)r$rHrxrD)rHrxrErs r)r$rcCs|dkr dSdS)N)rrTFrD)r$rDrDrErsr)rKrrcCslt|tst|dkrtd|ddkrh|ddkrh|ddkrhtd|}|rh|d}d|}|S) z Prior to 0.10.1, we named values blocks like: values_block_0 an the name values_0, adjust the given name if necessary. Parameters ---------- name : str version : Tuple[int, int, int] Returns ------- str z6Version is incorrect, expected sequence of 3 integers.rrNrrzvalues_block_(\d+)Zvalues_)r@rJrUrrrr)rKrrgrprDrDrErVs$   rV) dtype_strrcCst|}|ds|dr"d}n|dr2d}n|drBd}n|dsV|dr\d}nn|drld}n^|d r|d }nN|d rd }n>|d rd }n.|d rd}n|dkrd}ntd|d|S)zA Find the "kind" string describing the given dtype name. rr rr^rirrr timedeltarrrrrzcannot interpret dtype of [r)rFr1r)rr$rDrDrErs.       r)r7cCsbt|tr|j}|jjdd}|jjdkr@t| d}nt|t rP|j }t|}||fS)zJ Convert the passed data into a storable form and a dtype string. rr)rMr') r@r2rrrKrbr$rArr+r-r,)r7rrDrDrErs    rc@sDeZdZdZd eeeeedddZddZdd Z d d Z dS) r`z Carries out a selection operation on a tables.Table object. Parameters ---------- table : a Table object where : list of Terms (or convertible to) start, stop: indices to start and/or stop selection N)rarrc CsR||_||_||_||_d|_d|_d|_d|_t|rt t t j |dd}|dksd|dkrt |}|jt jkr|j|j}}|dkrd}|dkr|jj}t ||||_nVt|jjt jr|jdk r||jks|jdk r||jkrt d||_WdQRX|jdkrN|||_|jdk rN|j\|_|_dS)NF)r%rbooleanrz3where must have index locations >= start and < stop)rarVrr conditionrZtermsrr#rrrr(rArrZbool_rr issubclassrrrgenerateevaluate)rrarVrrinferredrDrDrErs<       zSelection.__init__c Cs~|dkr dS|j}yt|||jjdStk rx}z2d|}td|d|d}t||Wdd}~XYnXdS)z) where can be a : dict,list,tuple,string N)rNrHrzz- The passed where expression: a* contains an invalid variable reference all of the variable references must be a reference to an axis (e.g. 'index' or 'columns'), or a data_column The currently defined references are: z ) rarNr5rH NameErrorr|rrr)rrVrTrZqkeysrrDrDrErs zSelection.generatecCsX|jdk r(|jjj|j|j|jdS|jdk rB|jj|jS|jjj|j|jdS)z( generate the selection N)rr) rraZ read_wherersrrrrr)rrDrDrEr6s   zSelection.selectcCs|j|j}}|jj}|dkr$d}n|dkr4||7}|dkrB|}n|dkrR||7}|jdk rx|jjj|j||ddS|jdk r|jSt ||S)z( generate the selection NrT)rrr) rrrarrZget_where_listrsrrAr)rrrrrDrDrErBs   zSelection.select_coords)NNN) rYrZr[rgrr rirrrrrDrDrDrEr`s * r`) rlNNFNTNNNNrer?) NrreNNNNFN)N)F)rg contextlibrr8rrrrrrtextwraprtypingrrrr r r r r rrZnumpyrAZpandas._configrrZ pandas._libsrrrZpandas._libs.tslibsrZpandas._typingrrrrrZpandas.compat._optionalrZpandas.compat.pickle_compatrZ pandas.errorsrZpandas.util._decoratorsrZpandas.core.dtypes.commonrrrr r!r"r#r$r%r&Zpandas.core.dtypes.missingr'rr(r)r*r+r,r-r.r/r0r1Zpandas.core.arraysr2r3r4Zpandas.core.commoncorecommonrZ pandas.core.computation.pytablesr5r6Zpandas.core.constructionr7Zpandas.core.indexes.apir8Zpandas.io.commonr9Zpandas.io.formats.printingr:r;rgr<r=r>rrGrFrIrLrOrirWrrXr\Warningr]rSr^rr_Z duplicate_docr)r=rZ dropna_docZ format_docZ config_prefixZregister_optionZis_boolZis_one_of_factoryrfrjrkrJrrrrr}rrnrrrrrr3rNr/rOrrVrrTrRrSrQrUrnrrrrr!rqrrrrrrVrrr`rDrDrDrEs$  ,      0 0       \+$Np3S ZZhd1B+ "<? % !