U Dx`@sdZddlmZddlmZmZmZddlmZddlm Z ddl Z ddl m Z m Z mZmZmZddlZddlZddlmmZddlmZmZmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%Gddde&Z'Gddde(Z)da*ddZ+e,dddZ-ddZ.ddZ/dTddZ0ddZ1dUd d!Z2dVd"d#Z3edWde d$d%d&Z4edXe5e e d$d(d&Z4dYee5ee e e fd$d)d&Z4edZde d$d*d+Z6ed[e5e e d$d,d+Z6d\ee5ee e e fd$d-d+Z6ed]de d$d.d/Z7ed^e5e e d$d0d/Z7d_ee5ee e e fd$d1d/Z7d`ddd3d4Z8dad5d6Z9e9Z:d7d8Z;dbd:d;Zd?d?e#Z>Gd@dAdAe>Z?dBdCdDdEdFdGdDdHZ@dIdJZAdKdLZBdMZCGdNdOdOe=ZDGdPdQdQe>ZEdcdRdSZFdS)dz| Collection of query wrappers / abstractions to both facilitate data retrieval and to reduce dependency on DB-specific API. )contextmanager)datedatetimetime) LooseVersion)partialN)IteratorListOptionalUnionoverload)is_datetime64tz_dtype is_dict_like is_list_like)DatetimeTZDtype)isna) DataFrameSeries) PandasObject) to_datetimec@s eZdZdS)SQLAlchemyRequiredN__name__ __module__ __qualname__rr4/tmp/pip-target-zr53vnty/lib/python/pandas/io/sql.pyrsrc@s eZdZdS) DatabaseErrorNrrrrrrsrcCsTtdkr2zddl}daWntk r0daYnXtrLddl}t||jjSdSdS)NrTF)_SQLALCHEMY_INSTALLED sqlalchemy ImportError isinstanceengineZ Connectableconrrrr_is_sqlalchemy_connectable)s r%)returncCsddl}t|jtdkS)ze Check if sqlalchemy.__version__ is at least 1.4.0, when several deprecations were made. rNz1.4.0)rr __version__)rrrr_gt14;sr(cCs6|g}|dk r2t|dr$||g7}n|t|g7}|S)z9Convert SQL and params args to DBAPI2.0 compliant format.Nkeys)hasattrlist)sqlparamsargsrrr_convert_paramsEs   r/cCs2|dks|dks|dkrg}nt|ds.|g}|S)z3Process parse_dates argument for read_sql functionsTNF__iter__)r* parse_datesrrr_process_parse_dates_argumentPs  r3cCst|trt|fddi|S|dkrJt|jjtjsFt|jjtjrJd}|dkrbt|d||dSt |jrxt|ddSt|d||d SdS) Nerrorsignores)Ddhmr6msusnsZcoerce)r4unitutcTr?)r4formatr?) r!dictr issubclassdtypetypenpfloatingintegerr )colr?rArrr_handle_date_column[s   rJc Csft|}|D]P\}}t|js*||krz ||}Wntk rNd}YnXt||d||<q|S)zz Force non-datetime columns to be read as such. Supports both string formatted and integer timestamp columns. NrA)r3itemsr rD TypeErrorrJ)Z data_framer2col_namedf_colfmtrrr_parse_date_columnsps  rQTcCs4tj|||d}t||}|dk r0|j|dd|S)z(Wrap result set of query in a DataFrame.columns coerce_floatNTZinplace)r from_recordsrQ set_index)datarS index_colrTr2framerrr _wrap_results  r[cCs2|dkrt|}n t|dd}t||}|j|S)aY Execute the given SQL query using the provided connection object. Parameters ---------- sql : string SQL query to be executed. con : SQLAlchemy connectable(engine/connection) or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by the library. If a DBAPI2 object, only sqlite3 is supported. cur : deprecated, cursor is obtained from connection, default: None params : list or tuple, optional, default: None List of parameters to pass to execute method. Returns ------- Results Iterable NT is_cursor)pandasSQL_builderr/execute)r,r$curr- pandas_sqlr.rrrr_s    r_) chunksizer&cCsdSNr table_namer$schemarYrTr2rSrbrrrread_sql_tables rgcCsdSrcrrdrrrrgs c Cst|}t|stdddl}ddlm} | ||d} z| j|gddWn8|jjk r} zt d|d | W5d} ~ XYnXt || d } | j ||||||d } | dk r| St d|d |dS) a~ Read SQL database table into a DataFrame. Given a table name and a SQLAlchemy connectable, returns a DataFrame. This function does not support DBAPI connections. Parameters ---------- table_name : str Name of SQL table in database. con : SQLAlchemy connectable or str A database URI could be provided as str. SQLite DBAPI connection mode not supported. schema : str, default None Name of SQL schema in database to query (if database flavor supports this). Uses default schema if None (default). index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. Can result in loss of Precision. parse_dates : list or dict, default None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. columns : list, default None List of column names to select from SQL table. chunksize : int, default None If specified, returns an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] A SQL table is returned as two-dimensional data structure with labeled axes. See Also -------- read_sql_query : Read SQL query into a DataFrame. read_sql : Read SQL query or database table into a DataFrame. Notes ----- Any datetime values with time zone information will be converted to UTC. Examples -------- >>> pd.read_sql_table('table_name', 'postgres:///db_name') # doctest:+SKIP z9read_sql_table only supported for SQLAlchemy connectable.rNMetaDatarfT)onlyZviewszTable z not found)metarYrTr2rSrb) _engine_builderr%NotImplementedErrorrsqlalchemy.schemarjreflectexcZInvalidRequestError ValueError SQLDatabase read_table)rer$rfrYrTr2rSrbrrjrmerrratablerrrrgs0A  $  cCsdSrcrr,r$rYrTr-r2rbrrrread_sql_query,s rzcCsdSrcrryrrrrz9s cCst|}|j||||||dS)a7 Read SQL query into a DataFrame. Returns a DataFrame corresponding to the result set of the query string. Optionally provide an `index_col` parameter to use one of the columns as the index, otherwise default integer index will be used. Parameters ---------- sql : str SQL query or SQLAlchemy Selectable (select or text object) SQL query to be executed. con : SQLAlchemy connectable, str, or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. Useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql : Read SQL query or database table into a DataFrame. Notes ----- Any datetime values with time zone information parsed via the `parse_dates` parameter will be converted to UTC. rYr-rTr2rb)r^ read_query)r,r$rYrTr-r2rbrarrrrzFs=cCsdSrcrr,r$rYrTr-r2rSrbrrrread_sqls r~cCsdSrcrr}rrrr~s c Cst|}t|tr(|j||||||dSz||} Wntk rNd} YnX| rz|jj|gd|j||||||dS|j||||||dSdS)a Read SQL query or database table into a DataFrame. This function is a convenience wrapper around ``read_sql_table`` and ``read_sql_query`` (for backward compatibility). It will delegate to the specific function depending on the provided input. A SQL query will be routed to ``read_sql_query``, while a database table name will be routed to ``read_sql_table``. Note that the delegated function might have more specific notes about their functionality not listed here. Parameters ---------- sql : str or SQLAlchemy Selectable (select or text object) SQL query to be executed or a table name. con : SQLAlchemy connectable, str, or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. The user is responsible for engine disposal and connection closure for the SQLAlchemy connectable; str connections are closed automatically. See `here `_. index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. columns : list, default: None List of column names to select from SQL table (only used when reading a table). chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql_query : Read SQL query into a DataFrame. r{F)rlrnN) r^r!SQLiteDatabaser| has_table Exceptionrmrrrv) r,r$rYrTr-r2rSrbraZ_is_table_namerrrr~s@B    failc Csj|dkrtd|dt||d} t|tr8|}nt|tsJtd| j||||||||| d dS)a; Write records stored in a DataFrame to a SQL database. Parameters ---------- frame : DataFrame, Series name : str Name of SQL table. con : SQLAlchemy connectable(engine/connection) or database string URI or sqlite3 DBAPI2 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. schema : str, optional Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). if_exists : {'fail', 'replace', 'append'}, default 'fail' - fail: If table exists, do nothing. - replace: If table exists, drop it, recreate it, and insert data. - append: If table exists, insert data. Create if does not exist. index : boolean, default True Write DataFrame index as a column. index_label : str or sequence, optional Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. chunksize : int, optional Specify the number of rows in each batch to be written at a time. By default, all rows will be written at once. dtype : dict or scalar, optional Specifying the datatype for columns. If a dictionary is used, the keys should be the column names and the values should be the SQLAlchemy types or strings for the sqlite3 fallback mode. If a scalar is provided, it will be applied to all columns. method : {None, 'multi', callable}, optional Controls the SQL insertion clause used: - None : Uses standard SQL ``INSERT`` clause (one per row). - 'multi': Pass multiple values in a single ``INSERT`` clause. - callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 )rreplaceappend'' is not valid for if_existsrkz9'frame' argument should be either a Series or a DataFrame) if_existsindex index_labelrfrbrDmethodN)rtr^r!rZto_framerrpto_sql) rZnamer$rfrrrrbrDrrarrrrs(:    rcCst||d}||S)a- Check if DataBase has named table. Parameters ---------- table_name: string Name of SQL table. con: SQLAlchemy connectable(engine/connection) or sqlite3 DBAPI2 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). Returns ------- boolean rk)r^r)rer$rfrarrrrfs rcCsBt|tr>z ddl}Wntk r.daYnX||}|S|S)zw Returns a SQLAlchemy engine from a URI (if con is a string) else it just return con without modifying it. rNF)r!strrr rZ create_enginer#rrrros    roFcCsBt|}t|rt|||dSt|tr2tdn t||dSdS)zm Convenience function to return the correct PandasSQL subclass based on the provided parameters. )rfrmz.Using URI string without sqlalchemy installed.r\N)ror%rur!rr r)r$rfrmr]rrrr^s   r^c@seZdZdZd(ddZdd Zd d Zd d ZddZddZ ddZ ddZ d)ddZ d*ddZ d+ddZddZddZd d!Zd,d"d#Zd$d%Zd&d'ZdS)-SQLTablez For mapping Pandas tables to SQL tables. Uses fact that table is reflected by SQLAlchemy to do better type conversions. Also holds various flags needed to avoid having to pass them between functions all the time. NTrpandasc Cs||_||_||_||_||||_||_||_| |_| |_ |dk rR| |_ n|j |j|j|_ |j dkrt d|ddS)NzCould not init table 'r)rpd_sqlprefixrZ _index_namerrfrr)rD_create_table_setuprx get_tablert) selfrZpandas_sql_enginerZrrrrrfr)rDrrr__init__s   zSQLTable.__init__cCs|j|j|jSrc)rrrrfrrrrexistsszSQLTable.existscCs$ddlm}t||j|jjS)Nr) CreateTable)rqrrrxcompiler connectable)rrrrr sql_schemas zSQLTable.sql_schemacCs:tr|j|jj|_n|j|jj|_|jdSrc)r(rxZ to_metadatarrmZ tometadatacreaterrrr_execute_createszSQLTable._execute_createcCsx|rl|jdkr&td|jdqt|jdkrL|j|j|j|qt|jdkrXqttd|jdn|dS)NrzTable 'z' already exists.rrrr)rrrtrr drop_tablerfrrrrrrs    zSQLTable.createcs(fdd|D}||j|dS)a< Execute SQL statement inserting data Parameters ---------- conn : sqlalchemy.engine.Engine or sqlalchemy.engine.Connection keys : list of str Column names data_iter : generator of list Each item contains a list of values to be inserted csg|]}tt|qSrrBzip.0rowr)rr sz,SQLTable._execute_insert..Nr_rxinsertrconnr) data_iterrXrrr_execute_inserts zSQLTable._execute_insertcs(fdd|D}||j|dS)a Alternative to _execute_insert for DBs support multivalue INSERT. Note: multi-value insert is usually faster for analytics DBs and tables containing a few columns but performance degrades quickly with increase of columns. csg|]}tt|qSrrrrrrrsz2SQLTable._execute_insert_multi..Nrrrrr_execute_insert_multiszSQLTable._execute_insert_multic Cs,|jdk rd|j}|j|j_z|jddWqjtk r`}ztd||W5d}~XYqjXn|j}ttt|j }t |}dg|}t | D]\}\}}|j } | jjdkr| } n(| jjdkr| dt} n | t} t| tjstt| |jrt| } d| | <| ||<q||fS)NTrUz!duplicate name in index/columns: Mr:i8)rrZcopynamesZ reset_indexrtr+maprrSlen enumeraterLZ_valuesrDkindZ to_pydatetimeviewastypeobjectr!rFZndarrayAssertionErrorrEZ _can_hold_nar) rtemprw column_namesZncols data_listi_Zservalsr8maskrrr insert_datas0   $      zSQLTable.insert_datac s|dkr|j}n2|dkr |j}n"t|r4t||}ntd||\}}t|j}|dkrddS|dkrr|}n|dkrtdt||d}|j ^}t |D]N} | |t | d||krqt fdd|D} |||| qW5QRXdS)NZmultizInvalid parameter `method`: rz%chunksize argument should be non-zerorhcsg|]}|qSrr)rZarrZend_iZstart_irrrLsz#SQLTable.insert..)rrcallablerrtrrrZintrrun_transactionrangeminr) rrbrZ exec_insertr)rZnrowschunksrrZ chunk_iterrrrr+s0     zSQLTable.insertccsZ||}|sqVqtj|||d|_|j|d|jdk rL|jj|jdd|jVqdS)z,Return generator through chunked result set.rRr1NTrU) fetchmanyrrVrZ_harmonize_columnsrrW)rresultrbrSrTr2rXrrr_query_iteratorOs   zSQLTable._query_iteratorc s|dk rpt|dkrpddlm}fdd|D}jdk rfjdddD]}|djj|qL||}n j}j|} | } |dk rj | || ||dS| } t j | | |d_j|djdk rjjjd d jSdS) Nr)selectcsg|]}jj|qSr)rxc)rnrrrrhsz!SQLTable.read..)rTr2rRr1TrU)rrrrrrxrrr_r)rfetchallrrVrZrrW) rrTr2rSrbrcolsidxZ sql_selectrrrXrrrreadcs8       z SQLTable.readcCs|dkr|jjj}|dk rJt|ts*|g}t||krFtd|n|S|dkrrd|jjkrr|jjjdkrrdgSddt |jjj DSn"t|t r|gSt|tr|SdSdS)NTz@Length of 'index_label' should match number of levels, which is rhrcSs&g|]\}}|dk r|nd|qS)NZlevel_r)rrlrrrrsz(SQLTable._index_name..) rZrnlevelsr!r+rrtrSrrrr)rrrrrrrrs2        zSQLTable._index_namecspg}jdk rHtjD].\}}jj|}|t||dfq|fddttjjD7}|S)NTcs6g|].}tjj|jjdd|fdfqS)NF)rrZrSZiloc)rr dtype_mapperrrrrsz8SQLTable._get_column_names_and_types..) rrrZZ_get_level_valuesrrrrrS)rrcolumn_names_and_typesrZ idx_labelZidx_typerrr_get_column_names_and_typess z$SQLTable._get_column_names_and_typesc sddlmm}m}||j}fdd|D}|jdk rtt|jsP|jg}n|j}||d|jdi}| ||j p|j j j }ddl m}||j |d} ||j| f|d |iS) Nr)ColumnPrimaryKeyConstraintTablecs g|]\}}}|||dqS))rr)rrtypis_indexrrrrsz0SQLTable._create_table_setup..rZ_pkrirkrf)rrrrr_sqlalchemy_typer)rrrrfrrmrqrj) rrrrrSr)Zpkcrfrjrmrrrrs       zSQLTable._create_table_setupc Cs(t|}|jjD]}|j}z|j|}||krlz ||}Wntk rTd}YnXt||d|j|<Wq||j}|t ks|t ks|t kr|t k}t||d|j|<n\|t kr|j |dd|j|<n>t||kr|tdks|tkr|j |dd|j|<Wqtk r YqXqdS)a Make the DataFrame's column types align with the SQL table column types. Need to work around limited NA value support. Floats are always fine, ints must always be floats if there are Null values. Booleans are hard because converting bool column with None replaces all Nones with false. Therefore only convert bool if there are no NA values. Datetimes should already be converted to np.datetime64 if supported, but here we also force conversion if required. NrKr@F)rint64)r3rxrSrrZrMrJ _get_dtyperErrrfloatrrcountrFrDboolKeyError)rr2Zsql_colrNrOrPcol_typer?rrrrs8     zSQLTable._harmonize_columnsc Csd|jpi}|j|kr |j|jStj|dd}ddlm}m}m}m}m }m } m } m } m } |dksj|dkrz|jjdk r|ddWSWn2tk rt|dddk r|ddYSYnX|S|d krtjd td d |S|d kr|jdkr| ddS| ddSn\|dkr$|jdkr| S|Sn<|dkr2|S|dkr@|S|dkrN| S|dkr`td| S)NTZskipnar) TIMESTAMP BigIntegerBooleanDateDateTimeFloatIntegerTextTime datetime64r)timezonetz timedelta64lthe 'timedelta' type is not supported, and will be written as integer values (ns frequency) to the database. stacklevelrGfloat32)Z precision5rHint32booleanrrcomplexComplex datatypes not supported)rDrlib infer_dtypesqlalchemy.typesrrrrrrrrrdtrAttributeErrorgetattrwarningswarn UserWarningrt) rrIrDrrrrrrrrrrrrrr sJ   ,            zSQLTable._sqlalchemy_typecCsddlm}m}m}m}m}m}t||r.tSt||rBt dSt||rZ|j sVt St St||rht St||rvtSt||rtStS)Nr)rrrrrrr)rrrrrrrr!rrFrDrrrrrr)rZsqltyperrrrrrrrrrHs         zSQLTable._get_dtype)NTrrNNNN)NN)TN)TNNN)N)rrr__doc__rrrrrrrrrrrrrrrrrrrrrrs8 ! $ %  %$ 3?rc@s eZdZdZddZddZdS) PandasSQLz7 Subclasses Should define read_sql and to_sql. cOs tddSNzMPandasSQL must be created with an SQLAlchemy connectable or sqlite connectionrtrr.kwargsrrrr~dszPandasSQL.read_sqlcOs tddSr rrrrrrjszPandasSQL.to_sqlN)rrrr r~rrrrrr _sr c@seZdZdZdddZeddZddZdd d Ze d d d Z d!ddZ e Z d"ddZ eddZd#ddZd$ddZd%ddZd&eeeeeeeeedddZdS)'rua This class enables conversion between DataFrame and SQL databases using SQLAlchemy to handle DataBase abstraction. Parameters ---------- engine : SQLAlchemy connectable Connectable to connect with the database. Using SQLAlchemy makes it possible to use any DB supported by that library. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). meta : SQLAlchemy MetaData object, default None If provided, this MetaData object is used instead of a newly created. This allows to specify database flavor specific arguments in the MetaData object. NcCs.||_|s$ddlm}||j|d}||_dS)Nrrirk)rrqrjrm)rr"rfrmrjrrrrs  zSQLDatabase.__init__c cs4|j }t|dr|Vn|jVW5QRXdS)Nr_)rbeginr*)rZtxrrrrs  zSQLDatabase.run_transactioncOs|jj||S)z,Simple passthrough to SQLAlchemy connectable)rZexecution_optionsr_rrrrr_szSQLDatabase.executeTc Cs"t||||d}|j||||dS)a Read SQL database table into a DataFrame. Parameters ---------- table_name : string Name of SQL table in database. index_col : string, optional, default: None Column to set as index. coerce_float : boolean, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. This can result in loss of precision. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg}``, where the arg corresponds to the keyword arguments of :func:`pandas.to_datetime`. Especially useful with databases without native Datetime support, such as SQLite. columns : list, default: None List of column names to select from SQL table. schema : string, default None Name of SQL schema in database to query (if database flavor supports this). If specified, this overwrites the default schema of the SQL database object. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame See Also -------- pandas.read_sql_table SQLDatabase.read_query )rrf)rTr2rSrb)rr) rrerYrTr2rSrfrbrxrrrrvs4zSQLDatabase.read_tableccs,||}|sq(qt|||||dVqdSz+Return generator through chunked result setrYrTr2N)rr[)rrbrSrYrTr2rXrrrrs zSQLDatabase._query_iteratorc Cs\t||}|j|}|} |dk r:|j||| |||dS|} t| | |||d} | SdS)a$ Read SQL query into a DataFrame. Parameters ---------- sql : string SQL query to be executed. index_col : string, optional, default: None Column name to use as index for the returned DataFrame object. coerce_float : boolean, default True Attempt to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'} parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql Nr)r/r_r)rrr[) rr,rYrTr2r-rbr.rrSrXrZrrrr|s*2   zSQLDatabase.read_queryrc  srtsfdd|Ddk rfddlm} m} D]&\} } t| | | s>td| dq>t|||||||d}|dd l m }z|j || d WnN|j k r}z.d }t |j}t||rtd |n|W5d}~XYnX|s|s|jj}|jR}trPdd l m}||}|j|pH|jjd}n|j|p`|jj|d}W5QRX||krd|d}t|tdS)a Write records stored in a DataFrame to a SQL database. Parameters ---------- frame : DataFrame name : string Name of SQL table. if_exists : {'fail', 'replace', 'append'}, default 'fail' - fail: If table exists, do nothing. - replace: If table exists, drop it, recreate it, and insert data. - append: If table exists, insert data. Create if does not exist. index : boolean, default True Write DataFrame index as a column. index_label : string or sequence, default None Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If specified, this overwrites the default schema of the SQLDatabase object. chunksize : int, default None If not None, then rows will be written in batches of this size at a time. If None, all rows will be written at once. dtype : single type or dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a SQLAlchemy type. If all columns are of the same type, one single value can be used. method : {None', 'multi', callable}, default None Controls the SQL insertion clause used: * None : Uses standard SQL ``INSERT`` clause (one per row). * 'multi': Pass multiple values in a single ``INSERT`` clause. * callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 csi|] }|qSrrrrNrDrr ksz&SQLDatabase.to_sql..Nr) TypeEngine to_instancez The type of z is not a SQLAlchemy type)rZrrrrfrD)rs)rz.(1054, "Unknown column 'inf' in 'field list'")zinf cannot be used with MySQL)inspectrk)rf connectionzThe provided table name 'z' is not found exactly as such in the database after writing the table, possibly due to case sensitivity issues. Consider using lower case table names.)rrrrrLr!rtrrrrsrZSQLAlchemyErrorrorigresearchisdigitislowerrr"connectr(rZget_table_namesrmrf table_namesrr r )rrZrrrrrfrbrDrrrrImy_typerxrsrwmsgZerr_textr"rrinspr!rrrr5sX5            zSQLDatabase.to_sqlcCs|jjSrc)rmtablesrrrrr%szSQLDatabase.tablescCsPtr.ddl}||j}|||p*|jjS|j|jjj||pH|jjSdS)Nr) r(rrrrrmrfZ run_callabledialect)rrrfsar$rrrrs  zSQLDatabase.has_tablecCsh|p |jj}|r*|jjd||g}n|jj|}ddlm}|jD]}t|j |rJd|j _ qJ|S)N.r)NumericF) rmrfr%getjoinrr)rSr!rEZ asdecimal)rrerfZtblr)columnrrrrs     zSQLDatabase.get_tablecCsH|p |jj}|||rD|jj|g|d||||jdS)N)rlrf)rmrfrrrrZdropclearrrerfrrrrs   zSQLDatabase.drop_table)rZrer)rDrfc Cs"t|||d|||d}t|SNF)rZrr)rDrf)rrrrrZrer)rDrfrxrrr_create_sql_schemas zSQLDatabase._create_sql_schema)NN)NTNNNN)NTN)NTNNN)rTNNNNN)N)N)N)NNN)rrrr rrrr_rv staticmethodrr|r~rpropertyr%rrrrrr r rBr1rrrrruqs^   <  K q    ruZTEXTZREALZINTEGERrZDATEZTIME)stringrGrHrrrrc CsTzt|ddd}Wn4tk rN}ztd|d|W5d}~XYnX|S)Nzutf-8strictz%Cannot convert identifier to UTF-8: 'r)rencodedecode UnicodeErrorrt)runamerwrrr_get_unicode_names $r:cCsFt|}t|std|d}|dkr2tdd|dddS)Nz$Empty table or column name specifiedrz%SQLite identifier cannot contain NULs"z"")r:rrtfindr)rr9Z nul_indexrrr_get_valid_sqlite_names r>zvThe spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.csXeZdZdZfddZddZddZdd Zd d Zd d Z ddZ ddZ Z S) SQLiteTablezw Patch the SQLTable for fallback support. Instead of a table variable just use the Create Table statement. cs*ddl}|tddtj||dS)NrcSs |dS)Nz %H:%M:%S.%f)strftime)rrrrz&SQLiteTable.__init__..)sqlite3Zregister_adapterrsuperr)rr.rrC __class__rrrszSQLiteTable.__init__cCstd|jS)Nz; )rr+rxrrrrrszSQLiteTable.sql_schemac Cs0|j}|jD]}||qW5QRXdSrc)rrrxr_)rrZstmtrrrr!s  zSQLiteTable._execute_createc sttt|jj}d}t|jdk rF|jdddD]}|d|q4fdd|D}d|}d|gt |dfddt |D}d |j d |d |}|S) N?rrcsg|] }|qSrr)rr,escaperrr/sz0SQLiteTable.insert_statement..,c3s|]}ddVqdS)()Nr)rr) row_wildcardsrr 3sz/SQLiteTable.insert_statement..z INSERT INTO  (z ) VALUES ) r+rrrZrSr>rrr+rrr) rnum_rowsrwldrZbracketed_namesZ col_names wildcardsinsert_statementr)rIrMrrS&s  zSQLiteTable.insert_statementcCs t|}||jdd|dS)NrhrP)r+Z executemanyrS)rrr)rrrrrr9szSQLiteTable._execute_insertcCs2t|}dd|D}||jt|d|dS)NcSsg|]}|D]}|q qSrr)rrxrrrr?sz5SQLiteTable._execute_insert_multi..rT)r+r_rSr)rrr)rrZflattened_datarrrr=sz!SQLiteTable._execute_insert_multic sl||j}td}dd|D}tt|j|rBtjt ddt fdd|D}|j dk rt |j rt |j s|j g}n|j }dfd d |D}|d |jd |d |jr|jd}nd}d||jdd|dg}dd|D} t | rhd| } dfdd | D}|dd|jd| d|jd|d |S)z Return a list of SQL statements that creates a table reflecting the structure of a DataFrame. The first entry will be a CREATE TABLE statement while the rest will be CREATE INDEX statements. z\s+cSsg|]\}}}|qSrr)rrNrrrrrKsz3SQLiteTable._create_table_setup..rcs"g|]\}}}|d|qS) r)rcnamectyperrHrrrQsNz, c3s|]}|VqdSrcrrrrHrrrNZsz2SQLiteTable._create_table_setup..z CONSTRAINT z_pk PRIMARY KEY (rLr(z CREATE TABLE z ( z, z )cSsg|]\}}}|r|qSrr)rrXrrrrrrksrrJc3s|]}|VqdSrcrrZrHrrrNnsz CREATE INDEX Zix_zON rO)r_sql_type_namerranyrrrr _SAFE_NAMES_WARNINGr>r)rrr+rrrf) rrpatrZcreate_tbl_stmtsr)Z cnames_brZ schema_nameZ create_stmtsZix_colsZcnamesrrHrrBsh          zSQLiteTable._create_table_setupcCs|jpi}|j|kr||jStj|dd}|dkrJtjdtddd}n,|dkrXd }n|d krfd }n|d krvtd |tkrd }t|S)NTrrrrrrHrremptyr4rr) rDrrrrr r rt _SQL_TYPES)rrIrDrrrrr\{s(   zSQLiteTable._sql_type_name) rrrr rrrrSrrrr\ __classcell__rrrErr?s 9r?c@seZdZdZdddZeddZddZedd d Z dd dZ ddZ dddZ d ddZ d!ddZd"ddZd#ddZd S)$rz Version of SQLDatabase to support SQLite connections (fallback without SQLAlchemy). This should only be used internally. Parameters ---------- con : sqlite connection object FcCs||_||_dSrc)r]r$)rr$r]rrrrszSQLiteDatabase.__init__ccsT|j}z:z|V|jWn tk r@|jYnXW5|XdSrc)r$cursorclosecommitrrollback)rr`rrrrs   zSQLiteDatabase.run_transactionc Os|jr|j}n |j}z|j|||WStk r}ztz|jWnBtk r}z$td|dd|d}||W5d}~XYnXtd|dd|}||W5d}~XYnXdS)NzExecution failed on sql: r z unable to rollbackzExecution failed on sql 'z': )r]r$rcr_rrfr)rr.rr`rsZ inner_excexrrrr_s   zSQLiteDatabase.executeNTccsH||}t|tkrt|}|s.|qDqt|||||dVqdSr)rrEtupler+rdr[)rcrbrSrYrTr2rXrrrrs  zSQLiteDatabase._query_iteratorc Csnt||}|j|}dd|jD} |dk rB|j||| |||dS||} |t| | |||d} | SdS)NcSsg|] }|dqS)rr)rZcol_descrrrrsz-SQLiteDatabase.read_query..r)r/r_ descriptionr_fetchall_as_listrdr[) rr,rYrTr-r2rbr.rcrSrXrZrrrr|s,   zSQLiteDatabase.read_querycCs|}t|tst|}|Src)rr!r+)rr`rrrrrks z SQLiteDatabase._fetchall_as_listrc srtsfdd|Ddk rVD]&\} } t| ts.t| d| dq.t||||||d} | | || dS)ai Write records stored in a DataFrame to a SQL database. Parameters ---------- frame: DataFrame name: string Name of SQL table. if_exists: {'fail', 'replace', 'append'}, default 'fail' fail: If table exists, do nothing. replace: If table exists, drop it, recreate it, and insert data. append: If table exists, insert data. Create if it does not exist. index : boolean, default True Write DataFrame index as a column index_label : string or sequence, default None Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. schema : string, default None Ignored parameter included for compatibility with SQLAlchemy version of ``to_sql``. chunksize : int, default None If not None, then rows will be written in batches of this size at a time. If None, all rows will be written at once. dtype : single type or dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a string. If all columns are of the same type, one single value can be used. method : {None, 'multi', callable}, default None Controls the SQL insertion clause used: * None : Uses standard SQL ``INSERT`` clause (one per row). * 'multi': Pass multiple values in a single ``INSERT`` clause. * callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 csi|] }|qSrrrrrrr?sz)SQLiteDatabase.to_sql..NrOz) not a string)rZrrrrD)rrLr!rrtr?rr) rrZrrrrrfrbrDrrIr"rxrrrr s"4   zSQLiteDatabase.to_sqlcCs*d}d|d}t|||gdkS)NrGz;SELECT name FROM sqlite_master WHERE type='table' AND name=;r)rr_r)rrrfrQqueryrrrrRs zSQLiteDatabase.has_tablecCsdSrcrr.rrrr\szSQLiteDatabase.get_tablecCsdt|}||dS)Nz DROP TABLE )r>r_)rrrfZdrop_sqlrrrr_szSQLiteDatabase.drop_tablec Cs"t|||d|||d}t|Sr/)r?rrr0rrrr1cs z!SQLiteDatabase._create_sql_schema)F)NTN)NTNNN)rTNNNNN)N)N)N)NNN)rrrr rrrr_r2rr|rkrrrrr1rrrrrs:    $  H  rcCst|d}|j|||||dS)a2 Get the SQL db table schema for the given frame. Parameters ---------- frame : DataFrame name : string name of SQL table keys : string or sequence, default: None columns to use a primary key con: an open SQL database connection object or a SQLAlchemy connectable Using SQLAlchemy makes it possible to use any DB supported by that library, default: None If a DBAPI2 object, only sqlite3 is supported. dtype : dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a SQLAlchemy type, or a string for sqlite3 fallback connection. schema: str, default: None Optional specifying the schema to be used in creating the table. .. versionadded:: 1.2.0 )r$)r)rDrf)r^r1)rZrr)r$rDrfrarrr get_schemaps rn)NN)NTN)NN)NNTNNN)NNTNNrh)NNTNNN)NTNNN)NTNNrh)NTNNN)NTNNNN)NTNNNrh)NTNNNN)NrTNNNN)N)NNF)NNNN)Gr  contextlibrrrrZdistutils.versionr functoolsrrtypingrr r r r rnumpyrFZpandas._libs.libZ_libsrZpandas.core.dtypes.commonr rrZpandas.core.dtypes.dtypesrZpandas.core.dtypes.missingrZpandas.core.apirrZpandas.core.baserZpandas.core.tools.datetimesrr rIOErrorrrr%rr(r/r3rJrQr[r_rgrrzr~rrZ table_existsror^rr rurar:r>r^r?rrnrrrrs<             _  H  m  S  ?u U