B @`@sdZddlmZddlmZmZmZddlmZddlZddl m Z m Z m Z m Z mZddlZddlZddlmmZddlmZmZmZddlmZdd lmZdd lmZmZdd l m!Z!dd l"m#Z#Gd dde$Z%Gddde&Z'da(ddZ)ddZ*ddZ+dQddZ,ddZ-dRddZ.dSddZ/edTded d!d"Z0edUe1e ed d$d"Z0dVe e1e ee efd d%d"Z0edWded d&d'Z2edXe1e ed d(d'Z2dYe e1e ee efd d)d'Z2edZded d*d+Z3ed[e1e ed d,d+Z3d\e e1e ee efd d-d+Z3d]dd/d0d1Z4d^d2d3Z5e5Z6d4d5Z7d_d7d8Z8Gd9d:d:e!Z9Gd;d<dd>e:Z;d?d@dAdBdCdDdAdEZdJZ?GdKdLdLe9Z@GdMdNdNe:ZAd`dOdPZBdS)az| Collection of query wrappers / abstractions to both facilitate data retrieval and to reduce dependency on DB-specific API. )contextmanager)datedatetimetime)partialN)IteratorListOptionalUnionoverload)is_datetime64tz_dtype is_dict_like is_list_like)DatetimeTZDtype)isna) DataFrameSeries) PandasObject) to_datetimec@s eZdZdS)SQLAlchemyRequiredN)__name__ __module__ __qualname__rr1/tmp/pip-unpacked-wheel-q9tj5l6a/pandas/io/sql.pyrsrc@s eZdZdS) DatabaseErrorN)rrrrrrrrsrcCsTtdkr2yddl}daWntk r0daYnXtrLddl}t||jjSdSdS)NrTF)_SQLALCHEMY_INSTALLED sqlalchemy ImportError isinstanceengineZ Connectable)conrrrr_is_sqlalchemy_connectable(s r"cCs6|g}|dk r2t|dr$||g7}n|t|g7}|S)z9Convert SQL and params args to DBAPI2.0 compliant format.Nkeys)hasattrlist)sqlparamsargsrrr_convert_params:s   r)cCs2|dks|dks|dkrg}nt|ds.|g}|S)z3Process parse_dates argument for read_sql functionsTNF__iter__)r$) parse_datesrrr_process_parse_dates_argumentEs  r,cCst|trt|fddi|S|dkrJt|jjtjsFt|jjtjrJd}|dkrbt|d||dSt |jrxt|ddSt|d||d SdS) Nerrorsignores)Ddhmr/msusnsZcoerce)r-unitutcT)r8)r-formatr8) rdictr issubclassdtypetypenpfloatingintegerr )colr8r9rrr_handle_date_columnPs   rBc Csjt|}x\|D]P\}}t|js,||kry ||}Wntk rPd}YnXt||d||<qW|S)zz Force non-datetime columns to be read as such. Supports both string formatted and integer timestamp columns. N)r9)r,itemsr r< TypeErrorrB)Z data_framer+col_namedf_colfmtrrr_parse_date_columnses  rHTcCs4tj|||d}t||}|dk r0|j|dd|S)z(Wrap result set of query in a DataFrame.)columns coerce_floatNT)inplace)r from_recordsrH set_index)datarI index_colrJr+framerrr _wrap_resultzs  rQcCs2|dkrt|}n t|dd}t||}|j|S)aY Execute the given SQL query using the provided connection object. Parameters ---------- sql : string SQL query to be executed. con : SQLAlchemy connectable(engine/connection) or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by the library. If a DBAPI2 object, only sqlite3 is supported. cur : deprecated, cursor is obtained from connection, default: None params : list or tuple, optional, default: None List of parameters to pass to execute method. Returns ------- Results Iterable NT) is_cursor)pandasSQL_builderr)execute)r&r!curr' pandas_sqlr(rrrrTs    rT) chunksizereturncCsdS)Nr) table_namer!schemarOrJr+rIrWrrrread_sql_tables r[cCsdS)Nr)rYr!rZrOrJr+rIrWrrrr[s c Cst|}t|stdddl}ddlm} | ||d} y| j|gddWn8|jjk r} zt d|d | Wdd} ~ XYnXt || d } | j ||||||d } | dk r| St d|d |dS) a~ Read SQL database table into a DataFrame. Given a table name and a SQLAlchemy connectable, returns a DataFrame. This function does not support DBAPI connections. Parameters ---------- table_name : str Name of SQL table in database. con : SQLAlchemy connectable or str A database URI could be provided as str. SQLite DBAPI connection mode not supported. schema : str, default None Name of SQL schema in database to query (if database flavor supports this). Uses default schema if None (default). index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. Can result in loss of Precision. parse_dates : list or dict, default None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. columns : list, default None List of column names to select from SQL table. chunksize : int, default None If specified, returns an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] A SQL table is returned as two-dimensional data structure with labeled axes. See Also -------- read_sql_query : Read SQL query into a DataFrame. read_sql : Read SQL query or database table into a DataFrame. Notes ----- Any datetime values with time zone information will be converted to UTC. Examples -------- >>> pd.read_sql_table('table_name', 'postgres:///db_name') # doctest:+SKIP z9read_sql_table only supported for SQLAlchemy connectable.rN)MetaData)rZT)onlyZviewszTable z not found)meta)rOrJr+rIrW) _engine_builderr"NotImplementedErrorrsqlalchemy.schemar]reflectexcZInvalidRequestError ValueError SQLDatabase read_table)rYr!rZrOrJr+rIrWrr]r_errrVtablerrrr[s,A  $ cCsdS)Nr)r&r!rOrJr'r+rWrrrread_sql_query!s rjcCsdS)Nr)r&r!rOrJr'r+rWrrrrj.s cCst|}|j||||||dS)a7 Read SQL query into a DataFrame. Returns a DataFrame corresponding to the result set of the query string. Optionally provide an `index_col` parameter to use one of the columns as the index, otherwise default integer index will be used. Parameters ---------- sql : str SQL query or SQLAlchemy Selectable (select or text object) SQL query to be executed. con : SQLAlchemy connectable, str, or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. Useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql : Read SQL query or database table into a DataFrame. Notes ----- Any datetime values with time zone information parsed via the `parse_dates` parameter will be converted to UTC. )rOr'rJr+rW)rS read_query)r&r!rOrJr'r+rWrVrrrrj;s=cCsdS)Nr)r&r!rOrJr'r+rIrWrrrread_sqls rlcCsdS)Nr)r&r!rOrJr'r+rIrWrrrrls c Cst|}t|tr(|j||||||dSy||} Wntk rNd} YnX| rz|jj|gd|j||||||dS|j||||||dSdS)a Read SQL query or database table into a DataFrame. This function is a convenience wrapper around ``read_sql_table`` and ``read_sql_query`` (for backward compatibility). It will delegate to the specific function depending on the provided input. A SQL query will be routed to ``read_sql_query``, while a database table name will be routed to ``read_sql_table``. Note that the delegated function might have more specific notes about their functionality not listed here. Parameters ---------- sql : str or SQLAlchemy Selectable (select or text object) SQL query to be executed or a table name. con : SQLAlchemy connectable, str, or sqlite3 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. The user is responsible for engine disposal and connection closure for the SQLAlchemy connectable; str connections are closed automatically. See `here `_. index_col : str or list of str, optional, default: None Column(s) to set as index(MultiIndex). coerce_float : bool, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. columns : list, default: None List of column names to select from SQL table (only used when reading a table). chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame or Iterator[DataFrame] See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql_query : Read SQL query into a DataFrame. )rOr'rJr+rWF)r^)rOrJr+rIrWN) rSrSQLiteDatabaserk has_table Exceptionr_rcrg) r&r!rOrJr'r+rIrWrVZ_is_table_namerrrrls:B  fail)rXc Csj|dkrtd|dt||d} t|tr8|}nt|tsJtd| j||||||||| d dS)a; Write records stored in a DataFrame to a SQL database. Parameters ---------- frame : DataFrame, Series name : str Name of SQL table. con : SQLAlchemy connectable(engine/connection) or database string URI or sqlite3 DBAPI2 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. schema : str, optional Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). if_exists : {'fail', 'replace', 'append'}, default 'fail' - fail: If table exists, do nothing. - replace: If table exists, drop it, recreate it, and insert data. - append: If table exists, insert data. Create if does not exist. index : boolean, default True Write DataFrame index as a column. index_label : str or sequence, optional Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. chunksize : int, optional Specify the number of rows in each batch to be written at a time. By default, all rows will be written at once. dtype : dict or scalar, optional Specifying the datatype for columns. If a dictionary is used, the keys should be the column names and the values should be the SQLAlchemy types or strings for the sqlite3 fallback mode. If a scalar is provided, it will be applied to all columns. method : {None, 'multi', callable}, optional Controls the SQL insertion clause used: - None : Uses standard SQL ``INSERT`` clause (one per row). - 'multi': Pass multiple values in a single ``INSERT`` clause. - callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 )rpreplaceappend'z' is not valid for if_exists)rZz9'frame' argument should be either a Series or a DataFrame) if_existsindex index_labelrZrWr<methodN)rerSrrZto_framerrato_sql) rPnamer!rZrtrurvrWr<rwrVrrrrxs$:    rxcCst||d}||S)a- Check if DataBase has named table. Parameters ---------- table_name: string Name of SQL table. con: SQLAlchemy connectable(engine/connection) or sqlite3 DBAPI2 connection Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). Returns ------- boolean )rZ)rSrn)rYr!rZrVrrrrn[s rncCsBt|tr>y ddl}Wntk r.daYnX||}|S|S)zw Returns a SQLAlchemy engine from a URI (if con is a string) else it just return con without modifying it. rNF)rstrrrrZ create_engine)r!rrrrr`vs    r`FcCsBt|}t|rt|||dSt|tr2tdn t||dSdS)zm Convenience function to return the correct PandasSQL subclass based on the provided parameters. )rZr_z.Using URI string without sqlalchemy installed.)rRN)r`r"rfrrzrrm)r!rZr_rRrrrrSs   rSc@seZdZdZd(ddZdd Zd d Zd d ZddZddZ ddZ ddZ d)ddZ d*ddZ d+ddZddZddZd d!Zd,d"d#Zd$d%Zd&d'ZdS)-SQLTablez For mapping Pandas tables to SQL tables. Uses fact that table is reflected by SQLAlchemy to do better type conversions. Also holds various flags needed to avoid having to pass them between functions all the time. NTrppandasc Cs||_||_||_||_||||_||_||_| |_| |_ |dk rR| |_ n|j |j|j|_ |j dkrt d|ddS)NzCould not init table 'rs)rypd_sqlprefixrP _index_namerurZrtr#r<_create_table_setupri get_tablere) selfryZpandas_sql_enginerPrurtr~rvrZr#r<rrr__init__s   zSQLTable.__init__cCs|j|j|jS)N)r}rnryrZ)rrrrexistsszSQLTable.existscCs$ddlm}t||j|jjS)Nr) CreateTable)rbrrzricompiler} connectable)rrrrr sql_schemas zSQLTable.sql_schemacCs |j|jj|_|jdS)N)riZ tometadatar}r_create)rrrr_execute_createszSQLTable._execute_createcCsx|rl|jdkr&td|jdqt|jdkrL|j|j|j|qt|jdkrXqttd|jdn|dS)NrpzTable 'z' already exists.rqrrrsz' is not valid for if_exists)rrtreryr} drop_tablerZr)rrrrrs    zSQLTable.createcs(fdd|D}||j|dS)a< Execute SQL statement inserting data Parameters ---------- conn : sqlalchemy.engine.Engine or sqlalchemy.engine.Connection keys : list of str Column names data_iter : generator of list Each item contains a list of values to be inserted csg|]}tt|qSr)r:zip).0row)r#rr sz,SQLTable._execute_insert..N)rTriinsert)rconnr# data_iterrNr)r#r_execute_inserts zSQLTable._execute_insertcs(fdd|D}||j|dS)a Alternative to _execute_insert for DBs support multivalue INSERT. Note: multi-value insert is usually faster for analytics DBs and tables containing a few columns but performance degrades quickly with increase of columns. csg|]}tt|qSr)r:r)rr)r#rrrsz2SQLTable._execute_insert_multi..N)rTrir)rrr#rrNr)r#r_execute_insert_multiszSQLTable._execute_insert_multic Cs0|jdk rd|j}|j|j_y|jddWqjtk r`}ztd||Wdd}~XYqjXn|j}ttt|j }t |}dg|}xt | D]\}\}}|j } | jjdkr| } n(| jjdkr| dt} n | t} t| tjstt| |jrt| } d| | <| ||<qW||fS)NT)rKz!duplicate name in index/columns: Mr3i8)rurPcopynamesZ reset_indexrer%maprzrIlen enumeraterCZ_valuesr<kindZ to_pydatetimeviewastypeobjectrr>ZndarrayAssertionErrorr=Z _can_hold_nar) rtemprh column_namesZncols data_listi_Zservalsr1maskrrr insert_datas0   $      zSQLTable.insert_datac s|dkr|j}n2|dkr |j}n"t|r4t||}ntd||\}}t|j}|dkrddS|dkrr|}n|dkrtdt||d}|j `}xXt |D]L} | |t | d||krPt fdd|D} |||| qWWdQRXdS)NZmultizInvalid parameter `method`: rz%chunksize argument should be non-zeror\csg|]}|qSrr)rZarr)end_istart_irrr>sz#SQLTable.insert..)rrcallablerrerrrPintr}run_transactionrangeminr) rrWrwZ exec_insertr#rZnrowschunksrrZ chunk_iterr)rrrrs0    zSQLTable.insertccs^xX||}|sPqtj|||d|_|j|d|jdk rN|jj|jdd|jVqWdS)z,Return generator through chunked result set.)rIrJ)r+NT)rK) fetchmanyrrLrP_harmonize_columnsrurM)rresultrWrIrJr+rNrrr_query_iteratorAs   zSQLTable._query_iteratorc s|dk rtt|dkrtddlm}fdd|D}jdk rjx,jdddD]}|djj|qNW||}n j}j|} | } |dk rj | || ||dS| } t j | | |d_j|djdk rjjjd d jSdS) Nr)selectcsg|]}jj|qSr)ric)rn)rrrrZsz!SQLTable.read..)rJr+)rIrJ)r+T)rK)rrrrurrirr}rTr#rfetchallrrLrPrrM) rrJr+rIrWrcolsidxZ sql_selectrrrNr)rrreadUs0       z SQLTable.readcCs|dkr|jjj}|dk rJt|ts*|g}t||krFtd|n|S|dkrrd|jjkrr|jjjdkrrdgSddt |jjj DSn"t|t r|gSt|tr|SdSdS)NTz@Length of 'index_label' should match number of levels, which is r\rucSs&g|]\}}|dk r|nd|qS)NZlevel_r)rrlrrrrsz(SQLTable._index_name..) rPrunlevelsrr%rrerIryrrrz)rrurvrrrrrzs(      zSQLTable._index_namecstg}jdk rLx.) rurrPZ_get_level_valuesrrrzrrrI)rrcolumn_names_and_typesrZ idx_labelZidx_typer)rrr_get_column_names_and_typess  z$SQLTable._get_column_names_and_typesc sddlmm}m}||j}fdd|D}|jdk rtt|jsP|jg}n|j}||d|jdi}| ||j p|j j j }ddl m}||j |d} ||j| f|d |iS) Nr)ColumnPrimaryKeyConstraintTablecs g|]\}}}|||dqS))rur)rrytypis_index)rrrrsz0SQLTable._create_table_setup..ryZ_pk)r])rZrZ)rrrrr_sqlalchemy_typer#rryrrrZr}r_rbr]) rrrrrIr#ZpkcrZr]r_r)rrrs       zSQLTable._create_table_setupc Cs,t|}x|jjD]}|j}y|j|}||krny ||}Wntk rXd}YnXt||d|j|<w||j}|t ks|t ks|t kr|t k}t||d|j|<n\|t kr|j |dd|j|<n>t||kr |tdks|tkr |j |dd|j|<Wqtk r"YqXqWdS)a Make the DataFrame's column types align with the SQL table column types. Need to work around limited NA value support. Floats are always fine, ints must always be floats if there are Null values. Booleans are hard because converting bool column with None replaces all Nones with false. Therefore only convert bool if there are no NA values. Datetimes should already be converted to np.datetime64 if supported, but here we also force conversion if required. N)r9)r8F)rint64)r,rirIryrPrDrB _get_dtyper=rrrfloatrrcountr>r<boolKeyError)rr+Zsql_colrErFrGcol_typer8rrrrs2     zSQLTable._harmonize_columnsc Cs^|jpi}|j|kr |j|jStj|dd}ddlm}m}m}m}m }m } m } m } m } |dksj|dkry|jjdk r|ddSWn.tk rt|dddk r|ddSYnX|S|d krtjd td d |S|d kr|jdkr| ddS| ddSn\|dkr|jdkr| S|Sn<|dkr,|S|dkr:|S|dkrH| S|dkrZtd| S)NT)skipnar) TIMESTAMP BigIntegerBooleanDateDateTimeFloatIntegerTextTime datetime64r)timezonetz timedelta64zlthe 'timedelta' type is not supported, and will be written as integer values (ns frequency) to the database.) stacklevelr?Zfloat32)Z precision5r@Zint32booleanrrcomplexzComplex datatypes not supported)r<rylib infer_dtypesqlalchemy.typesrrrrrrrrrdtrAttributeErrorgetattrwarningswarn UserWarningre) rrAr<rrrrrrrrrrrrrrsH   ,            zSQLTable._sqlalchemy_typecCsddlm}m}m}m}m}m}t||r.tSt||rBt dSt||rZ|j sVt St St||rht St||rvtSt||rtStS)Nr)rrrrrrr)rrrrrrrrrr>r<rrrrrr)rZsqltyperrrrrrrrrr:s         zSQLTable._get_dtype)NTrpr|NNNN)NN)TN)TNNN)N)rrr__doc__rrrrrrrrrrrrrrrrrrrrrr{s2  $ %  %$ 3?r{c@s eZdZdZddZddZdS) PandasSQLz7 Subclasses Should define read_sql and to_sql. cOs tddS)NzMPandasSQL must be created with an SQLAlchemy connectable or sqlite connection)re)rr(kwargsrrrrlVszPandasSQL.read_sqlcOs tddS)NzMPandasSQL must be created with an SQLAlchemy connectable or sqlite connection)re)rr(rrrrrx\szPandasSQL.to_sqlN)rrrrrlrxrrrrrQsrc@seZdZdZdddZeddZddZdd d Ze d d d Z d!ddZ e Z d"ddZ eddZd#ddZd$ddZd%ddZd&eeeeeeeeedddZdS)'rfa This class enables conversion between DataFrame and SQL databases using SQLAlchemy to handle DataBase abstraction. Parameters ---------- engine : SQLAlchemy connectable Connectable to connect with the database. Using SQLAlchemy makes it possible to use any DB supported by that library. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If None, use default schema (default). meta : SQLAlchemy MetaData object, default None If provided, this MetaData object is used instead of a newly created. This allows to specify database flavor specific arguments in the MetaData object. NcCs.||_|s$ddlm}||j|d}||_dS)Nr)r])rZ)rrbr]r_)rr rZr_r]rrrrws  zSQLDatabase.__init__c cs4|j }t|dr|Vn|jVWdQRXdS)NrT)rbeginr$)rZtxrrrrs  zSQLDatabase.run_transactioncOs|jj||S)z,Simple passthrough to SQLAlchemy connectable)rZexecution_optionsrT)rr(rrrrrTszSQLDatabase.executeTc Cs"t||||d}|j||||dS)a Read SQL database table into a DataFrame. Parameters ---------- table_name : string Name of SQL table in database. index_col : string, optional, default: None Column to set as index. coerce_float : boolean, default True Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. This can result in loss of precision. parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg}``, where the arg corresponds to the keyword arguments of :func:`pandas.to_datetime`. Especially useful with databases without native Datetime support, such as SQLite. columns : list, default: None List of column names to select from SQL table. schema : string, default None Name of SQL schema in database to query (if database flavor supports this). If specified, this overwrites the default schema of the SQL database object. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame See Also -------- pandas.read_sql_table SQLDatabase.read_query )rurZ)rJr+rIrW)r{r) rrYrOrJr+rIrZrWrirrrrgs 4zSQLDatabase.read_tableccs0x*||}|sPqt|||||dVqWdS)z+Return generator through chunked result set)rOrJr+N)rrQ)rrWrIrOrJr+rNrrrrs zSQLDatabase._query_iteratorc Cs\t||}|j|}|} |dk r:|j||| |||dS|} t| | |||d} | SdS)a$ Read SQL query into a DataFrame. Parameters ---------- sql : string SQL query to be executed. index_col : string, optional, default: None Column name to use as index for the returned DataFrame object. coerce_float : boolean, default True Attempt to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. params : list, tuple or dict, optional, default: None List of parameters to pass to execute method. The syntax used to pass parameters is database driver dependent. Check your database driver documentation for which of the five syntax styles, described in PEP 249's paramstyle, is supported. Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'} parse_dates : list or dict, default: None - List of column names to parse as dates. - Dict of ``{column_name: format string}`` where format string is strftime compatible in case of parsing string times, or is one of (D, s, ns, ms, us) in case of parsing integer timestamps. - Dict of ``{column_name: arg dict}``, where the arg dict corresponds to the keyword arguments of :func:`pandas.to_datetime` Especially useful with databases without native Datetime support, such as SQLite. chunksize : int, default None If specified, return an iterator where `chunksize` is the number of rows to include in each chunk. Returns ------- DataFrame See Also -------- read_sql_table : Read SQL database table into a DataFrame. read_sql N)rOrJr+)r)rTr#rrrQ) rr&rOrJr+r'rWr(rrIrNrPrrrrks&2  zSQLDatabase.read_queryrpc  shrtsfdd|Ddk rjddlm} m} x2D]&\} } t| | | s@td| dq@Wt|||||||d}|dd l m }y|j || d WnN|j k r}z.d }t |j}t||rtd |n|Wdd}~XYnX|sd|sd|jj}|j}|j|p0|jj|d }WdQRX||krdd|d}t|tdS)a Write records stored in a DataFrame to a SQL database. Parameters ---------- frame : DataFrame name : string Name of SQL table. if_exists : {'fail', 'replace', 'append'}, default 'fail' - fail: If table exists, do nothing. - replace: If table exists, drop it, recreate it, and insert data. - append: If table exists, insert data. Create if does not exist. index : boolean, default True Write DataFrame index as a column. index_label : string or sequence, default None Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. schema : string, default None Name of SQL schema in database to write to (if database flavor supports this). If specified, this overwrites the default schema of the SQLDatabase object. chunksize : int, default None If not None, then rows will be written in batches of this size at a time. If None, all rows will be written at once. dtype : single type or dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a SQLAlchemy type. If all columns are of the same type, one single value can be used. method : {None', 'multi', callable}, default None Controls the SQL insertion clause used: * None : Uses standard SQL ``INSERT`` clause (one per row). * 'multi': Pass multiple values in a single ``INSERT`` clause. * callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 csi|] }|qSrr)rrE)r<rr ]sz&SQLDatabase.to_sql..Nr) TypeEngine to_instancez The type of z is not a SQLAlchemy type)rPrurtrvrZr<)rd)rwz.(1054, "Unknown column 'inf' in 'field list'")zinf cannot be used with MySQL)rZ connectionzThe provided table name 'z' is not found exactly as such in the database after writing the table, possibly due to case sensitivity issues. Consider using lower case table names.)r rrrrCrrer{rrrdrZSQLAlchemyErrorrzorigresearchisdigitislowerrr connect table_namesr_rZrrr)rrPryrtrurvrZrWr<rwrrrAmy_typerirdrhmsgZerr_textr rrr)r<rrx'sD5        zSQLDatabase.to_sqlcCs|jjS)N)r_tables)rrrrrszSQLDatabase.tablescCs|j|jjj||p|jjS)N)rZ run_callabledialectrnr_rZ)rryrZrrrrnszSQLDatabase.has_tablecCsl|p |jj}|r*|jjd||g}n|jj|}ddlm}x"|jD]}t|j |rLd|j _ qLW|S)N.r)NumericF) r_rZrgetjoinrrrIrr=Z asdecimal)rrYrZZtblrcolumnrrrrs     zSQLDatabase.get_tablecCsH|p |jj}|||rD|jj|g|d||||jdS)N)r^rZ)r_rZrnrcrZdropclear)rrYrZrrrrs   zSQLDatabase.drop_table)rPrYr#r<rZc Cs"t|||d|||d}t|S)NF)rPrur#r<rZ)r{rzr)rrPrYr#r<rZrirrr_create_sql_schemaszSQLDatabase._create_sql_schema)NN)NTNNNN)NTN)NTNNN)rpTNNNNN)N)N)N)NNN)rrrrrrrrTrg staticmethodrrkrlrxpropertyrrnrrrrzr rr:rrrrrrfcs@  4  D _    rfZTEXTZREALZINTEGERrZDATEZTIME)stringr?r@rrrrc CsTyt|ddd}Wn4tk rN}ztd|d|Wdd}~XYnX|S)Nzutf-8strictz%Cannot convert identifier to UTF-8: 'rs)rzencodedecode UnicodeErrorre)ryunamerhrrr_get_unicode_names $rcCsFt|}t|std|d}|dkr2tdd|dddS)Nz$Empty table or column name specifiedrz%SQLite identifier cannot contain NULs"z"")rrrefindrq)ryr Z nul_indexrrr_get_valid_sqlite_names rzvThe spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.csXeZdZdZfddZddZddZdd Zd d Zd d Z ddZ ddZ Z S) SQLiteTablezw Patch the SQLTable for fallback support. Instead of a table variable just use the Create Table statement. cs*ddl}|tddtj||dS)NrcSs |dS)Nz %H:%M:%S.%f)strftime)rrrrz&SQLiteTable.__init__..)sqlite3Zregister_adapterrsuperr)rr(rr) __class__rrrszSQLiteTable.__init__cCstd|jS)Nz; )rzrri)rrrrrszSQLiteTable.sql_schemac Cs4|j }x|jD]}||qWWdQRXdS)N)r}rrirT)rrZstmtrrrrs  zSQLiteTable._execute_createc sttt|jj}d}t|jdk rJx$|jdddD]}|d|q6Wfdd|D}d|}d|gt |dfddt |D}d |j d |d |}|S) N?rrcsg|] }|qSrr)rr)escaperrrsz0SQLiteTable.insert_statement..,c3s|]}ddVqdS)()Nr)rr) row_wildcardsrr sz/SQLiteTable.insert_statement..z INSERT INTO z (z ) VALUES ) r%rrzrPrIrrurrrrry) rnum_rowsrwldrZbracketed_namesZ col_names wildcardsinsert_statementr)rrrr$ s  zSQLiteTable.insert_statementcCs t|}||jdd|dS)Nr\)r!)r%Z executemanyr$)rrr#rrrrrrszSQLiteTable._execute_insertcCs2t|}dd|D}||jt|d|dS)NcSsg|]}|D]}|q qSrr)rrxrrrr#sz5SQLiteTable._execute_insert_multi..)r!)r%rTr$r)rrr#rrZflattened_datarrrr!sz!SQLiteTable._execute_insert_multic sl||j}td}dd|D}tt|j|rBtjt ddt fdd|D}|j dk rt |j rt |j s|j g}n|j }dfd d |D}|d |jd |d |jr|jd}nd}d||jdd|dg}dd|D} t | rhd| } dfdd | D}|dd|jd| d|jd|d |S)z Return a list of SQL statements that creates a table reflecting the structure of a DataFrame. The first entry will be a CREATE TABLE statement while the rest will be CREATE INDEX statements. z\s+cSsg|]\}}}|qSrr)rrErrrrr/sz3SQLiteTable._create_table_setup..)rcs"g|]\}}}|d|qS) r)rcnamectyper)rrrr6sNz, c3s|]}|VqdS)Nr)rr)rrrr >sz2SQLiteTable._create_table_setup..z CONSTRAINT z_pk PRIMARY KEY (rrz CREATE TABLE z ( z, z )cSsg|]\}}}|r|qSrr)rr(rrrrrrOsrrc3s|]}|VqdS)Nr)rr)rrrr Rsz CREATE INDEX Zix_zON z ()r_sql_type_namerranyrrrr_SAFE_NAMES_WARNINGrr#rrrrrryrZ) rrpatrZcreate_tbl_stmtsr#Z cnames_brZ schema_nameZ create_stmtsZix_colsZcnamesr)rrr&s2     &  6zSQLiteTable._create_table_setupcCs|jpi}|j|kr||jStj|dd}|dkrJtjdtddd}n,|dkrXd }n|d krfd }n|d krvtd |tkrd }t|S)NT)rrzlthe 'timedelta' type is not supported, and will be written as integer values (ns frequency) to the database.r)rr@rremptyrrzComplex datatypes not supported) r<ryrrrrrre _SQL_TYPES)rrAr<rrrrr+_s&   zSQLiteTable._sql_type_name) rrrrrrrr$rrrr+ __classcell__rr)rrrs 9rc@seZdZdZdddZeddZddZedd d Z dd dZ ddZ dddZ d ddZ d!ddZd"ddZd#ddZd S)$rmz Version of SQLDatabase to support SQLite connections (fallback without SQLAlchemy). This should only be used internally. Parameters ---------- con : sqlite connection object FcCs||_||_dS)N)rRr!)rr!rRrrrrszSQLiteDatabase.__init__ccsT|j}z:y|V|jWn tk r@|jYnXWd|XdS)N)r!cursorcommitrorollbackclose)rrUrrrrs   zSQLiteDatabase.run_transactionc Os|jr|j}n |j}y|j|||Stk r}zty|jWnBtk r}z$td|dd|d}||Wdd}~XYnXtd|dd|}||Wdd}~XYnXdS)NzExecution failed on sql: r z unable to rollbackzExecution failed on sql 'z': )rRr!r2rTror4r)rr(rrUrdZ inner_excexrrrrTs  zSQLiteDatabase.executeNTccsLxF||}t|tkr t|}|s0|Pqt|||||dVqWdS)z+Return generator through chunked result set)rOrJr+N)rr=tupler%r5rQ)r2rWrIrOrJr+rNrrrrs  zSQLiteDatabase._query_iteratorc Csnt||}|j|}dd|jD} |dk rB|j||| |||dS||} |t| | |||d} | SdS)NcSsg|] }|dqS)rr)rZcol_descrrrrsz-SQLiteDatabase.read_query..)rOrJr+)r)rT descriptionr_fetchall_as_listr5rQ) rr&rOrJr'r+rWr(r2rIrNrPrrrrks(   zSQLiteDatabase.read_querycCs|}t|tst|}|S)N)rrr%)rrUrrrrr:s z SQLiteDatabase._fetchall_as_listrpc srtsfdd|Ddk rZx2D]&\} } t| ts0t| d| dq0Wt||||||d} | | || dS)ai Write records stored in a DataFrame to a SQL database. Parameters ---------- frame: DataFrame name: string Name of SQL table. if_exists: {'fail', 'replace', 'append'}, default 'fail' fail: If table exists, do nothing. replace: If table exists, drop it, recreate it, and insert data. append: If table exists, insert data. Create if it does not exist. index : boolean, default True Write DataFrame index as a column index_label : string or sequence, default None Column label for index column(s). If None is given (default) and `index` is True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. schema : string, default None Ignored parameter included for compatibility with SQLAlchemy version of ``to_sql``. chunksize : int, default None If not None, then rows will be written in batches of this size at a time. If None, all rows will be written at once. dtype : single type or dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a string. If all columns are of the same type, one single value can be used. method : {None, 'multi', callable}, default None Controls the SQL insertion clause used: * None : Uses standard SQL ``INSERT`` clause (one per row). * 'multi': Pass multiple values in a single ``INSERT`` clause. * callable with signature ``(pd_table, conn, keys, data_iter)``. Details and a sample callable implementation can be found in the section :ref:`insert method `. .. versionadded:: 0.24.0 csi|] }|qSrr)rrE)r<rrr#sz)SQLiteDatabase.to_sql..Nz (z) not a string)rPrurtrvr<)r rCrrzrerrr) rrPryrtrurvrZrWr<rwrArrir)r<rrxs 4  zSQLiteDatabase.to_sqlcCs*d}d|d}t|||gdkS)Nrz;SELECT name FROM sqlite_master WHERE type='table' AND name=;r)rrTr)rryrZr"queryrrrrn6s zSQLiteDatabase.has_tablecCsdS)Nr)rrYrZrrrr@szSQLiteDatabase.get_tablecCsdt|}||dS)Nz DROP TABLE )rrT)rryrZZdrop_sqlrrrrCszSQLiteDatabase.drop_tablec Cs"t|||d|||d}t|S)NF)rPrur#r<rZ)rrzr)rrPrYr#r<rZrirrrrGsz!SQLiteDatabase._create_sql_schema)F)NTN)NTNNN)rpTNNNNN)N)N)N)NNN)rrrrrrrrTrrrkr:rxrnrrrrrrrrms.       >  rmcCst|d}|j|||||dS)a2 Get the SQL db table schema for the given frame. Parameters ---------- frame : DataFrame name : string name of SQL table keys : string or sequence, default: None columns to use a primary key con: an open SQL database connection object or a SQLAlchemy connectable Using SQLAlchemy makes it possible to use any DB supported by that library, default: None If a DBAPI2 object, only sqlite3 is supported. dtype : dict of column name to SQL type, default None Optional specifying the datatype for columns. The SQL type should be a SQLAlchemy type, or a string for sqlite3 fallback connection. schema: str, default: None Optional specifying the schema to be used in creating the table. .. versionadded:: 1.2.0 )r!)r#r<rZ)rSr)rPryr#r!r<rZrVrrr get_schemaTs r=)NN)NTN)NN)NNTNNN)NNTNNr\)NNTNNN)NTNNN)NTNNr\)NTNNN)NTNNNN)NTNNNr\)NTNNNN)NrpTNNNN)N)NNF)NNNN)Cr contextlibrrrr functoolsrrtypingrrr r r rZnumpyr>Zpandas._libs.libZ_libsrZpandas.core.dtypes.commonr r rZpandas.core.dtypes.dtypesrZpandas.core.dtypes.missingrZpandas.core.apirrZpandas.core.baserZpandas.core.tools.datetimesrrrIOErrorrrr"r)r,rBrHrQrTr[rrjrlrxrnZ table_existsr`rSr{rrfr0rrr-rrmr=rrrrs         V@dH  <gU