U Dx`8@sddlZddlZddlZddlZddlZddlmZddl Z ddl m Z m Z mZmZedddZGdddZGd d d eZGd d d eZGd ddeZddZddZdddZdS)N)join) implements_stringify_path _is_path_like _DEPR_MSGzfilesystem.LocalFileSystemz2.0.0zfs.LocalFileSystemc@seZdZdZddZddZd(ddZd d Zd d Zd dZ d)ddZ ddZ ddZ d*ddZ ddZddZddZddZd+d!d"Zd,d$d%Zed&d'Zd S)- FileSystemz( Abstract filesystem interface. c Cs,||d}|W5QRSQRXdS)z Return contents of file as a bytes object. Parameters ---------- path : str File path to read content from. Returns ------- contents : bytes rbN)openread)selfpathfr9/tmp/pip-target-oguziej0/lib/python/pyarrow/filesystem.pycat)s zFileSystem.catcCstdS)z Return list of file paths. Parameters ---------- path : str Directory to list contents from. NNotImplementedErrorr r rrrls9s z FileSystem.lsFcCstdS)z Delete the indicated file or directory. Parameters ---------- path : str Path to delete. recursive : bool, default False If True, also delete child paths for directories. Nrr r recursiverrrdeleteDs zFileSystem.deletec Cslt|}||}|ddkr&|dSd}||D]2\}}}|D]"}|||}|||d7}qBq4|S)z Compute bytes used by all contents under indicated path in file tree. Parameters ---------- path : str Can be a file path or directory. Returns ------- usage : int kindfilesizer)rstatwalk _path_join) r r Z path_infototalroot directoriesfilesZ child_pathabspathrrr disk_usageQs    zFileSystem.disk_usagecGs |j|SN)pathsepr)r argsrrrrkszFileSystem._path_joincCs tddS)zl Information about a filesystem entry. Returns ------- stat : dict FileSystem.statNrrrrrrnsr'cCs|j||dS)z. Alias for FileSystem.delete. r)rrrrrrmxsz FileSystem.rmcCs |||S)z. Alias for FileSystem.rename. )renamer r new_pathrrrmv~sz FileSystem.mvcCs tddS)z Rename file, like UNIX mv command. Parameters ---------- path : str Path to alter. new_path : str Path to move to. FileSystem.renameNrr+rrrr*s r.TcCstdS)z Create a directory. Parameters ---------- path : str Path to the directory. create_parents : bool, default True If the parent directories don't exists create them as well. Nrr r Zcreate_parentsrrrmkdirs zFileSystem.mkdircCstdS)z Return True if path exists. Parameters ---------- path : str Path to check. Nrrrrrexistss zFileSystem.existscCstdS)z Return True if path is a directory. Parameters ---------- path : str Path to check. Nrrrrrisdirs zFileSystem.isdircCstdS)z Return True if path is a file. Parameters ---------- path : str Path to check. Nrrrrrisfiles zFileSystem.isfilecCstdS)zf Returns True if this FileSystem is a unix-style file store with directories. Nrr rrr _isfilestoreszFileSystem._isfilestoreNc Cs,ddlm}|||||d}|j|||dS)a] Read Parquet data from path in file system. Can read from a single file or a directory of files. Parameters ---------- path : str Single file path or directory columns : List[str], optional Subset of columns to read. metadata : pyarrow.parquet.FileMetaData Known metadata to validate files against. schema : pyarrow.parquet.Schema Known schema to validate files against. Alternative to metadata argument. use_threads : bool, default True Perform multi-threaded column reads. use_pandas_metadata : bool, default False If True and file has custom pandas schema metadata, ensure that index columns are also loaded. Returns ------- table : pyarrow.Table r)ParquetDataset)schemametadata filesystem)columns use_threadsuse_pandas_metadata)Zpyarrow.parquetr6r ) r r r:r8r7r;r<r6Zdatasetrrr read_parquets zFileSystem.read_parquetrcCstdS)3 Open file for reading or writing. Nrr r moderrrr szFileSystem.opencCsdS)N/rr4rrrr%szFileSystem.pathsep)F)F)T)NNNTF)r)__name__ __module__ __qualname____doc__rrrr#rrr)r-r*r0r1r2r3r5r=r propertyr%rrrrr$s,       ! rcseZdZdZfddZeddZeddZee j dd Z ee j dd d Z ee j d dZ ee j ddZ ee jddZee jddZee jdddZeddZddZZS)LocalFileSystemNcstjttddtdSN stacklevel)warningswarn _FS_DEPR_MSGDeprecationWarningsuper__init__r4 __class__rrrQszLocalFileSystem.__init__c Cs6|jdkr0ttdt|_W5QRX|jS)Nignore) _instancerLcatch_warnings simplefilterrGclsrrr _get_instances    zLocalFileSystem._get_instancecCstjttdd|SrH)rLrMrNrOrZrXrrr get_instanceszLocalFileSystem.get_instancecs$ttfddtDS)Nc3s|]}t|VqdSr$)pjoin.0xr rr sz%LocalFileSystem.ls..)rsortedoslistdirrrr`rrszLocalFileSystem.lsTcCs&t|}|rt|n t|dSr$)rrcmakedirsr0r/rrrr0 s zLocalFileSystem.mkdircCst|}tj|Sr$)rrcr r2rrrrr2szLocalFileSystem.isdircCst|}tj|Sr$)rrcr r3rrrrr3szLocalFileSystem.isfilecCsdS)NTrr4rrrr5szLocalFileSystem._isfilestorecCst|}tj|Sr$)rrcr r1rrrrr1"szLocalFileSystem.existsrcCst|}t||dSr>)r@)rr r?rrrr 'szLocalFileSystem.opencCstjjSr$)rcr sepr4rrrr%/szLocalFileSystem.pathsepcCst|}t|S)z8 Directory tree generator, see os.walk. )rrcrrrrrr3szLocalFileSystem.walk)T)r)rBrCrDrUrQ classmethodrZr[rrrr0r2r3r5r1r rFr%r __classcell__rrrRrrGs.           rGc@seZdZdZddZeejddZeejddZeej dd Z eej dd d Z eej d dZ eej dddZ eej dddZ dddZddZdS)DaskFileSystemzJ Wraps s3fs Dask filesystem implementation like s3fs, gcsfs, etc. cCstjdtdd||_dS)Nz~The pyarrow.filesystem.DaskFileSystem/S3FSWrapper are deprecated as of pyarrow 3.0.0, and will be removed in a future version.rIrJ)rLrMrOfs)r rkrrrrQ@s zDaskFileSystem.__init__cCs tddSNzUnsupported file system APIrrrrrr2GszDaskFileSystem.isdircCs tddSrlrrrrrr3KszDaskFileSystem.isfilecCsdS)zh Object Stores like S3 and GCSFS are based on key lookups, not true file-paths. Frr4rrrr5OszDaskFileSystem._isfilestoreFcCst|}|jj||dS)Nr()rrkr)rrrrrWszDaskFileSystem.deletecCst|}|j|Sr$)rrkr1rrrrr1\szDaskFileSystem.existsTcCs(t|}|r|j|S|j|SdSr$)rrkZmkdirsr0r/rrrr0as zDaskFileSystem.mkdirrcCst|}|jj||dSrf)rrkr r?rrrr iszDaskFileSystem.opencCst|}|jj||dS)N)detail)rrkr)r r rmrrrrqszDaskFileSystem.lscCst|}|j|S)z9 Directory tree generator, like os.walk. )rrkrrrrrruszDaskFileSystem.walkN)F)T)r)F)rBrCrDrErQrrr2r3r5rr1r0r rrrrrrrj;s$        rjc@s:eZdZeejddZeejddZd ddZdS) S3FSWrappercCs\tt|}z4|j|}t|dkr8|d|kr8WdSWdSWntk rVYdSXdS)NrFT _sanitize_s3rrkrlenOSErrorr r contentsrrrr2s   zS3FSWrapper.isdircCsNtt|}z&|j|}t|dko0|d|kWStk rHYdSXdS)NrorFrprtrrrr3s   zS3FSWrapper.isfileFc#stt|}tt}t|jj||dD]<}|d}|ddkrP|q,|ddkr^q,||q,tfdd|D}tddD||fVD]}|j||dEd Hqd S) z Directory tree generator, like os.walk. Generator version of what is in s3fs, which yields a flattened list of files. )refreshZKeyZ StorageClassZ DIRECTORYZBUCKETcs"g|]}|krt|dqSro posixpathsplit)r^r r rr sz$S3FSWrapper.walk..cSsg|]}t|dqSrwrxr]rrrr|sN) rqrsetlistrkZ_lsaddrbr)r r rvr!key directoryrr{rrs"      zS3FSWrapper.walkN)F)rBrCrDrrr2r3rrrrrrn}s   rncCs|dr|ddS|SdS)Nzs3://) startswithreplacer`rrrrqs  rqcCsNt|}t|tsFdtjkr6tjd}t||jr6|Std|n|SdS)NfsspeczUnrecognized filesystem: {}) type issubclassrsysmodules isinstanceZAbstractFileSystemrsformat)rkZfs_typerrrr_ensure_filesystems    rc Cs,t|s |dk rtd||fS|dk rdt|}t|trDt|}nt|tsXtdn|}||fSt|}tj |}|j dks|j dkr|j d}|d}|dkrd }n|j d |}d}t|d kr|d rt|d }tjj||d }|j}n(|j dkrt}|j}n t}|}||fS)zn Return filesystem from path which could be an HDFS URI, a local URI, or a plain filesystem path. NzVfilesystem passed but where is file-like, so there is nothing to open with filesystem.zPExpected string path; path-like objects are only allowed with a local filesystemhdfsZviewfs:rrdefaultz://rIro)hostportr)r ValueErrorrrrGrstr TypeErrorurllibparseurlparseschemenetlocrzrr isnumericintpar_connectr rZ) wherer9r Z parsed_uriZ netloc_splitrrrkZfs_pathrrrresolve_filesystem_and_pathsB       r)N)rcryr urllib.parserrLos.pathrr\ZpyarrowrZ pyarrow.utilrrrrrrNrrGrjrnrqrrrrrrs( OIB7