U C^L/@sddlZddlZddlZddlZddlZedZGdddeZGdddeZ GdddeZ Gd d d eZ Gd d d eZ ee e e e d Z dS)NZfsspecc@s.eZdZdZddZddZedddZd S) BaseCacheagPass-though cache: doesn't keep anything, calls every time Acts as base class for other cachers Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is cCs||_||_||_dSN) blocksizefetchersizeselfrrrr 2/tmp/pip-install-6_kvzl1k/fsspec/fsspec/caching.py__init__szBaseCache.__init__cCs |||Sr)r)rstartendr r r _fetchszBaseCache._fetch)itemcCst|tstdt||jr<|jdkres z$MMapCache._fetch..rr) rrangepopminrrr$r!add) rr r Z start_blockZ end_blockZneedr7Zsstartsendr r8r rbs    zMMapCache._fetchcCs|j}|d=|S)Nr$)__dict__copyrstater r r __getstate__qs zMMapCache.__getstate__cCs|j|||_dSr)r?updater#r$rAr r r __setstate__ws zMMapCache.__setstate__)NN) rrrrr r#rrCrE __classcell__r r r%r r=s rcs(eZdZdZfddZddZZS)ReadAheadCachea" Cache which reads only when we get beyond a block of data This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file). cs&t|||d|_d|_d|_dS)Nr)rr r$r r rr%r r r szReadAheadCache.__init__cCst|j|}||}||jkr"dS||jkrP||jkrP|j||j||jS|j|krh|jkrnn(|j||jd}|t|8}|j}nd}t|j||j}||||_||_|jt|j|_||jd|SNrH)r<rr r r$lenrr)rr r lpartr r r rs    zReadAheadCache._fetch)rrrrr rrFr r r%r rG}s rGcs^eZdZdZdfdd ZddZddZd d Zd d Zd dZ fddZ ddZ Z S) BlockCachea Cache holding memory as a set of blocks. Requests are only ever made `blocksize` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than `maxblocks` are stored. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for `blocksize`, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. cs<t|||t|||_||_t||j|_ dSr) rr mathceilnblocks maxblocks functools lru_cache _fetch_block_fetch_block_cached)rrrrrRr%r r r szBlockCache.__init__cCsd|j|j|jS)Nz.)rrrrQr8r r r __repr__s zBlockCache.__repr__cCs |jS)z The statistics on the block cache. Returns ---------- NamedTuple Returned directly from the LRU Cache used internally. )rV cache_infor8r r r rXs zBlockCache.cache_infocCs|j}|d=|S)NrV)r?rAr r r rCszBlockCache.__getstate__cCs&|j|t|d|j|_dS)NrR)r?rDrSrTrUrVrAr r r rEs  zBlockCache.__setstate__cCsx||krtd||||jkr4td||j||j}||j}t||dD]}||qV|j||||dS)Nz('end' ({}) is smaller than 'start' ({}).z 'end={}' larger than size ('{}')r)start_block_numberend_block_number)rrrrr:rU _read_cache)rr r rYrZ block_numberr r r rs      zBlockCache._fetchcsN||jkrtd||j||j}||j}td|t||}|S)z= Fetch the block of data for `block_number`. z;'block_number={}' is greater than the number of blocks ({})zBlockCache fetching block %d)rQrrrloggerinforr)rr\r r Zblock_contentsr%r r rUs    zBlockCache._fetch_blockc Cs||j}||j}||kr2||}|||Sg}||||dt|d|D]} ||| q\|||d|d|SdS)z Read from our block cache. Parameters ---------- start, end : int The start and end byte positions. start_block_number, end_block_number : int The start and end block numbers. NrrH)rrVappendr:join) rr r rYrZZ start_posZend_posblockoutr\r r r r[s    zBlockCache._read_cache)rN) rrrrr rWrXrCrErrUr[rFr r r%r rMs  rMcs2eZdZdZd fdd ZddZddZZS) BytesCacheaKCache which holds data in a in-memory bytes object Implements read-ahead by the block size, for semi-random reads progressing through the file. Parameters ---------- trim: bool As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it. Tcs,t|||d|_d|_d|_||_dSrI)rr r$r r trim)rrrrrdr%r r r 3s zBytesCache.__init__cCs|jdk rH||jkrH|jdk rH||jkrH||j}|j||||S|jrbt|j||j}n|}||ksx||jkr|dS|jdks||jkr|jdks||jkr||||_||_n||jkr|j||jkr||||_||_n |||j}||_||j|_n^||jkrl|j|jkr*nB||j|jkrR||||_||_n||j|}|j||_|jt|j|_||j}|j||||}|jr|j|j|jd}|dkr|j|j|7_|j|j|d|_|S)NrHr) r r r$rr<rrrJrd)rr r offsetZbendnewrbnumr r r r:s\      zBytesCache._fetchcCs t|jSr)rJr$r8r r r __len__qszBytesCache.__len__)T)rrrrr rrhrFr r r%r rc&s 7rc)noner)bytesZ readaheadra)r+r/rSloggingrO getLoggerr]objectrrrGrMrccachesr r r r s" 3@%P