ó ùµÈ[c@sßdZdgZddlZddlZddlZddlZddlZddlmZddl Z ddl Z yddl ZWne k r–nXddlmZddlmZmZdd lmZejd ksëejd krd „Zd „Znd„Zd„Zejejeƒdefd„ƒYZdejjfd„ƒYZdejjfd„ƒYZd„Zd„Z d„Z!dd„Z"d„Z#e$d„Z%defd„ƒYZ&defd„ƒYZ'dS( sDataset generator.t DataLoaderiÿÿÿÿN(tForkingPickleri(tsampleri(tndtcontext(t MXRecordIOtdarwintwin32cGstjtjj|ŒƒS(s*Rebuild ndarray from pickled shared memory(RtNDArraytndarrayt_new_from_shared_mem(targs((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytrebuild_ndarray*scCst|jƒfS(s&Reduce ndarray to shared memory handle(R t_to_shared_mem(tdata((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytreduce_ndarray/scCsVtjddkr(tjj|ƒ}n |jƒ}tjtjj ||||ƒƒS(s*Rebuild ndarray from pickled shared memoryii( tsyst version_infotmultiprocessingt reductiontrebuild_handletdetachRRR R (tpidtfdtshapetdtype((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR 3s cCsƒ|jtjddƒƒ}|jƒ\}}}}tjddkr[tjj|ƒ}ntjj |ƒ}t ||||ffS(s&Reduce ndarray to shared memory handlet cpu_sharedii( t as_in_contextRtContextR RRRRt reduce_handletDupFdR (RRRRR((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR<s tConnectionWrappercBs2eZdZd„Zd„Zd„Zd„ZRS(s[Connection wrapper for multiprocessing that supports sending NDArray via shared memory.cCs ||_dS(N(t_conn(tselftconn((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__init__NscCs<tjƒ}t|tjƒj|ƒ|j|jƒƒdS(s Send objectN(tiotBytesIORtpickletHIGHEST_PROTOCOLtdumpt send_bytestgetvalue(R!tobjtbuf((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytsendQs cCs|jƒ}tj|ƒS(sReceive object(t recv_bytesR&tloads(R!R,((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytrecvWs cCs"|jjddƒ}t||ƒS(s Emmulate connR N(t__dict__tgettNonetgetattr(R!tnametattr((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt __getattr__\s(t__name__t __module__t__doc__R#R-R0R7(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRJs    tQueuecBseZdZd„ZRS(sHWrapper for multiprocessing queue that dumps NDArray with shared memory.cOsštjddkr/tt|ƒj||Žn%tt|ƒjdtjƒ||Žt|jƒ|_t|j ƒ|_ |j j |_ |jj |_ dS(Niitctx(RRtsuperR;R#Rt get_contextRt_readert_writerR-t_sendR0t_recv(R!R tkwargs((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR#ds(R8R9R:R#(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR;bst SimpleQueuecBseZdZd„ZRS(s†Wrapper for multiprocessing SimpleQueue that dumps NDArray with shared memory. SimpleQueue don't use threading internally. cOsštjddkr/tt|ƒj||Žn%tt|ƒjdtjƒ||Žt|jƒ|_t|j ƒ|_ |j j |_ |jj |_ dS(NiiR<(RRR=RDR#RR>RR?R@R-RAR0RB(R!R RC((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR#ts(R8R9R:R#(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRDpscCsˆt|dtjƒr#tj|ŒSt|dtƒr_t|Œ}g|D]}t|ƒ^qIStj|ƒ}tj |d|j ƒSdS(sCollate data into batch.iRN( t isinstanceRRtstackttupletziptdefault_batchify_fntnptasarraytarrayR(Rti((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRIs  cCsât|dtjƒrktjt|ƒf|djd|djdtjddƒƒ}tj d||ŒSt|dt ƒr§t |Œ}g|D]}t |ƒ^q‘St j|ƒ}tj|d|jdtjddƒƒSdS(s8Collate data into batch. Use shared memory for stacking.iRR<RtoutN(RERRtemptytlenRRRRRFRGRHtdefault_mp_batchify_fnRJRKRL(RRNRM((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRQ‹s- cCsXt|tjƒr|j|ƒSt|ttfƒrTg|D]}t||ƒ^q;S|S(sMove data into new context.(RERRRtlistRGt_as_in_context(RR<td((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRSšs   iècCs€||krdSt|tƒr6|jƒ|jƒnFt|dƒr|x4|jjƒD] \}}t||d|ƒqUWndS(sRecursively find instance of MXRecordIO and reset file handler. This is required for MXRecordIO which holds a C pointer to a opened file after fork. NR1i(RERtclosetopenthasattrR1titemst_recursive_fork_recordio(R+tdeptht max_deptht_tv((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRY¢s   c Cs¥tjƒ}t|dtd|dƒƒ}t|d|ƒxbtr |jƒ\}}|dkrgPn|g|D]}||^qqƒ} |j|| fƒq?WdS(s+Worker loop for multiprocessing DataLoader.ii iiN( RtgetrecursionlimittmintmaxRYtTrueR2R3tput( tdatasett key_queuet data_queuet batchify_fntlimittmax_recursion_depthtidxtsamplesRMtbatch((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt worker_loop¯s    #cCspxitrk|jƒ\}}|dkr+Pn|rIt|tjƒƒ}nt|tjƒƒ}|||tdƒ‚|jƒt‚nxNtr¡|j|jkrT|jj|jƒ}|jd7_|j ƒ|SqTWdS(Ns)call __next__ after shutdown is forbiddens*Data buffer should be empty at this momenti( R‚RtR}R~R|R’t StopIterationRatpopR‹(R!Rk((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__next__ÿs    cCs |jƒS(N(R˜(R!((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR” scCs|S(N((R!((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__iter__scCsS|jsOx't|jƒD]}|jjdƒqW|jjdƒt|_ndS(s7Shutdown internal workers by pushing terminate signals.N(NN(NN(R‚RƒRvRzRbR3R{Ra(R!R\((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR’s  ( R8R9R:RRlR#R‘R“R‹R˜R”R™R’(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRrÍs !      c BsAeZdZdeddddded„Zd„Zd„ZRS(sRLoads data from a dataset and returns mini-batches of data. Parameters ---------- dataset : Dataset Source dataset. Note that numpy and mxnet arrays can be directly used as a Dataset. batch_size : int Size of mini-batch. shuffle : bool Whether to shuffle the samples. sampler : Sampler The sampler to use. Either specify sampler or shuffle, not both. last_batch : {'keep', 'discard', 'rollover'} How to handle the last batch if batch_size does not evenly divide `len(dataset)`. keep - A batch with less samples than previous batches is returned. discard - The last batch is discarded if its incomplete. rollover - The remaining samples are rolled over to the next epoch. batch_sampler : Sampler A sampler that returns mini-batches. Do not specify batch_size, shuffle, sampler, and last_batch if batch_sampler is specified. batchify_fn : callable Callback function to allow users to specify how to merge samples into a batch. Defaults to `default_batchify_fn`:: def default_batchify_fn(data): if isinstance(data[0], nd.NDArray): return nd.stack(*data) elif isinstance(data[0], tuple): data = zip(*data) return [default_batchify_fn(i) for i in data] else: data = np.asarray(data) return nd.array(data, dtype=data.dtype) num_workers : int, default 0 The number of multiprocessing workers to use for data preprocessing. pin_memory : boolean, default False If ``True``, the dataloader will copy NDArrays into pinned memory before returning them. Copying from CPU pinned memory to GPU is faster than from normal CPU memory. ic CsN||_| |_|dkr´|dkr9tdƒ‚n|dkr{|rctjt|ƒƒ}qtjt|ƒƒ}n|rtdƒ‚ntj|||r¨|ndƒ}n9|dk sÞ|sÞ|dk sÞ|dk rítdƒ‚n||_ |dkr|nd|_ |dkrA|dkr5t |_ qJt |_ n ||_ dS(Ns>batch_size must be specified unless batch_sampler is specifieds5shuffle must not be specified if sampler is specifiedtkeeps`batch_size, shuffle, sampler and last_batch must not be specified if batch_sampler is specified.i(Rwt _pin_memoryR3t ValueErrort_samplert RandomSamplerRPtSequentialSamplert BatchSamplerRyRvRQRxRI( R!Rct batch_sizetshuffleRt last_batchRRfRŒRp((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR#Is.           csJˆjdkr%‡fd†}|ƒStˆjˆjˆjˆjˆjƒS(Nic3sgx`ˆjD]U}ˆjg|D]}ˆj|^qƒ}ˆjrZt|tjƒƒ}n|Vq WdS(N(RyRxRwR›RSRRm(RkRitret(R!(s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytsame_process_iterns ) (RvRrRwRxRyR›(R!R¥((R!s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR™ls cCs t|jƒS(N(RPRy(R!((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR‘zsN(R8R9R:R3RR#R™R‘(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRs ,  ! ((R:t__all__R&R$RRtmultiprocessing.queuestmultiprocessing.reductionRRˆtnumpyRJtmultiprocessing.resource_sharert ImportErrortRRRRtrecordioRtplatformR RtregisterRtobjectRtqueuesR;RDRIRQRSRYRlRRqRrR(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/dataloader.pytsB                O