ó R;]c@s/dZdgZddlZddlZddlmZddlZddlZddlZddl Z ddl m Z ddl mZmZd „Zd „Zejejeƒd efd „ƒYZd ejjfd„ƒYZd„Zd„Zd„Zdefd„ƒYZdefd„ƒYZdS(sDataset generator.t DataLoaderiÿÿÿÿN(tForkingPickleri(tsampleri(tndtcontextcGstjtjj|ŒƒS(s*Rebuild ndarray from pickled shared memory(RtNDArraytndarrayt_new_from_shared_mem(targs((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pytrebuild_ndarray#scCst|jƒfS(s&Reduce ndarray to shared memory handle(R t_to_shared_mem(tdata((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pytreduce_ndarray)stConnectionWrappercBs2eZdZd„Zd„Zd„Zd„ZRS(s[Connection wrapper for multiprocessing that supports sending NDArray via shared memory.cCs ||_dS(N(t_conn(tselftconn((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__init__4scCs<tjƒ}t|tjƒj|ƒ|j|jƒƒdS(s Send objectN(tiotBytesIORtpickletHIGHEST_PROTOCOLtdumpt send_bytestgetvalue(Rtobjtbuf((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pytsend7s cCs|jƒ}tj|ƒS(sReceive object(t recv_bytesRtloads(RR((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pytrecv=s cCs"|jjddƒ}t||ƒS(s Emmulate connRN(t__dict__tgettNonetgetattr(Rtnametattr((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt __getattr__Bs(t__name__t __module__t__doc__RRRR%(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR 0s    tQueuecBseZdZd„ZRS(sHWrapper for multiprocessing queue that dumps NDArray with shared memory.cOsštjddkr/tt|ƒj||Žn%tt|ƒjdtjƒ||Žt|jƒ|_t|j ƒ|_ |j j |_ |jj |_ dS(Niitctx(tsyst version_infotsuperR)Rtmultiprocessingt get_contextR t_readert_writerRt_sendRt_recv(RRtkwargs((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRJs(R&R'R(R(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR)HscCsˆt|dtjƒr#tj|ŒSt|dtƒr_t|Œ}g|D]}t|ƒ^qIStj|ƒ}tj |d|j ƒSdS(sCollate data into batch.itdtypeN( t isinstanceRRtstackttupletziptdefault_batchify_fntnptasarraytarrayR5(R ti((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR:Vs  cCsât|dtjƒrktjt|ƒf|djd|djdtjddƒƒ}tj d||ŒSt|dt ƒr§t |Œ}g|D]}t |ƒ^q‘St j|ƒ}tj|d|jdtjddƒƒSdS(s8Collate data into batch. Use shared memory for stacking.iR5R*t cpu_sharedtoutN(R6RRtemptytlentshapeR5RtContextR7R8R9tdefault_mp_batchify_fnR;R<R=(R R@R>((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyREbs- cCsixbtrd|jƒ\}}|dkr+Pn|g|D]}||^q5ƒ}|j||fƒqWdS(s+Worker loop for multiprocessing DataLoader.N(tTrueR R!tput(tdatasett key_queuet data_queuet batchify_fntidxtsamplesR>tbatch((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt worker_loopqs   #t_MultiWorkerItercBsVeZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z RS( s-Interal multi-worker iterator for DataLoader.c Cs9|dks!tdj|ƒƒ‚||_||_||_||_tƒ|_td|jƒ|_i|_ d|_ d|_ t |jƒ|_ t|_g}xgt|jƒD]V}tjdtd|j|j|j|jfƒ}t|_|jƒ|j|ƒq³Wx%td|jƒD]}|jƒq!WdS(Nis&_MultiWorkerIter is not for {} workersittargetR(tAssertionErrortformatt _num_workerst_datasett _batchify_fnt_batch_samplerR)t _key_queuet _data_queuet _data_buffert _rcvd_idxt _sent_idxtitert_itertFalset _shutdowntrangeR.tProcessRORFtdaemontstarttappendt _push_next(Rt num_workersRHRKt batch_samplertworkerst_tworker((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyR|s,!          !  cCs t|jƒS(N(RBRW(R((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__len__—scCs|jƒdS(N(tshutdown(R((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyt__del__šscCsNt|jdƒ}|dkr"dS|jj|j|fƒ|jd7_dS(s&Assign next batch workload to workers.Ni(tnextR^R!RXRGR\(Rtr((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRfs  cCsÈ|j stdƒ‚|j|jkrQ|j s>tdƒ‚|jƒt‚nxptrÃ|j|jkrž|jj|jƒ}|jd7_|j ƒ|S|j j ƒ\}}||j|eZdZdedddddd„Zd„Zd„ZRS(s§Loads data from a dataset and returns mini-batches of data. Parameters ---------- dataset : Dataset Source dataset. Note that numpy and mxnet arrays can be directly used as a Dataset. batch_size : int Size of mini-batch. shuffle : bool Whether to shuffle the samples. sampler : Sampler The sampler to use. Either specify sampler or shuffle, not both. last_batch : {'keep', 'discard', 'rollover'} How to handle the last batch if batch_size does not evenly divide `len(dataset)`. keep - A batch with less samples than previous batches is returned. discard - The last batch is discarded if its incomplete. rollover - The remaining samples are rolled over to the next epoch. batch_sampler : Sampler A sampler that returns mini-batches. Do not specify batch_size, shuffle, sampler, and last_batch if batch_sampler is specified. batchify_fn : callable Callback function to allow users to specify how to merge samples into a batch. Defaults to `default_batchify_fn`:: def default_batchify_fn(data): if isinstance(data[0], nd.NDArray): return nd.stack(*data) elif isinstance(data[0], tuple): data = zip(*data) return [default_batchify_fn(i) for i in data] else: data = np.asarray(data) return nd.array(data, dtype=data.dtype) num_workers : int, default 0 The number of multiprocessing workers to use for data preprocessing. `num_workers > 0` is not supported on Windows yet. ic CsE||_|dkr«|dkr0tdƒ‚n|dkrr|rZtjt|ƒƒ}q‡tjt|ƒƒ}n|r‡tdƒ‚ntj|||rŸ|ndƒ}n9|dk sÕ|sÕ|dk sÕ|dk rätdƒ‚n||_|dkrÿ|nd|_ |dkr8|dkr,t |_ qAt |_ n ||_ dS(Ns>batch_size must be specified unless batch_sampler is specifieds5shuffle must not be specified if sampler is specifiedtkeeps`batch_size, shuffle, sampler and last_batch must not be specified if batch_sampler is specified.i( RUR!t ValueErrort_samplert RandomSamplerRBtSequentialSamplert BatchSamplerRWRTRERVR:( RRHt batch_sizetshuffleRt last_batchRhRKRg((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRòs,          csDˆjdkr%‡fd†}|ƒStˆjˆjˆjˆjƒS(Nic3sBgˆjD]0}ˆjg|D]}ˆj|^qƒV^q dS(N(RWRVRU(RNRL(R(s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyts(RTRPRURVRW(Rt generator((Rs[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRts cCs t|jƒS(N(RBRW(R((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRlsN(R&R'R(R!R_RRtRl(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyRÈs )   (R(t__all__R.tmultiprocessing.queuestmultiprocessing.reductionRRRR+tnumpyR;tRRxRRR R tregisterRtobjectR tqueuesR)R:RERORPR(((s[/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/data/dataloader.pyts(           N