ó ùµÈ[c@sHdZddddddgZddlZddlZddlZddlZddlZddlZd d l m Z d d l m Z m Z mZd dl mZmZmZmZde jfd„ƒYZdefd„ƒYZde jfd„ƒYZdefd„ƒYZde jfd„ƒYZde jfd„ƒYZdS(sDataset container.tMNISTt FashionMNISTtCIFAR10tCIFAR100tImageRecordDatasettImageFolderDatasetiÿÿÿÿNi(tdataseti(tdownloadt check_sha1t_get_repo_file_urli(tndtimagetrecordiotbasecBsAeZdZejjejƒddƒedd„Z d„Z RS(s)MNIST handwritten digits dataset from http://yann.lecun.com/exdb/mnist Each sample is an image (in 3D NDArray) with shape (28, 28, 1). Parameters ---------- root : str, default $MXNET_HOME/datasets/mnist Path to temp folder for storing data. train : bool, default True Whether to load the training or testing set. transform : function, default None A user defined callback that transforms each sample. For example: :: transform=lambda data, label: (data.astype(np.float32)/255, label) tdatasetstmnistcCsS||_d |_d |_d |_d |_d |_tt|ƒj||ƒdS(Nstrain-images-idx3-ubyte.gzt(6c95f4b05d2bf285e1bfb0e7960c31bd3b3f8a7dstrain-labels-idx1-ubyte.gzt(2a80914081dc54586dbdf242f9805a6b8d2a15fcst10k-images-idx3-ubyte.gzt(c3a25af1f52dad7f726cce8cacb138654b760d48st10k-labels-idx1-ubyte.gzt(763e7fa3757d93b0cdec073cef058b2004252c17R(strain-images-idx3-ubyte.gzs(6c95f4b05d2bf285e1bfb0e7960c31bd3b3f8a7d(strain-labels-idx1-ubyte.gzs(2a80914081dc54586dbdf242f9805a6b8d2a15fc(st10k-images-idx3-ubyte.gzs(c3a25af1f52dad7f726cce8cacb138654b760d48(st10k-labels-idx1-ubyte.gzs(763e7fa3757d93b0cdec073cef058b2004252c17( t_traint _train_datat _train_labelt _test_datat _test_labelt _namespacetsuperRt__init__(tselftrootttraint transform((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR6s      c Cs|jr|j|j}}n|j|j}}d|j}tt||dƒd|jd|dƒ}tt||dƒd|jd|dƒ}t j |dƒJ}t j d|j dƒƒtj|j ƒd tjƒjtjƒ}WdQXt j |dƒ\}t j d |j d ƒƒtj|j ƒd tjƒ}|jt|ƒd d dƒ}WdQXtj|d |jƒ|_||_dS( Nsgluon/dataset/itpatht sha1_hashitrbs>IIitdtypes>IIIIii(RRRRRRRR t_roottgziptopentstructtunpacktreadtnpt frombuffertuint8tastypetint32treshapetlenR tarrayR#t_datat_label(Rtdatatlabelt namespacet data_filet label_filetfin((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyt _get_dataDs&      0$N( t__name__t __module__t__doc__tosR tjoinR tdata_dirtTruetNoneRR:(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR$s cBs8eZdZejjejƒddƒedd„Z RS(s£A dataset of Zalando's article images consisting of fashion products, a drop-in replacement of the original MNIST dataset from https://github.com/zalandoresearch/fashion-mnist Each sample is an image (in 3D NDArray) with shape (28, 28, 1). Parameters ---------- root : str, default $MXNET_HOME/datasets/fashion-mnist' Path to temp folder for storing data. train : bool, default True Whether to load the training or testing set. transform : function, default None A user defined callback that transforms each sample. For example: :: transform=lambda data, label: (data.astype(np.float32)/255, label) Rs fashion-mnistcCsS||_d |_d |_d |_d |_d |_tt|ƒj||ƒdS(Nstrain-images-idx3-ubyte.gzt(0cf37b0d40ed5169c6b3aba31069a9770ac9043dstrain-labels-idx1-ubyte.gzt(236021d52f1e40852b06a4c3008d8de8aef1e40bst10k-images-idx3-ubyte.gzt(626ed6a7c06dd17c0eec72fa3be1740f146a2863st10k-labels-idx1-ubyte.gzt(17f9ab60e7257a1620f4ad76bbbaf857c3920701s fashion-mnist(strain-images-idx3-ubyte.gzs(0cf37b0d40ed5169c6b3aba31069a9770ac9043d(strain-labels-idx1-ubyte.gzs(236021d52f1e40852b06a4c3008d8de8aef1e40b(st10k-images-idx3-ubyte.gzs(626ed6a7c06dd17c0eec72fa3be1740f146a2863(st10k-labels-idx1-ubyte.gzs(17f9ab60e7257a1620f4ad76bbbaf857c3920701( RRRRRRRRR(RRRR((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRss      N( R;R<R=R>R R?R R@RARBR(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR_scBsJeZdZejjejƒddƒedd„Z d„Z d„Z RS(s:CIFAR10 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html Each sample is an image (in 3D NDArray) with shape (32, 32, 1). Parameters ---------- root : str, default $MXNET_HOME/datasets/cifar10 Path to temp folder for storing data. train : bool, default True Whether to load the training or testing set. transform : function, default None A user defined callback that transforms each sample. For example: :: transform=lambda data, label: (data.astype(np.float32)/255, label) Rtcifar10cCs\||_d|_dddddg|_dg|_d|_tt|ƒj||ƒdS(Nscifar-10-binary.tar.gzt(fab780a1e191a7eda0f345501ccd62d20f7ed891sdata_batch_1.bint(aadd24acce27caa71bf4b10992e9e7b2d74c2540sdata_batch_2.bint(c0ba65cce70568cd57b4e03e9ac8d2a5367c1795sdata_batch_3.bint(1dd00a74ab1d17a6e7d73e185b69dbf31242f295sdata_batch_4.bint(aab85764eb3584312d3c7f65fd2fd016e36a258esdata_batch_5.bint(26e2849e66a845b7f1e4614ae70f4889ae604628stest_batch.bint(67eb016db431130d61cd03c7ad570b013799c88cRG(scifar-10-binary.tar.gzs(fab780a1e191a7eda0f345501ccd62d20f7ed891(sdata_batch_1.bins(aadd24acce27caa71bf4b10992e9e7b2d74c2540(sdata_batch_2.bins(c0ba65cce70568cd57b4e03e9ac8d2a5367c1795(sdata_batch_3.bins(1dd00a74ab1d17a6e7d73e185b69dbf31242f295(sdata_batch_4.bins(aab85764eb3584312d3c7f65fd2fd016e36a258e(sdata_batch_5.bins(26e2849e66a845b7f1e4614ae70f4889ae604628(stest_batch.bins(67eb016db431130d61cd03c7ad570b013799c88c(Rt _archive_fileRRRRRR(RRRR((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR”s     cCs¢t|dƒ1}tj|jƒdtjƒjdd ƒ}WdQX|dd…dd…fjddddƒjdd ddƒ|dd…dfjtjƒfS( NR"R#iÿÿÿÿi iii iii ( R&R*R+R)R,R/t transposeR-R.(RtfilenameR9R4((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyt _read_batch¡s0=cs td„‡fd†ˆjˆjDƒDƒƒršdˆj}tt|ˆjdƒdˆjdˆjdƒ}tj |ƒ}|j ˆjƒWdQXnˆj r¯ˆj}n ˆj}t ‡fd†|DƒŒ\}}t j|ƒ}t j|ƒ}tj|d |jƒˆ_|ˆ_dS( Ncss8|].\}}tjj|ƒ p/t||ƒ VqdS(N(R>R texistsR(t.0R tsha1((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pys ©sc3s3|])\}}tjjˆj|ƒ|fVqdS(N(R>R R?R$(RTtnameRU(R(s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pys ªssgluon/dataset/iR R!ic3s6|],\}}ˆjtjjˆj|ƒƒVqdS(N(RRR>R R?R$(RTRVt_(R(s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pys ¸sR#(tanyRRRRR ROR$ttarfileR&t extractallRtzipR*t concatenateR R1R#R2R3(RR6RQttart data_filesR4R5((Rs`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR:¨s$       N( R;R<R=R>R R?R R@RARBRRRR:(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR‚s  cBsDeZdZejjejƒddƒee dd„Z d„Z RS(sÅCIFAR100 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html Each sample is an image (in 3D NDArray) with shape (32, 32, 1). Parameters ---------- root : str, default $MXNET_HOME/datasets/cifar100 Path to temp folder for storing data. fine_label : bool, default False Whether to load the fine-grained (100 classes) or coarse-grained (20 super-classes) labels. train : bool, default True Whether to load the training or testing set. transform : function, default None A user defined callback that transforms each sample. For example: :: transform=lambda data, label: (data.astype(np.float32)/255, label) Rtcifar100cCsY||_d|_d g|_d g|_||_d|_tt|ƒj||ƒdS( Nscifar-100-binary.tar.gzt(a0bb982c76b83111308126cc779a992fa506b90bs train.bint(e207cd2e05b73b1393c74c7f5e7bea451d63e08estest.bint(8fb6623e830365ff53cf14adec797474f5478006R_(scifar-100-binary.tar.gzs(a0bb982c76b83111308126cc779a992fa506b90b(s train.bins(e207cd2e05b73b1393c74c7f5e7bea451d63e08e(stest.bins(8fb6623e830365ff53cf14adec797474f5478006( RRORRt _fine_labelRRRR(RRt fine_labelRR((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRÕs      cCs©t|dƒ1}tj|jƒdtjƒjdd ƒ}WdQX|dd…dd…fjddddƒjdddd ƒ|dd…d|jfjtj ƒfS( NR"R#iÿÿÿÿi iii iii ( R&R*R+R)R,R/RPRcR-R.(RRQR9R4((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRRßs0=N( R;R<R=R>R R?R R@tFalseRARBRRR(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRÁs cBs&eZdZddd„Zd„ZRS(sA dataset wrapping over a RecordIO file containing images. Each sample is an image and its corresponding label. Parameters ---------- filename : str Path to rec file. flag : {0, 1}, default 1 If 0, always convert images to greyscale. If 1, always convert images to colored (RGB). transform : function, default None A user defined callback that transforms each sample. For example: :: transform=lambda data, label: (data.astype(np.float32)/255, label) icCs,tt|ƒj|ƒ||_||_dS(N(RRRt_flagt _transform(RRQtflagR((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRûs cCsztt|ƒj|ƒ}tj|ƒ\}}|jdk r^|jtj||j ƒ|j ƒStj||j ƒ|j fS(N( RRt __getitem__R R(RgRBR timdecodeRfR5(Rtidxtrecordtheadertimg((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRis "N(R;R<R=RBRRi(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRçscBs8eZdZddd„Zd„Zd„Zd„ZRS(skA dataset for loading image files stored in a folder structure like:: root/car/0001.jpg root/car/xxxa.jpg root/car/yyyb.jpg root/bus/123.jpg root/bus/023.jpg root/bus/wwww.jpg Parameters ---------- root : str Path to root directory. flag : {0, 1}, default 1 If 0, always convert loaded images to greyscale (1 channel). If 1, always convert loaded images to colored (3 channels). transform : callable, default None A function that takes data and label and transforms them: :: transform = lambda data, label: (data.astype(np.float32)/255, label) Attributes ---------- synsets : list List of class names. `synsets[i]` is the name for the integer label `i` items : list of tuples List of all images in (filename, label) pairs. icCsMtjj|ƒ|_||_||_dddg|_|j|jƒdS(Ns.jpgs.jpegs.png(R>R t expanduserR$RfRgt_extst _list_images(RRRhR((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyR&s   cCs;g|_g|_x"ttj|ƒƒD] }tjj||ƒ}tjj|ƒsrtj d|ddƒq(nt |jƒ}|jj |ƒxŸttj|ƒƒD]ˆ}tjj||ƒ}tjj |ƒd}|j ƒ|jkrtj d||dj|jƒfƒq§n|jj ||fƒq§Wq(WdS(Ns&Ignoring %s, which is not a directory.t stackleveliis'Ignoring %s of type %s. Only support %ss, (tsynsetstitemstsortedR>tlistdirR R?tisdirtwarningstwarnR0tappendtsplitexttlowerRp(RRtfolderR R5RQtext((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRq-s"   cCsZtj|j|d|jƒ}|j|d}|jdk rP|j||ƒS||fS(Nii(R timreadRtRfRgRB(RRkRnR5((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRiAs  cCs t|jƒS(N(R0Rt(R((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyt__len__HsN(R;R<R=RBRRqRiR€(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyRs   (R=t__all__R>R%RYR'RxtnumpyR*tRtutilsRRR R R R R t_DownloadedDatasetRRRRtRecordFileDatasetRtDatasetR(((s`/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/data/vision/datasets.pyts"        ";#?&!