U bA @sddlZddlZddlZddlmZddlZddlZej ej e Z e ej e ddZeeZW5QRXdddZd d Zd d ZdddZedkre\ZZeejeejZZdS)N)requestz config.jsonr /tmp/dataTc Cstj|st||r$d}d}nd}d}td}td}||fD]:}tjd|}tj||}tj|sF||||qFt |||S)zDownload MNIST dataset and convert it to numpy array Args: data_dir (str): directory to save the data train (bool): download training set Returns: tuple of images and labels as numpy arrays ztrain-images-idx3-ubyte.gzztrain-labels-idx1-ubyte.gzzt10k-images-idx3-ubyte.gzzt10k-labels-idx1-ubyte.gzs3Z public_bucketzdatasets/image/MNIST) ospathexistsmakedirsboto3clientCONFIGjoinZ download_file_convert_to_numpy) data_dirZtrain images_file labels_filerbucketobjkeydestrl/home/ec2-user/SageMaker/sagemaker-hosting/Advanced-Model-Deployment/Multi-Container-Endpoint/utils/mnist.pymnist_to_numpys     rc Csttj||d&}tj|tjdd ddd}W5QRXttj||d}tj|tjdd}W5QRX||fS)zByte string to numpy arraysrb)offset) gzipopenrrr np frombufferreaduint8reshape)rrrfZimageslabelsrrrr.s * rcCs<ttj}tj||dd}tj||dd|}|||S)NT)axiskeepdims)r!finfofloatepsmeanstd)xr(r,r-r.rrr normalize9s r0pytorchcCsX|jdkrt|d}|dkr"|S|dkr6t|dS|dkrB|Stdd|d S) aAdjust a ``numpy.ndarray`` to be used as input for specified framework Args: x (numpy.ndarray): Batch of images to be adjusted to follow the convention in pytorch / tensorflow / mxnet framework (str): Framework to use. Takes value in ``pytorch``, ``tensorflow`` or ``mxnet`` Return: numpy.ndarray following the convention of tensors in the given framework )r1mxnetZ tensorlfow)rr2r3r4zframework must be one of z$[pytorch, tensorflow, mxnet], got {}N)ndimr! expand_dims transpose ValueErrorformat)r/Z frameworkrrradjust_to_frameworkBs    r;__main__)rT)r1)rjsonrurllibrr numpyr!rdirnameabspath__file__r r r&loadr rrr0r;__name__XYastypefloat32int8rrrrs