v]c @`s|dZddlmZmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZddlZddljZddljZyddljZWnek r dZnXyddlZWnek r0nXddl Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(dd l)m*Z*d Z+d Z,d Z-dd Z.ddZ/ddZ0dZ1dZ2ddZ3dZ4ddde5dZ6dddZ7dZ8dZ9dddddde5ddZ:dddde5dddZ;ddddde5dZ<dddde5dZ=d d d!Z>d d d d"Z?d d#Z@d$ZAd%ZBddd&ZCd'ZDdde5d(ZEddd`e5d+ZFddddae5d,ZGddd-ZHdddbd.ZId/ZJd0ZKde5d1ZLe-d2ZMe-d3ZNdd4eOe-d5ZPdd6d7ddeOdde-d8 ZQd4ddde5e-d9ZRd:ddd;dde5e-d<ZSddd=dd>d?ZTd@d;dddeOde5e5ejUdA ZVdBZWdde5dCdDZXdEZYdFZZdGZ[dHZ\dddIZ]dJZ^dKZ_dLdMZ`dNZaedOZbdPe!jcjdfdQYZedRZfdSdTZgdUdVZhdSdWZidSdXZjdSdCdYdZd[Zkddd\Zlddd4d:eOd]Zmd^enfd_YZodS(csTools for testing.i(tabsolute_importtprint_functiontdivisionN(tcontextmanageri(tContexttcurrent_context(t_STORAGE_TYPE_STR_TO_ID(tarray(tSymbolcC`stS(s(Get default context for regression test.(R(((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytdefault_context4scC`s|tj_dS(sSet default context.N(Rt _default_ctxtvalue(tctx((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytset_default_context;scC`stjS(s*Get default data type for regression test.(tnptfloat32(((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt default_dtype@scC`s|dkrdS|S(s4Get default numerical threshold for regression test.g#B ;N(tNone(tatol((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_atolFscC`s|dkrdS|S(s4Get default numerical threshold for regression test.gh㈵>N(R(trtol((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_rtolMscC`s|dkrdS|S(s4Get default numerical threshold for regression test.iN(R(tetol((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_etolSscG`sOg|D]$}tjj|jt^q}t|dkrK|dS|S(s"Generate some random numpy arrays.ii(RtrandomtrandntastypeRtlen(tshapeststarrays((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt random_arraysYs .cC`sJd|kot|kns(t|}tjj||d|!S(sKReturn a k length list of the elements chosen from the population sequence.i(RtAssertionErrorRRtshuffle(t populationtktpopulation_copy((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt random_samplebs(tuniformcC`st|||}|dks,|dkr;tdn|dksS|dkrbtdn|dkr|d|krtd|||fqndS( s9Validates inputs for csr generation helper functions iis!density has to be between 0 and 1s-num_rows or num_cols should be greater than 0tpowerlawisvnot supported for this density: %s for this shape (%s, %s) Please keep : num_rows * num_cols * density >= 2 * num_rowsN(tintt ValueError(tnum_rowstnum_colstdensityt distributiont total_nnz((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt_validate_csr_generation_inputsjs cC`st|jd}xkt|D]]}|j|}|j|d}tj|j||!}tjj|||j||+q WdS(sShuffle CSR column indices per row This allows validation of unordered column indices, which is not a requirement for a valid CSR matrix iN(RtindptrtrangeRRtindicesRR!(tcsrt row_counttit start_indext end_indextsublist((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytshuffle_csr_column_indices~s g?c C`sBt|||ddyddlm}|j|||d|dd}|d k ri|jj|n|tkrt|nt j jj |j|j |j fd||fd|}Wntk r=|d kstd | std t j jjd||fd|} | | |k} | jd}nX|S( sIReturns CSRNDArray with uniform distribution This generates a csr matrix with totalnnz unique randomly chosen numbers from num_rows*num_cols and arranges them in the 2d array in the following way: row_index = (random_number_generated / num_rows) col_index = random_number_generated - row_index * num_cols R-R&i(tsparsetdtypetformatR3tshapes6data_init option is not supported when scipy is absents@shuffle_csr_indices option is not supported when scipy is absentN(R/tscipyR:trandRtdatatfilltTrueR9tmxtndt csr_matrixR2R0t ImportErrorR RR&ttostype( R*R+R,R;t data_inittshuffle_csr_indicestspspR3tresulttdnst masked_dns((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt_get_uniform_dataset_csrs( !   $    $c C`st|||ddt|||}|}tj||fd|}xat|D]S}dtjdd||d<|d}|dkrXtjj |j dSqXWd}xt|D]}t ||} | |krB|| krBdtjdd||<|| d}|dkrtjj |j dSqnxdtd| D]S} dtjdd||| <|d}|dkrRtjj |j dSqRW|d}qW|dkrt d |||fntjj |j dSd S( s<Returns CSRNDArray with powerlaw distribution with exponentially increasing number of non zeros in each row. Not supported for cases where total_nnz < 2*num_rows. This is because the algorithm first tries to ensure that there are rows with no zeros by putting non zeros at beginning of each row. R-R'R;igMbP?iiR3s9not supported for this density: %s for this shape (%s,%s)N( R/R(RtzerosR1trndR&RCRDRRGtminR)( R*R+R,R;R.t unused_nnzt output_arrtrowtcol_maxt col_limitt col_index((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt_get_powerlaw_dataset_csrs:      cC`s|dkrtj|}ntj|ddg}tj|j}tj|ddgddg}x;|js|d}|||d<|j|jqlW|S(sIReturn ndarray composed of passing each array value through some functiontflagstf_indextop_flagst writeonlyiN(RRRtnditerROR=tfinishedtiternext(t the_inputtfunctiontoutputtit_inputtit_outt val_input((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt assign_eachs !   c C`s|dkrtj|}n|j|jks6ttj|ddg}tj|ddg}tj|j}tj|ddgddg}xR|js|d}|d}||||d<|j|j|jqW|S(sIReturn ndarray composed of passing two array values through some functionRYRZR[R\iN( RRRR=R R]ROR^R_( tinput1tinput2RaRbt it_input1t it_input2Rdt val_input1t val_input2((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt assign_each2s !     c  C`s| r | nt} |dkr-tjn|}|dkrHtn|}|dkr`dn|}|dkr|dkstd||dk r|} t| |dkstn.tj|d} tj| |kj } | j ddkrLt j j |ddd|d| } | tjgd|tjgffStj| j d|dj|} |dk r| j|n|dk rt| |} nt j jj| | fd |d|d| }|| | ffS|d krt|d ks t|dkrjt|d|d|d |d |d|j| }||j|j|jffS|dkrt|d|dd|d|j| }||j|j|jffStstd|tSntstdtSdS(sGenerate a random sparse ndarray. Returns the ndarray, value(np) and indices(np) Parameters ---------- shape: list or tuple stype: str valid values: "csr" or "row_sparse" density: float, optional should be between 0 and 1 distribution: str, optional valid values: "uniform" or "powerlaw" dtype: numpy.dtype, optional default value is None Returns ------- Result of type CSRNDArray or RowSparseNDArray Examples -------- Below is an example of the powerlaw distribution with csr as the stype. It calculates the nnz using the shape and density. It fills up the ndarray with exponentially increasing number of elements. If there are enough unused_nnzs, n+1th row will have twice more nnzs compared to nth row. else, remaining unused_nnzs will be used in n+1th row If number of cols is too small and we have already reached column size it will fill up all following columns in all followings rows until we reach the required density. >>> csr_arr, _ = rand_sparse_ndarray(shape=(5, 16), stype="csr", density=0.50, distribution="powerlaw") >>> indptr = csr_arr.indptr.asnumpy() >>> indices = csr_arr.indices.asnumpy() >>> data = csr_arr.data.asnumpy() >>> row2nnz = len(data[indptr[1]:indptr[2]]) >>> row3nnz = len(data[indptr[2]:indptr[3]]) >>> assert(row3nnz == 2*row2nnz) >>> row4nnz = len(data[indptr[3]:indptr[4]]) >>> assert(row4nnz == 2*row3nnz) R&t row_sparses,Distribution %s not supported for row_sparseitstypeR;R iR=R3iRHRIR'R,sDistribution not supported: %ssunknown storage typeN(R RRPR?RR RRtargwheretflattenR=RCRDRORRRARfR:trow_sparse_arrayRNt as_in_contextR0R2R@RXtFalse(R=RoR,R;R-RHt rsp_indicest modifier_funcRIR R2t idx_sampleRKtvaltarrR3((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytrand_sparse_ndarraysH+   $(&  -   ,tdefaultc C`s|r |nt}|dkrHtjjt|d|d|}n9t||d|d|d|d|d|d|\}} |S(s@Generate a random sparse ndarray. Returns the generated ndarray.R{R;R R,RvRIR-(R RCRDRRRz( R=RoR,R;RvRIR-R Ryt_((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt rand_ndarray]s ' g?c C`s|dkrv|d k r4tj|}|jnd }t||d|d|d|d|d|\} \} } na|dkrt||d|d|d|d|d|\} \} } } nd |} t| | S( sHCreate a sparse array, For Rsp, assure indices are in a canonical formatRnR,RHRuR;RvR3RIsUnknown storage type: N(RRtasarraytsortRzR ( R=RoRHRuR;RvR,RIt arr_indicestarr_dataR|tmsg((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcreate_sparse_arrayks,       cC`sq|dkr@d}|d k r@t||dks=tq@nt||d|d|d|d|d|d |S( s@Create sparse array, using only rsp_indices to determine densityRngiRHRuR;RvR,RIN(RRR R(R=RoR,RHRuR;RvRI((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcreate_sparse_array_zds  " i cC`s*tjd|dtjd|dfS(Ni(RPtrandint(tdim0tdim1((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt rand_shape_2dscC`s=tjd|dtjd|dtjd|dfS(Ni(RPR(RRtdim2((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt rand_shape_3dscC`s ttjd|dd|S(Nitsize(ttupleRPR(tnum_dimtdim((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt rand_shape_ndscC`sFtjj||dtj}tjj||dtj}||fS(NR;(RRRtint64(tx_lowtx_highty_lowty_hightxty((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt rand_coord_2dscC`st|tr|g}n-|dk r3t|ntt|j}|}x,tt|D]}||d|}qaW|rt|j}x|D]}d||s Rtis_trainii(R titemstbindtforwardtoutputstasnumpyR(tsymR RtinputstexeRR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytsimple_forwardxs" c`st|tttfstdksNtjtjtjfksNtt|trt |j t |j krt dt t |j t t |j fqndt|j |D}fd|jD}|S(sgParses the given location to a dictionary. Arguments of the provided op `sym` are used as dictionary keys and elements of `location` are used as values. Parameters ---------- sym : Symbol Symbol containing op location : list or tuple or dict Argument values location - if type is list or tuple of `np.ndarray` inner elements are arrays correspoding to ``sym.list_arguments()``. - if type is dict of str -> `np.ndarray` maps the name of arguments to the corresponding `np.ndarray`. *In either case, value of all the arguments must be provided.* ctx : Context Device context. dtype: "asnumpy" or np.float16 or np.float32 or np.float64 If dtype is "asnumpy" then the mx.nd.array created will have the same type as th numpy array from which it is copied. Otherwise, dtype is the explicit datatype for all mx.nd.array objects created in this function. Returns ------- dict Dictionary with `sym` arguments as keys and `location` elements as values. Examples ------- >>> a = mx.symbol.Variable('a') >>> b = mx.symbol.Variable('b') >>> l1 = np.ndarray([2,3]) >>> l2 = np.ndarray([3,4]) >>> _parse_location(a * b, [l1, l2], None) {'a': , 'b': } >>> _parse_location(a * b, {'a': l1, 'b': l2}, None) {'a': , 'b': } >>> _parse_location(a * b, {'a': l1}, None) ValueError: Symbol arguments and keys of the given location do not match. Rs_Symbol arguments and keys of the given location do not match.symbol args:%s, location.keys():%scS`si|]\}}||qS(((RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s c `sdi|]Z\}}t|tjrWtjj|dddkrN|jnn||qS(R R;R(RRtndarrayRCRDRR;(RR#R(R R;(s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s (RtdictRRR Rtfloat16Rtfloat64tsettkeystlist_argumentsR)RtzipR(RtlocationR R;((R R;s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt_parse_locations.0$:c`sdks0tjtjtjfks0t|dk rt|trt|j t|j krt dt t|j t t|j fqn=t|t tfr|j }dt||D}nfd|jD}n|S(sParses the given auxiliary states to a dictionary. Auxiliary states of the provided op `sym` are used as dictionary keys and elements of `aux_states` are used as values. Parameters ---------- sym : Symbol Symbol containing op aux_states : None or list or dict Aux states - if type is list or tuple of `np.ndarray` inner elements are arrays correspoding to ``sym.list_auxiliary_states()``. - if type is dict of str -> `np.ndarray` maps the name of arguments to the corresponding `np.ndarray`. *In either case, all aux states of `sym` must be provided.* ctx : Context Device context. dtype: "asnumpy" or np.float16 or np.float32 or np.float64 If dtype is "asnumpy" then the mx.nd.array created will have the same type as th numpy array from which it is copied. Otherwise, dtype is the explicit datatype for all mx.nd.array objects created in this function. Returns ------- dict Dictionary with `sym` aux states as keys and `aux_states` elements as values. Examples ------- >>> data = mx.symbol.Variable('data') >>> weight = mx.sym.Variable(name='fc1_weight') >>> fc1 = mx.symbol.FullyConnected(data = data, weight=weight, name='fc1', num_hidden=128) >>> fc2 = mx.symbol.BatchNorm(fc1, name='batchnorm0') >>> mean_states = np.ones(3) >>> var_states = np.ones(3) >>> _parse_aux_states(fc2, [mean_states, var_states], None) {'batchnorm0_moving_var': , 'batchnorm0_moving_mean': } >>> _parse_aux_states(fc2, {'batchnorm0_moving_var': mean_states, ... 'batchnorm0_moving_mean': var_states}, None) {'batchnorm0_moving_var': , 'batchnorm0_moving_mean': } >>> _parse_aux_states(fc2, {'batchnorm0_moving_var': mean_states}, None) ValueError: Symbol aux_states names and given aux_states do not match. RsaSymbol aux_states names and given aux_states do not match.symbol aux_names:%s, aux_states.keys:%scS`si|]\}}||qS(((RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s c `sLi|]B\}}tjj|dddkr<|jn|qS(R R;R(RCRDRR;(RR#R(R R;(s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s N(RRRRR RRRRRtlist_auxiliary_statesR)RRRRR(Rt aux_statesR R;t aux_names((R R;s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt_parse_aux_statess10 $% g-C6?c`sd}tjtjtjfks-tfd|jD}xV|jD]H\}} |j|j} | dkrV|| | d|j|(qVqVWx+|D]#}tj||dd||2s R{R;tordertCRg@RiN(RRRRR Rtarg_dictRoR~R;tkindRR1tprodR=travelRtaux_dictRRRtsum(texecutorRRtepstuse_forward_trainR;Rt approx_gradsR#RRot old_valueR5tkeyRxtf_pepstadstypetf_nepst approx_grad((R;s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt numeric_gradsH $  $ !   $!gMbP?g{Gz?c  `s!tjtjtjfks$ttjtjfkrQ|dksQtnd#kritnd} td|d|dd}d|jD} t d|d|dd}|d#k rd |jD} nd#} |d#kr |j }d |D}ndt |t t frTt |}d |D}n0t |tr~|j}|j}ntd |jD}|j|\}}}tjjd }||}tjj|}tt |jd tjj| |dddfg}tg|D].}|tjjddd||jf^q1d tjjddd|dfg}fd|jD}| d#k rOt | tstdxx| jD]g\}}||kr|tkr|dkrtjj||j||j||j||| numpy.ndarray, maps the name of arguments to the corresponding numpy.ndarray. *In either case, value of all the arguments must be provided.* aux_states : list or tuple or dict, optional The auxiliary states required when generating the executor for the symbol. numeric_eps : float, optional Delta for the finite difference method that approximates the gradient. check_eps : float, optional relative error eps used when comparing numeric grad to symbolic grad. grad_nodes : None or list or tuple or dict, optional Names of the nodes to check gradient on use_forward_train : bool Whether to use is_train=True when computing the finite-difference. ctx : Context, optional Check the gradient computation on the specified device. grad_stype_dict : dict of str->str, optional Storage type dictionary for gradient ndarrays. dtype: np.float16 or np.float32 or np.float64 Datatype for mx.nd.array. References --------- [1] https://github.com/Theano/Theano/blob/master/theano/gradient.py gh㈵>cS`stjj|d}|S(sGet a random weight matrix with not too small elements Parameters ---------- shape : list or tuple g?(RRR?(R=tplain((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytrandom_projections RRR R;cS`s%i|]\}}|j|qS((R(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s RcS`s%i|]\}}|j|qS((R(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s cS`si|]}d|qS(twrite((RR#((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s cS`si|]}d|qS(R((RR#((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s cS`s"i|]\}}|j|qS((R=(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s t __random_projig{Gz?Rc`s7i|]-\}}tjj|dd|qS(R R;(RCRDR(RR#R(R R;(s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s sgrad_stype_dict must be a dictR{tgrad_reqRt args_gradsTExecutor arg_arrays and and location len do not match.Got %d inputs and %d locationsiRc`s&i|]}j|j|qS((t grad_dictR(RR#(R(s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s RRRs NUMERICAL_%ss BACKWARD_%staddtnulls#Invalid grad_req %s for argument %sN()RRRRR RR RRRRRRRRRRR)t infer_shapeRCRtVariablet make_lossRDRRtnormalR=RROtcontextR;Rt arg_arraysRRRRBtbackwardRR(RRRt numeric_epsRRt grad_nodesRR tgrad_stype_dictR;Rt location_npytaux_states_npyR t input_shapeR|t out_shapetprojtoutR#t args_grad_npyR Rtinpstsymbolic_gradstnumeric_gradientstnametfd_gradt orig_gradtsym_grad((R R;Rs1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcheck_numeric_gradient[s*$   !       5, $!       c  `sdks0tjtjtjfks0tdkrHtntd|d|dd}td|d|dd}t |t rg|j D]} || ^q}nfd|j D} |j dd|d | d|} x| jD] } d | (q W| jd tg| jD]} | j^q7}xSt|j ||D]9\}}}t||||d |d |fd|qhW| jS(s4Compares a symbol's forward results with the expected ones. Prints error messages if the forward results are not the same as the expected ones. Parameters --------- sym : Symbol output symbol location : list of np.ndarray or dict of str to np.ndarray The evaluation point - if type is list of np.ndarray Contains all the numpy arrays corresponding to `sym.list_arguments()`. - if type is dict of str to np.ndarray Contains the mapping between argument names and their values. expected : list of np.ndarray or dict of str to np.ndarray The expected output value - if type is list of np.ndarray Contains arrays corresponding to exe.outputs. - if type is dict of str to np.ndarray Contains mapping between sym.list_output() and exe.outputs. check_eps : float, optional Relative error to check to. aux_states : list of np.ndarray of dict, optional - if type is list of np.ndarray Contains all the NumPy arrays corresponding to sym.list_auxiliary_states - if type is dict of str to np.ndarray Contains the mapping between names of auxiliary states and their values. ctx : Context, optional running context dtype: "asnumpy" or np.float16 or np.float32 or np.float64 If dtype is "asnumpy" then the mx.nd.array created will have the same type as th numpy array from which it is copied. Otherwise, dtype is the explicit datatype for all mx.nd.array objects created in this function. equal_nan: Boolean if True, `nan` is a valid value for checking equivalency (ie `nan` == `nan`) Example ------- >>> shape = (2, 2) >>> lhs = mx.symbol.Variable('lhs') >>> rhs = mx.symbol.Variable('rhs') >>> sym_dot = mx.symbol.dot(lhs, rhs) >>> mat1 = np.array([[1, 2], [3, 4]]) >>> mat2 = np.array([[5, 6], [7, 8]]) >>> ret_expected = np.array([[19, 22], [43, 50]]) >>> check_symbolic_forward(sym_dot, [mat1, mat2], [ret_expected]) RRRR R;Rc `sOi|]E\}}tjj|jdddkr?|jn|qS(R R;R(RCRDtemptyR=R;(RR#R(R R;(s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys #s RR iRs EXPECTED_%ss FORWARD_%sRN(RRRRR RR RRRRt list_outputsRRt grad_arraysRRtRRRR(RRtexpectedRRRR RR;R#targs_grad_dataRtgRRt output_nametexpectRb((R R;s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcheck_symbolic_forwards(50  ! &$ "( gh㈵>Rc  `sV| tjtjtjfks$t|dkr<t}ntd|d|d|d| }td|d|d|d| }t |t t frdt |j |D}nd|jD} i} x| jD]\}}tjj|d|d| }| dk rp|| krp| |}|dk r]|dkr]t|j|d d }n|}|| |str dictionary of mapping argument name to stype for the gradient equal_nan: Boolean if True, `nan` is a valid value for checking equivalency (ie `nan` == `nan`) dtype: np.float16 or np.float32 or np.float64 Datatype for mx.nd.array. Example ------- >>> lhs = mx.symbol.Variable('lhs') >>> rhs = mx.symbol.Variable('rhs') >>> sym_add = mx.symbol.elemwise_add(lhs, rhs) >>> mat1 = np.array([[1, 2], [3, 4]]) >>> mat2 = np.array([[5, 6], [7, 8]]) >>> grad1 = mx.nd.zeros(shape) >>> grad2 = mx.nd.zeros(shape) >>> exec_add = sym_add.bind(default_context(), args={'lhs': mat1, 'rhs': mat2}, ... args_grad={'lhs': grad1, 'rhs': grad2}, grad_req={'lhs': 'write', 'rhs': 'write'}) >>> exec_add.forward(is_train=True) >>> ograd = mx.nd.ones(shape) >>> grad_expected = ograd.copy().asnumpy() >>> check_symbolic_backward(sym_add, [mat1, mat2], [ograd], [grad_expected, grad_expected]) RRR R;RcS`si|]\}}||qS(((RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys xs cS`s1i|]'\}}tjjd|j|qS(R(RRRR=(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys zs R{R,gc`si|]}|qS(((RR#(R (s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s cS`si|]\}}||qS(((RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s RR R RcS`s%i|]\}}|j|qS((R(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s Rs EXPECTED_%ss BACKWARD_%sRR R s#Invalid grad_req %s for argument %sN(RRRRR RR RRRRRRRRRCRDRRR=RRRRBRtappendRRRR)(RRt out_gradsR*RRRR R t grad_stypesRR;RR+R#RRDRoRRtoutgRytgradsR"((R s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcheck_symbolic_backward3st=$  ! "  "  (  %      itwholecK`sl|d krt}n|d kr-d}n|d krp|jd|d||}d|jjD}nPt|tstdt||jd|d|d|jD}x;|jD]-\}} | j |j|j |j|(qW|dkr|j dt |j d |jx|jD]} | jq7Wtj} x7t|D])} |j dt |j d |jqdWtjjtj} | | d |}|S|d kr\|j dtx|jD]} | jqWtj} x$t|D]} |j dtqWtjjtj} | | d |}|Std d S(sSCheck the running speed of a symbol. Parameters ---------- sym : Symbol Symbol to run the speed test. location : none or dict of str to np.ndarray Location to evaluate the inner executor. ctx : Context Running context. N : int, optional Repeat times. grad_req : None or str or list of str or dict of str to str, optional Gradient requirements. typ : str, optional "whole" or "forward" - "whole" Test the forward_backward speed. - "forward" Only test the forward speed. RR R cS`s7i|]-\}}tjjd|jdd|qS(Rtscaleg?(RRRR=(RR#Ry((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s sExpect dict, get "location"=%scS`s"i|]\}}|j|qS((R=(RR#R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pys s R6RR1g?Rs%typ can only be "whole" or "forward".N(RR t simple_bindRRRRR RRR;RRBRRt wait_to_readttimeR1RCRDtwaitallRtR)(RRR tNR ttypRRR"tiarrRbtticR|ttoctforward_backward_timet forward_time((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt check_speedsL     %%        g?c  C`s|dkridtjtj6dtjtj6dtjtj6dtjtj6dtjtj6dtjtj6}nt |t j ri|tjtj6|tjtj6|tjtj6|tjtj6|tjtj6|tjtj6}nt |dks,t t |trQ|gt |}nt |t |ksot |dj} |dj} g}xit||D]X\}}|j| kst |j| kst |j|jd||qW|dkrin|}|dkr+in|}x|djjD]\}}||krE| rtjjddd d d |jj| ||>> # create the symbol >>> sym = mx.sym.Convolution(num_filter=3, kernel=(3,3), name='conv') >>> # initialize the running context >>> ctx_list =[{'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float64}}, {'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float32}}, {'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float16}}, {'ctx': mx.cpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float64}}, {'ctx': mx.cpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float32}}] >>> check_consistency(sym, ctx_list) >>> sym = mx.sym.Concat(name='concat', num_args=2) >>> ctx_list = [{'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10), 'type_dict': {'concat_arg0': np.float64, 'concat_arg1': np.float64}}, {'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10), 'type_dict': {'concat_arg0': np.float32, 'concat_arg1': np.float32}}, {'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10), 'type_dict': {'concat_arg0': np.float16, 'concat_arg1': np.float16}}, {'ctx': mx.cpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10), 'type_dict': {'concat_arg0': np.float64, 'concat_arg1': np.float64}}, {'ctx': mx.cpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10), 'type_dict': {'concat_arg0': np.float32, 'concat_arg1': np.float32}}] >>> check_consistency(sym, ctx_list) g?gMbP?gh㈵>iiR tlowgq= ףpthighgq= ףp?RR7R R;R RRRRs#Predict Err: ctx %d vs ctx %d at %ss!Train Err: ctx %d vs ctx %d at %sN(/RRR;RRRtuint8tint32RRtnumberstNumberRR RR(RRR0R8RRRR&R=RRRR)RORRt output_dictRtupdateR RRtt enumerateRRtprintt tracebackt print_excRRBR(Rtctx_listR7R t arg_paramst aux_paramsRt raise_on_errt ground_truthRt use_uniformt rand_typet output_namest arg_namestexe_listRR RRyRR"tdtypestmax_idxtgtR5tgtarrRtcurr((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcheck_consistencys3        ',     (       (   cC`sttjjS(sReturn a list of GPUs Returns ------- list of int: If there are n GPUs, then return a list [0,1,...,n-1]. Otherwise returns []. (R1RCtutilt get_gpu_count(((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt list_gpuss ic C`s,|dkstd|dkr:|jdd}n|dkr[tjj|}ntjj||}|dkrtjj|sy!tj d|tj |Wqt k r}|j t j krt d|qqXqn| rtjj|rtj d||Sx|d dkrytj|d t}|jd ksgtd |t|d ;}x0|jddD]}|r|j|qqWPWdQXWq tk r} |d 8}|dkr| qtdj||d krdndq Xq Wtj d|||S(sDownload an given URL Parameters ---------- url : str URL to download fname : str, optional filename of the downloaded file. If None, then will guess a filename from url. dirname : str, optional output directory name. If None, then guess from fname or use the current directory overwrite : bool, optional Default is false, which means skipping download if the local file exists. If true, then download the url to overwrite the local file if exists. retries : integer, default 5 The number of times to attempt the download in case of failure or non 200 return codes Returns ------- str The filename of the downloaded file is&Number of retries should be at least 0t/itscreate directory %ssfailed to create s%s exists, skipping downloaditstreamisfailed to open %stwbt chunk_sizeiNs,download failed, retrying, {} attempt{} leftRs"downloaded %s into %s successfully(R RtsplittostpathtdirnametjointexiststloggingtinfotmakedirstOSErrorterrnotEEXISTtrequeststgetRBt status_codetopent iter_contentRt ExceptionRMR<( turltfnameRkt overwritetretriestexctrRtchunkR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytdownloadsD        'cC`sid}d}||d|d\}}||d|d\}}i|d6|d6|d 6|d 6S( smDownload and load the MNIST dataset Returns ------- dict A dict containing the data c S`stjtjj|>}tjd|jdtj |jdtj }WdQXtjtjj|d}tjd|jd\}}}}tj |jdtj j t |||}|j |jddd d jtjd }WdQX||fS( Ns>IIiR;trbs>IIIIiiiii(tgzipRwRCt test_utilsRtstructtunpacktreadRt frombuffertint8RFRRR=RR( t label_urlt image_urltflbltlabeltfimgR|trowstcolstimage((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt read_datas$!'35s http://data.mxnet.io/data/mnist/strain-labels-idx1-ubyte.gzstrain-images-idx3-ubyte.gzst10k-labels-idx1-ubyte.gzst10k-images-idx3-ubyte.gzt train_datat train_labelt test_datat test_label((RRjt train_lblt train_imgttest_lblttest_img((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt get_mnists cC`sKtjjds"tjdntjjdsGtdddndS(skDownloads MNIST dataset as a pkl.gz into a directory in the current directory with the name `data` R@sdata/mnist.pkl.gzs/http://deeplearning.net/data/mnist/mnist.pkl.gzRkN(RiRjtisdirRpRmR(((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt get_mnist_pkls  cC`stjjds"tjdntjjd sntjjd sntjjd sntjjd rtddd}tj|}|jdWdQXndS( sDownloads ubyte version of the MNIST dataset into a directory in the current directory with the name `data` and extracts all files in the zip archive to this directory. R@sdata/train-images-idx3-ubytesdata/train-labels-idx1-ubytesdata/t10k-images-idx3-ubytesdata/t10k-labels-idx1-ubytes)http://data.mxnet.io/mxnet/data/mnist.zipRkN( RiRjRRpRmRtzipfiletZipFilet extractall(t zip_file_pathtzf((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_mnist_ubytes  cC`stjjds"tjdntjjd sntjjd sntjjd sntjjd rtddd}tj|}|jdWdQXndS( sDownloads CIFAR10 dataset into a directory in the current directory with the name `data`, and then extracts all files into the directory `data/cifar`. R@sdata/cifar/train.recsdata/cifar/test.recsdata/cifar/train.lstsdata/cifar/test.lsts+http://data.mxnet.io/mxnet/data/cifar10.zipRkN( RiRjRRpRmRRRR(RR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt get_cifar10)s  cC`stt|dkrtnt}tjjddddd|d|dtd |d |d |}tjjdd dd d|d|d |d |d |}||fS(s@Returns training and validation iterators for MNIST dataset iRsdata/train-images-idx3-ubyteRsdata/train-labels-idx1-ubyteRt batch_sizeR!tflatt num_partst part_indexsdata/t10k-images-idx3-ubytesdata/t10k-labels-idx1-ubyte(RRRtRBRCtiot MNISTIter(RRRRRttrain_dataitert val_dataiter((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_mnist_iterator8s(    cC`sctjj||}tjj|s_t|d|dttj|}|jd|ndS(sDownload and extract zip data. Parameters ---------- data_dir : str Absolute or relative path of the directory name to store zip files url : str URL to download data from data_origin_name : str Name of the downloaded zip file Examples -------- >>> get_zip_data("data_dir", "http://files.grouplens.org/datasets/movielens/ml-10m.zip", "ml-10m.zip") RkR|RjN( RiRjRlRmRRtRRR(tdata_dirRztdata_origin_nametzip_file((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt get_zip_dataTs cC`stjj||}tjj||}tjj|st|d|d|dttj|d}t|d/}x|D]}|j |qW|j WdQXtj |ndS(s:Download and extract bz2 data. Parameters ---------- data_dir : str Absolute or relative path of the directory name to store bz2 files data_name : str Name of the output file in which bz2 contents will be extracted url : str URL to download data from data_origin_name : str Name of the downloaded b2 file Examples -------- >>> get_bz2_data("data_dir", "kdda.t", "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/kdda.t.bz2", "kdda.t.bz2") R{RkR|RRfN( RiRjRlRmRRttbz2tBZ2FileRwRtclosetremove(Rt data_nameRzRtbz_filetfouttline((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt get_bz2_datams RdcC`s&tjj||}|tj|<|S(sVSet environment variable Parameters ---------- key : str Env var to set val : str New value assigned to the env var default_val : str, optional Default value returned if the env var doesn't exist Returns ------- str The value of env var before it is set to the new value (RitenvironRu(RRxt default_valtprev_val((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt set_env_vars cC`sY|d7(t|j|js5|d8(tS|d8(t|j|jS(s%Check whether two NDArrays sharing the same memory block Parameters ---------- array1 : NDArray First NDArray to be checked array2 : NDArray Second NDArray to be checked Returns ------- bool Whether two NDArrays share the same memory i(RRRt(tarray1tarray2((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt same_arrays    c c`sttjdu}yUtjj}tj|}ztj|j|dVWdtj||XWntk rdVnXWdQXdS(sc Discards error output of a routine if invoked as: with discard_stderr(): ... twN( RwRitdevnulltsyststderrtfilenotduptdup2tAttributeError(t bit_buckett stderr_filenot old_stderr((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytdiscard_stderrs  t DummyItercB`s)eZdZdZdZdZRS(sA dummy iterator that always returns the same batch of data (the first data batch of the real data iter). This is usually used for speed testing. Parameters ---------- real_iter: mx.io.DataIter The real data iterator where the first batch of data comes from cC`sStt|j||_|j|_|j|_|j|_t||_dS(N( tsuperRt__init__t real_itert provide_datat provide_labelRtnextt the_batch(tselfR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyRs     cC`s|S(N((R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt__iter__scC`s|jS(sGet a data batch from iterator. The first data batch of real iter is always returned. StopIteration will never be raised. Returns ------- DataBatch The data of next batch. (R(R((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyRs (t__name__t __module__t__doc__RRR(((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyRs  cC`s|dkstgt|D]}d|^q}gt|D]6}||t|||dt|f^qB}||fS(sGenerate the buckets and probabilities for chi_square test when the ppf (Quantile function) is specified. Parameters ---------- ppf : function The Quantile function that takes a probability and maps it back to a value. It's the inverse of the cdf function nbuckets : int size of the buckets Returns ------- buckets : list of tuple The generated buckets probs : list The generate probabilities ig?i(R R1tfloat(tppftnbucketsR|tprobsR5tbuckets((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytgen_buckets_probs_with_ppfs#Ii@BcC`sgtj||}|j}||d|tj|ko`||d|tj|k}|S(shTest the generator by matching the mean. We test the sample mean by checking if it falls inside the range (mu - 3 * sigma / sqrt(n), mu + 3 * sigma / sqrt(n)) References:: @incollection{goucher2009beautiful, title={Beautiful Testing: Leading Professionals Reveal How They Improve Software}, author={Goucher, Adam and Riley, Tim}, year={2009}, chapter=10 } Examples:: generator = lambda x: np.random.normal(0, 1.0, size=x) mean_check_ret = mean_check(generator, 0, 1.0) Parameters ---------- generator : function The generator function. It's expected to generate N i.i.d samples by calling generator(N). mu : float sigma : float nsamples : int Returns ------- ret : bool Whether the mean test succeeds i(RRtmeantsqrt(t generatortmutsigmatnsamplestsamplest sample_meanR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt mean_check s ! !!t MXNET_HOMEcC`s|tjkrtj|}ntjjtj}tjj|dd}tjj|rb|Stjj|dddd}tjj|r|StddS(sGet path to the im2rec.py tool Parameters ---------- home_env : str Env variable that holds the path to the MXNET folder Returns ------- str The path to im2rec.py ttoolss im2rec.pys..s&Could not find path to tools/im2rec.pyN( RiRRjRkRCt__file__RltisfiletIOError(thome_envt mxnet_patht im2rec_path((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytget_im2rec_path0scC`stj||}|jdd}||ddtjd|d|dko||ddtjd|d|dk}|S(sTest the generator by matching the variance. It will need a large number of samples and is not recommended to use We test the sample variance by checking if it falls inside the range (sigma^2 - 3 * sqrt(2 * sigma^4 / (n-1)), sigma^2 + 3 * sqrt(2 * sigma^4 / (n-1))) References:: @incollection{goucher2009beautiful, title={Beautiful Testing: Leading Professionals Reveal How They Improve Software}, author={Goucher, Adam and Riley, Tim}, year={2009}, chapter=10 } Examples:: generator = lambda x: np.random.normal(0, 1.0, size=x) var_check_ret = var_check(generator, 0, 1.0) Parameters ---------- generator : function The generator function. It's expected to generate N i.i.d samples by calling generator(N). sigma : float nsamples : int Returns ------- ret : bool Whether the variance test succeeds tddofiiii(RRRR(RRRRt sample_varR((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyt var_checkNs !11c C`s\tstdnt|ts*t||}t|t|ksTtt|dttfrIt}tj t|dfdtj }xt |D]\}}||d||dkst|t|dkr||d||ddkstn||d||d<||d||dd 0.05 (alpha) threshold to get the final result. Examples:: buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.norm.ppf(x, 0, 1), 5) generator = lambda x: np.random.normal(0, 1.0, size=x) p = chi_square_check(generator=generator, buckets=buckets, probs=probs) assert(p > 0.05) Parameters ---------- generator: function A function that is assumed to generate i.i.d samples from a specific distribution. generator(N) should generate N random samples. buckets: list of tuple or list of number The buckets to run the chi-square the test. Make sure that the buckets cover the whole range of the distribution. Also, the buckets must be in ascending order and have no intersection probs: list or tuple The ground-truth probability of the random value fall in a specific bucket. nsamples:int The number of samples to generate for the testing Returns ------- p : float p value that the generator has the expected distribution. A higher value indicates a larger confidence obs_freq : list Observed frequency of buckets expected_freq : list The expected (ground-truth) frequency of the buckets sPscipy is not available. Please check if the scipy python bindings are installed.iiR;itsidetrightR=tf_obstf_exp(tssRFRRR RRRBRRORRLRtRRRGt searchsortedR(Rt chisquare( RRRRRtcontinuous_distt buckets_npyR5R|t expected_freqtsample_bucket_idstobs_freqtp((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytchi_square_checkus8, %")!( !g?g?c C`sg}g}g} xet|D]W} td|d|d|d|\} } } |j| |j| | j| qWtj||kj}|||krtdt|t|t| t|t|fn|S(s}Verify whether the generator is correct using chi-square testing. The test is repeated for "nrepeat" times and we check if the success rate is above the threshold (25% by default). Parameters ---------- generator: function A function that is assumed to generate i.i.d samples from a specific distribution. generator(N) should generate N random samples. buckets: list of tuple or list of number The buckets to run the chi-square the test. Make sure that the buckets cover the whole range of the distribution. Also, the buckets must be in ascending order and have no intersection probs: list or tuple The ground-truth probability of the random value fall in a specific bucket. nsamples: int The number of samples to generate for the testing nrepeat: int The times to repeat the test success_rate: float The desired success rate alpha: float The desired threshold for type-I error i.e. when a true null hypothesis is rejected Returns ------- cs_ret_l: list The p values of the chi-square test. RRRRsZGenerator test fails, Chi-square p=%s, obs_freq=%s, expected_freq=%s. buckets=%s, probs=%s(R1RR0RRRR R(RRRRtnrepeatt success_ratetalphatcs_ret_lt obs_freq_ltexpected_freq_lR|tcs_retRRt success_num((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytverify_generators  cC`s|dk r|dk rt|tr]xXt||D]\}}t||||q7Wqt|j|jd|d|ndS(sCompare ndarray tuple.RRN(RRRRtcompare_ndarray_tupleRR(tt1tt2RRts1ts2((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyR s c C`s|dkrEtjjd|dtd|} | jt} nQ|dkrt||ddd|} | jtjd} n td |dkrtjjd|dtd|} | jt} nK|dkrt||d|} | jtjd} n td |jd | } |jd | }|r`t | |n|j d | | | |j d | | ||rt | |d |d |nt | j | j d |d |d S(sCompare opt1 and opt2.R{R=R R;RnR3R,istype not supported yetiRRN(RnR3(RnR3( RCRR&R tcopytoR}RGRytcreate_state_multi_precisionR tupdate_multi_precisionRR(topt1topt2R=R;tw_stypetg_stypeRRtcompare_statestw2tw1tg2tg1tstate1tstate2((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pytcompare_optimizers. $   $  t EnvManagercB`s)eZdZdZdZdZRS(s7Environment variable setter and unsetter via with idiomcC`s||_||_d|_dS(N(t_keyt _next_valRt _prev_val(RRRx((s1/tmp/pip-install-Qvdv_2/mxnet/mxnet/test_utils.pyRs  cC`s/tjj|j|_|jtj|js                        2    V          -  <AI   L I   F     !    '  ' N0