v]c @@s:dZddlmZmZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddl mZddl mZdd l mZdd l mZdd l mZdd lmZmZdd lmZddlmZddlmZmZm Z ddl m!Z!ddl"m#Z#e$Z%yddl&m'Z'e'Z%Wne(k rse)Z*nXe dddddgZ+dZ,dZ-dZ.dZ/dZ0dddZ2dZ3ddddddddddd Z4e5d Z6d!Z7dd"l8m9Z9d#e%fd$YZ:dS(%sMXNet model modulei(tabsolute_importtprint_functionN(t namedtuplei(tio(tndarray(tsymbol(t optimizer(tmetric(tkvstore(tContexttcpu(tUniform(t get_updater(tDataParallelExecutorManagert_check_argumentst _load_data(tDataDesc(t mx_real_t(t BaseEstimatortBatchEndParamstepochtnbatcht eval_metrictlocalscC@s\t}t|tjr!|}n1t|trBtj|}ntd|||fS(sCreate kvstore assuming some parameters' storage types are row_sparse. Parameters ---------- kvstore : KVStore or str The kvstore. Returns ------- kvstore : KVStore update_on_kvstore : bool. Always True. sWCannot create '%s' KVStore with row_sparse parameters. The type must be KVStore or str.(tTruet isinstancetkvstKVStoretstrtcreatet TypeError(Rtupdate_on_kvstoretkv((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_create_sparse_kvstore:s  cC@stttjdd}|d kr3d }nt|tjrN|}nt|tr|dkr~d|kr~d }qtj |}|dkrt d|j D}|d krt }qqn t d |d krt }n||fS( sWCreate kvstore This function select and create a proper kvstore if given the kvstore type. Parameters ---------- kvstore : KVStore or str The kvstore. num_device : int The number of devices arg_params : dict of str to `NDArray`. Model parameter, dict of name to `NDArray` of net's weights. tMXNET_UPDATE_ON_KVSTOREt1itdisttlocalcs@s!|]}tj|jVqdS(N(tnptprodtshape(t.0tparam((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys msiis$kvstore must be KVStore, str or NoneNii(tbooltinttostgetenvtNoneRRRRRtmaxtvaluestFalseR(Rt num_devicet arg_paramsRR tmax_size((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_create_kvstoreRs$           cC@s_xXt|D]J\}}||}|j||||r |j||d| q q WdS(sInitialize kvstoretpriorityN(t enumeratetinittpull(Rt param_arraysR4t param_namesRtidxt param_on_devstname((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_initialize_kvstoreys  cC@sCgt|D]"\}}|ddk r |^q }g|D]}||^q<}g|D]}||^qY} g|D]}||^qv} t|} d} d} ttjd| }x| | kr>| || kr| |n| }|j| | |!|| |!d| |j| | |!| | |!d| |} qWdS(s@Perform update of param_arrays from grad_arrays on NCCL kvstore.it16tMXNET_UPDATE_AGGREGATION_SIZER7N(R8R/tlenR,R-R.tpushR:(R;t grad_arraysRR<tindext grad_listt valid_indicestitvalid_grad_arraystvalid_param_arraystvalid_param_namestsizetstartt default_batchtbatchtend((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_update_params_on_kvstore_nccls2  %%c C@sx}tt||D]f\}}|\}}|ddkrDqn||}|j||d| |j||d| qWdS(s;Perform update of param_arrays from grad_arrays on kvstore.iR7N(R8tzipR/RDR:( R;RERR<RFtpairtarg_listRGR?((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_update_params_on_kvstores"  cC@sEgt|D] }g^q }xtt||D]\}} | \} } | ddkrcq5n|} |r|| } |j| | d| |j| | d| nxQtt| | D]:\}}|\}}||j| ||||fqWq5Wx<|D]4}|r t|\}}}||||q q WdS(s?Perform update of param_arrays from grad_arrays not on kvstore.iR7N(trangeR8RSR/RDR:tappend(R;REtupdaterR3RR<t_tupdatesRIRTRURGRFR?tktptwtgt dev_updates((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_update_paramss""  " * cO@sKt|tr1x|D]}|||qWdS|rG|||ndS(sSends args and kwargs to any configured callbacks. This handles the cases where the 'callbacks' variable is ``None``, a single function, or a list. N(Rtlist(t callbackstargstkwargstcb((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_multiple_callbackss  c%C@s<|dkrt}ntd|d|d|d| d|d|d|d|d | }|rj|j|n|j||| st| }n | j| | rtd | d |jd |d|j d | n| j x[t ||D]J}t j }|j d}xt rt }xm| D]e}|j||dk rN|jn|jdt |j| rd| jkrt|j|j| |j qt|j|j| |j n4t|j|jd|dt|d | d|j |dk r|jn|j||j|d7}|dk retd|d|d|dt}t||n| dk r"|| kr"t}Pq"q"W|r|jd|| j n| dks|| krPqqWt j } |jd|| ||s|d|kr|j ||nt||||||r|j |j d}!xt!|D]\}"}#|j|#|jdt|j||#j|dk rtd|d|"d|dt}t||n|!d7}!q]W|dk r'td|d|!d|dt}$t||$n|j qqWdS(sInternal training function on multiple devices. This function will also work for single device as well. Parameters ---------- symbol : Symbol The network configuration. ctx : list of Context The training devices. arg_names: list of str Name of all arguments of the network. param_names: list of str Name of all trainable parameters of the network. aux_names: list of str Name of all auxiliary states of the network. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. begin_epoch : int The begining training epoch. end_epoch : int The end training epoch. epoch_size : int, optional Number of batches in a epoch. In default, it is set to ``ceil(num_train_examples / batch_size)``. optimizer : Optimizer The optimization algorithm train_data : DataIter Training data iterator. eval_data : DataIter Validation data iterator. eval_metric : EvalMetric An evaluation function or a list of evaluation functions. epoch_end_callback : callable(epoch, symbol, arg_params, aux_states) A callback that is invoked at end of each epoch. This can be used to checkpoint model each epoch. batch_end_callback : callable(BatchEndParams) A callback that is invoked at end of each batch. This can be used to measure speed, get result from evaluation metric. etc. kvstore : KVStore The KVStore. update_on_kvstore : bool Whether or not perform weight updating on kvstore. logger : logging logger When not specified, default logger will be used. work_load_list : list of float or int, optional The list of work load for different devices, in the same order as ``ctx``. monitor : Monitor, optional Monitor installed to executor, for monitoring outputs, weights, and gradients for debugging. Notes ----- - This function will inplace update the NDArrays in `arg_params` and `aux_states`. Rtsym_gentctxt train_dataR<t arg_namest aux_namestwork_load_listtloggerRR;R4Ritis_traintncclRYR3iRRRRs!Epoch[%d] Resetting Data IteratorsEpoch[%d] Time cost=%.3fN("R/tloggingR tinstall_monitort set_paramsR t set_optimizerR@R;R<tresetRWttimeRtload_data_batchttictforwardtbackwardttypeRRRERVRaRCt toc_printt update_metrictlabelt BatchEndParamRRgR2tinfotcopy_toR8(%RRiRkR<RlR4t aux_paramst begin_epocht end_epocht epoch_sizeRRRRjt eval_dataRtepoch_end_callbacktbatch_end_callbackRnRmtmonitorteval_end_callbackteval_batch_end_callbackRhtexecutor_managerRYRRxRtdo_resett data_batchtbatch_end_paramsttocttotal_num_batchRIt eval_batchteval_end_params((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_train_multi_devicesA                                         cC@s|dk r&|jd|d|nd|jD}|jd|jDd||f}tj||tjd|dS(sCheckpoint the model data into file. Parameters ---------- prefix : str Prefix of model name. epoch : int The epoch number of the model. symbol : Symbol The input Symbol. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. remove_amp_cast : bool, optional Whether to remove the amp_cast and amp_multicast operators, before saving the model. Notes ----- - ``prefix-symbol.json`` will be saved for symbol. - ``prefix-epoch.params`` will be saved for parameters. s%s-symbol.jsontremove_amp_castcS@s/i|]%\}}|jtd|qS(sarg:%s(t as_in_contextR (R)R\tv((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys s cS@s/i|]%\}}|jtd|qS(saux:%s(RR (R)R\R((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys s s%s-%04d.paramssSaved checkpoint to "%s"N(R/tsavetitemstupdatetndRqR(tprefixRRR4RRt save_dictt param_name((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pytsave_checkpoints c C@stjd|}tjd||f}i}i}xd|jD]V\}}|jdd\}} |dkr||| 2s c@s+i|]!\}}|kr||qS(((R)R\R(Rl(s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys 6s ( RRR/RRRRR4tsettlist_argumentsRRtlist_auxiliary_states(R((RkRls,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyR$s     % cC@s|jdp|jdS(s!Check if name is a data argument.tdataR~(tendswith(R?((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt _is_data_arg:scC@spg|D]'}t|tr"|n t|^q}d|D}|jj|\}}}|dk sqtd|D}|jj|\} }} | dk st|jj} |j} g| D]} | | kr| ^q}|jj }gt | || D]}|d|kr|^q}d|D}gt ||| D]}|d|krU|^qU}d|D}x_|j D]Q\}}|j r||j kr| r|j |||(q|j ||qWx_|j D]Q\}}|jr7||jkr7| r7|j|||(q|j ||qW||_ ||_| t||fS(s2Initialize weight parameters and auxiliary states.cS@si|]}|j|jqS((R(R?(R)titem((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys Bs cS@si|]}|j|jqS((tdtypeR?(R)R((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys Es icS@s4i|]*\}}}tjd|d||qS(R(R(Rtzeros(R)R\tstt((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys Ps cS@s4i|]*\}}}tjd|d||qS(R(R(RR(R)R\RR((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys Ts N(RRRt infer_shapeR/Rt infer_typeRtkeysRRSRR4RRRb(Rtinputst overwritetxt input_shapest arg_shapesRZt aux_shapest input_dtypest arg_dtypest aux_dtypesRkt input_namestkeyR<Rltparam_name_attrsR4taux_name_attrsRR\R((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt _init_params?s<4 %    cC@s|jj}d|d<|S(NR(t__dict__RR/(Rtthis((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt __getstate__gs cC@s|jj|dS(N(RR(Rtstate((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt __setstate__lsc @sfdjD}|jt|jdk rjj|\}}}|dk sntdgjjD]}|j ^q{}||krdSnjj j dddd||}|j jj tj|_dS(s7Initialize the predictor module for running prediction.c@s#i|]}j|j|qS((R4R((R)R?(R(s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys qs sIncomplete input shapesNitgrad_reqtnullt type_dict(R4RtdictRR/RRRt arg_arraysR(t simple_bindRitcopy_params_fromRR( RRRtshapesRRZRt pred_shapest pred_exec((Rs,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_init_predictoros"   cC@st|tjtjfr^|d krU|r<tdqUtj|jd}nt|tjtjfst dn|jd|jdkrtdn|j dkr|jddkr|j }n|j dkrtdn|r/t j ||t|jd|jd|d d St j ||t|jd|jdtSnt|t jst d n|S( s$Initialize the iterator given input.s+y must be specified when X is numpy.ndarrayis)y must be ndarray when X is numpy.ndarrays/The numbers of data points and labels not equaliis3Label must be 1D or 2D (with 2nd dimension being 1)tshuffletlast_batch_handlet roll_overs,X must be DataIter, NDArray or numpy.ndarrayN(RR&RRtNDArrayR/t ValueErrorRR(RtndimtflattenRt NDArrayItertminRR2tDataIter(RtXtyRo((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt _init_iters( "% /cC@s|dkr|St|ttfrt|dkr|ddk r|ddkrut|dtjru|dSt|dtrtj|dn|d}t|dtrtj|dn|d}|j ||dt St dnt|tjst dn|S(s(Initialize the iterator given eval_data.iiiRosEval data is NONEs\Eval data must be DataIter, or NDArray/numpy.ndarray/list pair (i.e. tuple/list of length 2)N( R/RttupleRbRCRRR&tarrayRRRR(RRt input_datat input_label((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pyt_init_eval_iters '&& & cC@so|j|ddt}|r+|jn|j}g|D]}|d^q;}td|jjD}xA|jD]6}t|t r|j ||j t|jD]*\}}||j|d|!jq6Wn|d7}|dk ri||kriPqiqiWg| D]}tj|^q}t|dkr|d}n|rgg|D]}tj|^q}g|D]}tj|^q}t|dkr;|d}nt|dkrZ|d}n|||fS|SdS(sRun the prediction, always only use one device. Parameters ---------- X : mxnet.DataIter num_batch : int or None The number of batch to run. Go though all batches if ``None``. Returns ------- y : numpy.ndarray or a list of numpy.ndarray if the network has multiple outputs. The predicted value of the output. Roics@s$|]\}}||jfVqdS(N(R(R)Rtvalue((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pys siN( RR/R2Rut provide_dataRR4RRRRR?RRt batch_sizeRtarg_dictRWRCtoutputst provide_labelRRytpadRSRXtasnumpyR8RR~R&t concatenate(RRt num_batcht return_dataRut data_shapesRt data_namesRRR?t data_arraysRZt output_listt data_listt label_listRIRPtpaddedt real_sizeto_listto_ndtjRRR~((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pytpredictsZ    #+    ""% " ""   taccc C@st|tjs$tj|}n|j|d dt}|rO|jn|j}g|D]}|d^q_}t d|j j D} xA|jD]6}t|t r|j | |j sRRRRiN(RRt EvalMetricRRR/R2RuRRR4RRRR?RRRRR8RRyRR~RRRRgtget(RRRRRRuRRRRR?RRIRPR((s,/tmp/pip-install-Qvdv_2/mxnet/mxnet/model.pytscores6  #    R%c *C@s|j||dt} |j|}|jrR|j| j|_|jn|j|jd<|j| j | j \}}}t |t j st j|}nt|t|j|j\}}i}|r|jt|nXxUt|D]G\}}x8tt|jD]!}|||t|j|s\           '     $