[c @@s7dZddlmZmZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddl mZddl mZdd l mZdd l mZdd l mZdd lmZmZdd lmZddlmZddlmZmZm Z ddl m!Z!ddl"m#Z#e$Z%yddl&m'Z'e'Z%Wne(k rse)Z*nXe dddddgZ+dZ,dZ-dZ.dZ/dZ0dddZ2dZ3ddddddddddd Z4d Z5d!Z6dd"l7m8Z8d#e%fd$YZ9dS(%sMXNet model modulei(tabsolute_importtprint_functionN(t namedtuplei(tio(tndarray(tsymbol(t optimizer(tmetric(tkvstore(tContexttcpu(tUniform(t get_updater(tDataParallelExecutorManagert_check_argumentst _load_data(tDataDesc(t mx_real_t(t BaseEstimatortBatchEndParamstepochtnbatcht eval_metrictlocalscC@s\t}t|tjr!|}n1t|trBtj|}ntd|||fS(sCreate kvstore assuming some parameters' storage types are row_sparse. Parameters ---------- kvstore : KVStore or str The kvstore. sWCannot create '%s' KVStore with row_sparse parameters. The type must be KVStore or str.(tTruet isinstancetkvstKVStoretstrtcreatet TypeError(Rtupdate_on_kvstoretkv((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_create_sparse_kvstore:s   cC@st}|dkrd}nt|tjr6|}nt|tr|dkrfd|krfd}qtj|}|dkrtd|jD}|d krt }qqn t d|dkrt }n||fS( sWCreate kvstore This function select and create a proper kvstore if given the kvstore type. Parameters ---------- kvstore : KVStore or str The kvstore. num_device : int The number of devices arg_params : dict of str to `NDArray`. Model parameter, dict of name to `NDArray` of net's weights. itdisttlocalcs@s!|]}tj|jVqdS(N(tnptprodtshape(t.0tparam((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys hsiis$kvstore must be KVStore, str or NoneNii( RtNoneRRRRRtmaxtvaluestFalseR(Rt num_devicet arg_paramsRR tmax_size((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_create_kvstoreMs$           cC@s_xXt|D]J\}}||}|j||||r |j||d| q q WdS(sInitialize kvstoretpriorityN(t enumeratetinittpull(Rt param_arraysR.t param_namesRtidxt param_on_devstname((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_initialize_kvstorets  cC@sCgt|D]"\}}|ddk r |^q }g|D]}||^q<}g|D]}||^qY} g|D]}||^qv} t|} d} d} ttjd| }x| | kr>| || kr| |n| }|j| | |!|| |!d| |j| | |!| | |!d| |} qWdS(s@Perform update of param_arrays from grad_arrays on NCCL kvstore.it16tMXNET_UPDATE_AGGREGATION_SIZER1N(R2R)tlentinttostgetenvtpushR4(R5t grad_arraysRR6tindext grad_listt valid_indicestitvalid_grad_arraystvalid_param_arraystvalid_param_namestsizetstartt default_batchtbatchtend((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_update_params_on_kvstore_nccl}s2  %%c C@sx}tt||D]f\}}|\}}|ddkrDqn||}|j||d| |j||d| qWdS(s;Perform update of param_arrays from grad_arrays on kvstore.iR1N(R2tzipR)RAR4( R5RBRR6RCtpairtarg_listRDR9((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_update_params_on_kvstores"  cC@sxtt||D]\}}|\}} | ddkrDqn|} |r|| } |j| | d| |j| | d| nxGtt|| D]0\} } | \}}|| || ||qWqWdS(s?Perform update of param_arrays from grad_arrays not on kvstore.iR1N(R2RPR)RAR4(R5RBtupdaterR-RR6RFRQRRRDRCR9tktptwtg((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_update_paramss"  " cO@sKt|tr1x|D]}|||qWdS|rG|||ndS(sSends args and kwargs to any configured callbacks. This handles the cases where the 'callbacks' variable is ``None``, a single function, or a list. N(Rtlist(t callbackstargstkwargstcb((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_multiple_callbackss  c%C@s<|dkrt}ntd|d|d|d| d|d|d|d|d | }|rj|j|n|j||| st| }n | j| | rtd | d |jd |d|j d | n| j x[t ||D]J}t j }|j d}xt rt }xm| D]e}|j||dk rN|jn|jdt |j| rd| jkrt|j|j| |j qt|j|j| |j n4t|j|jd|dt|d | d|j |dk r|jn|j||j|d7}|dk retd|d|d|dt}t||n| dk r"|| kr"t}Pq"q"W|r|jd|| j n| dks|| krPqqWt j } |jd|| ||s|d|kr|j ||nt||||||r|j |j d}!xt!|D]\}"}#|j|#|jdt|j||#j|dk rtd|d|"d|dt}t||n|!d7}!q]W|dk r'td|d|!d|dt}$t||$n|j qqWdS(sInternal training function on multiple devices. This function will also work for single device as well. Parameters ---------- symbol : Symbol The network configuration. ctx : list of Context The training devices. arg_names: list of str Name of all arguments of the network. param_names: list of str Name of all trainable parameters of the network. aux_names: list of str Name of all auxiliary states of the network. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. begin_epoch : int The begining training epoch. end_epoch : int The end training epoch. epoch_size : int, optional Number of batches in a epoch. In default, it is set to ``ceil(num_train_examples / batch_size)``. optimizer : Optimizer The optimization algorithm train_data : DataIter Training data iterator. eval_data : DataIter Validation data iterator. eval_metric : EvalMetric An evaluation function or a list of evaluation functions. epoch_end_callback : callable(epoch, symbol, arg_params, aux_states) A callback that is invoked at end of each epoch. This can be used to checkpoint model each epoch. batch_end_callback : callable(BatchEndParams) A callback that is invoked at end of each batch. This can be used to measure speed, get result from evaluation metric. etc. kvstore : KVStore The KVStore. update_on_kvstore : bool Whether or not perform weight updating on kvstore. logger : logging logger When not specified, default logger will be used. work_load_list : list of float or int, optional The list of work load for different devices, in the same order as ``ctx``. monitor : Monitor, optional Monitor installed to executor, for monitoring outputs, weights, and gradients for debugging. Notes ----- - This function will inplace update the NDArrays in `arg_params` and `aux_states`. Rtsym_gentctxt train_dataR6t arg_namest aux_namestwork_load_listtloggerRR5R.Ritis_traintncclRTR-iRRRRs!Epoch[%d] Resetting Data IteratorsEpoch[%d] Time cost=%.3fN("R)tloggingR tinstall_monitort set_paramsR t set_optimizerR:R5R6tresettrangettimeRtload_data_batchttictforwardtbackwardttypeRORBRSRYR=t toc_printt update_metrictlabelt BatchEndParamRR_R,tinfotcopy_toR2(%RRaRcR6RdR.t aux_paramst begin_epocht end_epocht epoch_sizeRRRRbt eval_dataRtepoch_end_callbacktbatch_end_callbackRfRetmonitorteval_end_callbackteval_batch_end_callbackR`texecutor_managerRTRRqRtdo_resett data_batchtbatch_end_paramsttocttotal_num_batchRFt eval_batchteval_end_params((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_train_multi_devicesA                                         cC@s|dk r |jd|nd|jD}|jd|jDd||f}tj||tjd|dS(sBCheckpoint the model data into file. Parameters ---------- prefix : str Prefix of model name. epoch : int The epoch number of the model. symbol : Symbol The input Symbol. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. Notes ----- - ``prefix-symbol.json`` will be saved for symbol. - ``prefix-epoch.params`` will be saved for parameters. s%s-symbol.jsoncS@s/i|]%\}}|jtd|qS(sarg:%s(t as_in_contextR (R'RUtv((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys s cS@s/i|]%\}}|jtd|qS(saux:%s(RR (R'RUR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys s s%s-%04d.paramssSaved checkpoint to "%s"N(R)tsavetitemstupdatetndRiRy(tprefixRRR.R{t save_dictt param_name((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pytsave_checkpoints c C@stjd|}tjd||f}i}i}xd|jD]V\}}|jdd\}} |dkr||| %s c@s+i|]!\}}|kr||qS(((R'RUR(Rd(sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys )s ( RRR)RRRRR.tsettlist_argumentsRR{tlist_auxiliary_states(R((RcRdsK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyRs     % cC@s|jdp|jdS(s!Check if name is a data argument.tdataRw(tendswith(R9((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt _is_data_arg-scC@spg|D]'}t|tr"|n t|^q}d|D}|jj|\}}}|dk sqtd|D}|jj|\} }} | dk st|jj} |j} g| D]} | | kr| ^q}|jj }gt | || D]}|d|kr|^q}d|D}gt ||| D]}|d|krU|^qU}d|D}x_|j D]Q\}}|j r||j kr| r|j |||(q|j ||qWx_|j D]Q\}}|jr7||jkr7| r7|j|||(q|j ||qW||_ ||_| t||fS(s2Initialize weight parameters and auxiliary states.cS@si|]}|j|jqS((R&R9(R'titem((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys 5s cS@si|]}|j|jqS((tdtypeR9(R'R((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys 8s icS@s4i|]*\}}}tjd|d||qS(R&R(Rtzeros(R'RUtstt((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys Cs cS@s4i|]*\}}}tjd|d||qS(R&R(RR(R'RURR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys Gs N(RRRt infer_shapeR)Rt infer_typeRtkeysRRPRR.RR{RZ(Rtinputst overwritetxt input_shapest arg_shapest_t aux_shapest input_dtypest arg_dtypest aux_dtypesRct input_namestkeyR6Rdtparam_name_attrsR.taux_name_attrsR{RUR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt _init_params2s<4 %    cC@s|jj}d|d<|S(NR(t__dict__RR)(Rtthis((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt __getstate__Zs cC@s|jj|dS(N(RR(Rtstate((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt __setstate___sc @sfdjD}|jt|jdk rjj|\}}}|dk sntdgjjD]}|j ^q{}||krdSnjj j dddd||}|j jj tj|_dS(s7Initialize the predictor module for running prediction.c@s#i|]}j|j|qS((R.R&(R'R9(R(sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys ds sIncomplete input shapesNitgrad_reqtnullt type_dict(R.RtdictRR)RRRt arg_arraysR&t simple_bindRatcopy_params_fromR{R( RRRtshapesRRRt pred_shapest pred_exec((RsK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_init_predictorbs"   cC@st|tjtjfr^|d krU|r<tdqUtj|jd}nt|tjtjfst dn|jd|jdkrtdn|j dkr|jddkr|j }n|j dkrtdn|r/t j ||t|jd|jd|d d St j ||t|jd|jdtSnt|t jst d n|S( s$Initialize the iterator given input.s+y must be specified when X is numpy.ndarrayis)y must be ndarray when X is numpy.ndarrays/The numbers of data points and labels not equaliis3Label must be 1D or 2D (with 2nd dimension being 1)tshuffletlast_batch_handlet roll_overs,X must be DataIter, NDArray or numpy.ndarrayN(RR$RRtNDArrayR)t ValueErrorRR&RtndimtflattenRt NDArrayItertminRR,tDataIter(RtXtyRg((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt _init_iterts( "% /cC@s|dkr|St|ttfrt|dkr|ddk r|ddkrut|dtjru|dSt|dtrtj|dn|d}t|dtrtj|dn|d}|j ||dt St dnt|tjst dn|S(s(Initialize the iterator given eval_data.iiiRgsEval data is NONEs\Eval data must be DataIter, or NDArray/numpy.ndarray/list pair (i.e. tuple/list of length 2)N( R)RttupleRZR=RRR$tarrayRRRR(RRt input_datat input_label((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pyt_init_eval_iters '&& & cC@so|j|ddt}|r+|jn|j}g|D]}|d^q;}td|jjD}xA|jD]6}t|t r|j ||j t|jD]*\}}||j|d|!jq6Wn|d7}|dk ri||kriPqiqiWg| D]}tj|^q}t|dkr|d}n|rgg|D]}tj|^q}g|D]}tj|^q}t|dkr;|d}nt|dkrZ|d}n|||fS|SdS(sRun the prediction, always only use one device. Parameters ---------- X : mxnet.DataIter num_batch : int or None The number of batch to run. Go though all batches if ``None``. Returns ------- y : numpy.ndarray or a list of numpy.ndarray if the network has multiple outputs. The predicted value of the output. Rgics@s$|]\}}||jfVqdS(N(R(R'Rtvalue((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pys siN( RR)R,Rmt provide_dataRR.RRRRR9RRt batch_sizeRtarg_dictRnR=toutputst provide_labelRRrtpadRPtappendtasnumpyR2RRwR$t concatenate(RRt num_batcht return_dataRmt data_shapesRt data_namesRRR9t data_arraysRt output_listt data_listt label_listRFRMtpaddedt real_sizeto_listto_ndtjRRRw((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pytpredictsZ    #+    ""% " ""   taccc C@st|tjs$tj|}n|j|d dt}|rO|jn|j}g|D]}|d^q_}t d|j j D} xA|jD]6}t|t r|j | |jsRRRRiN(RRt EvalMetricRRR)R,RmRRR.RRRR9RRRRR2RRrRRwRRxRR_tget(RRRRRRmRRRRR9RRFRMR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/model.pytscores6  #    R#c *C@s|j||dt} |j|}|jrR|j| j|_|jn|j|jd<|j| j | j \}}}t |t j st j|}nt|t|j|j\}}i}|r|jt|nXxUt|D]G\}}x8tt|jD]!}|||t|j|s\           '      $