R;]c @@s.dZddlmZmZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddl mZddl mZdd l mZdd l mZdd l mZdd lmZmZdd lmZddlmZddlmZmZm Z ddl m!Z!ddl"m#Z#e$Z%yddl&m'Z'e'Z%Wne(k rse)Z*nXe dddddgZ+dZ,dZ-dZ.dZ/dddZ1dZ2ddddddddddd Z3dZ4d Z5dd!l6m7Z7d"e%fd#YZ8dS($sMXNet model modulei(tabsolute_importtprint_functionN(t namedtuplei(tio(tndarray(tsymbol(t optimizer(tmetric(tkvstore(tContexttcpu(tUniform(t get_updater(tDataParallelExecutorManagert_check_argumentst _load_data(tDataDesc(t mx_real_t(t BaseEstimatortBatchEndParamstepochtnbatcht eval_metrictlocalscC@st}|dkrd}nt|tjr6|}nt|tr|dkrfd|krfd}qtj|}|dkrtd|jD}|d krt }qqn t d|dkrt }n||fS( sWCreate kvstore This function select and create a proper kvstore if given the kvstore type. Parameters ---------- kvstore : KVStore or str The kvstore. num_device : int The number of devices arg_params : dict of str to `NDArray`. Model parameter, dict of name to `NDArray` of net's weights. itdisttlocalcs@s!|]}tj|jVqdS(N(tnptprodtshape(t.0tparam((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys Usiis$kvstore must be KVStore, str or NoneNii( tTruetNonet isinstancetkvstKVStoretstrtcreatetmaxtvaluestFalset TypeError(Rt num_devicet arg_paramstupdate_on_kvstoretkvtmax_size((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_create_kvstore:s$           cC@s_xXt|D]J\}}||}|j||||r |j||d| q q WdS(sInitialize kvstoretpriorityN(t enumeratetinittpull(Rt param_arraysR+t param_namesR,tidxt param_on_devstname((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_initialize_kvstoreas  cC@sCgt|D]"\}}|ddk r |^q }g|D]}||^q<}g|D]}||^qY} g|D]}||^qv} t|} d} d} ttjd| }x| | kr>| || kr| |n| }|j| | |!|| |!d| |j| | |!| | |!d| |} qWdS(s@Perform update of param_arrays from grad_arrays on NCCL kvstore.iitMXNET_UPDATE_AGGREGATION_SIZER0N(R1R tlentinttostgetenvtpushR3(R4t grad_arraysRR5tindext grad_listt valid_indicestitvalid_grad_arraystvalid_param_arraystvalid_param_namestsizetstartt default_batchtbatchtend((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_update_params_on_kvstore_nccljs2  %%c C@sx}tt||D]f\}}|\}}|ddkrDqn||}|j||d| |j||d| qWdS(s;Perform update of param_arrays from grad_arrays on kvstore.iR0N(R1tzipR R?R3( R4R@RR5RAtpairtarg_listRBR8((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_update_params_on_kvstore~s"  cC@sxtt||D]\}}|\}} | ddkrDqn|} |r|| } |j| | d| |j| | d| nxGtt|| D]0\} } | \}}|| || ||qWqWdS(s?Perform update of param_arrays from grad_arrays not on kvstore.iR0N(R1RNR R?R3(R4R@tupdaterR*RR5RDRORPRBRAR8tktptwtg((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_update_paramss"  " cO@sKt|tr1x|D]}|||qWdS|rG|||ndS(sSends args and kwargs to any configured callbacks. This handles the cases where the 'callbacks' variable is ``None``, a single function, or a list. N(R!tlist(t callbackstargstkwargstcb((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_multiple_callbackss  c%C@s<|dkrt}ntd|d|d|d| d|d|d|d|d | }|rj|j|n|j||| st| }n | j| | rtd | d |jd |d|j d | n| j x[t ||D]J}t j }|j d}xt rt }xm| D]e}|j||dk rN|jn|jdt |j| rd| jkrt|j|j| |j qt|j|j| |j n4t|j|jd|dt|d | d|j |dk r|jn|j||j|d7}|dk retd|d|d|dt}t||n| dk r"|| kr"t}Pq"q"W|r|jd|| j n| dks|| krPqqWt j } |jd|| ||s|d|kr|j ||nt||||||r|j |j d}!xt!|D]\}"}#|j|#|jdt|j||#j|dk rtd|d|"d|dt}t||n|!d7}!q]W|dk r'td|d|!d|dt}$t||$n|j qqWdS(sInternal training function on multiple devices. This function will also work for single device as well. Parameters ---------- symbol : Symbol The network configuration. ctx : list of Context The training devices. arg_names: list of str Name of all arguments of the network. param_names: list of str Name of all trainable parameters of the network. aux_names: list of str Name of all auxiliary states of the network. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. begin_epoch : int The begining training epoch. end_epoch : int The end training epoch. epoch_size : int, optional Number of batches in a epoch. In default, it is set to ``ceil(num_train_examples / batch_size)``. optimizer : Optimizer The optimization algorithm train_data : DataIter Training data iterator. eval_data : DataIter Validation data iterator. eval_metric : EvalMetric An evaluation function or a list of evaluation functions. epoch_end_callback : callable(epoch, symbol, arg_params, aux_states) A callback that is invoked at end of each epoch. This can be used to checkpoint model each epoch. batch_end_callback : callable(BatchEndParams) A callback that is invoked at end of each batch. This can be used to measure speed, get result from evaluation metric. etc. kvstore : KVStore The KVStore. update_on_kvstore : bool Whether or not perform weight updating on kvstore. logger : logging logger When not specified, default logger will be used. work_load_list : list of float or int, optional The list of work load for different devices, in the same order as ``ctx``. monitor : Monitor, optional Monitor installed to executor, for monitoring outputs, weights, and gradients for debugging. Notes ----- - This function will inplace update the NDArrays in `arg_params` and `aux_states`. Rtsym_gentctxt train_dataR5t arg_namest aux_namestwork_load_listtloggerRR4R+R,itis_traintncclRRR*iRRRRs!Epoch[%d] Resetting Data IteratorsEpoch[%d] Time cost=%.3fN("R tloggingR tinstall_monitort set_paramsR t set_optimizerR9R4R5tresettrangettimeRtload_data_batchttictforwardtbackwardttypeRMR@RQRWR;t toc_printt update_metrictlabelt BatchEndParamRR]R(tinfotcopy_toR1(%RR_RaR5RbR+t aux_paramst begin_epocht end_epocht epoch_sizeRRR,R`t eval_dataRtepoch_end_callbacktbatch_end_callbackRdRctmonitorteval_end_callbackteval_batch_end_callbackR^texecutor_managerRRRRoRtdo_resett data_batchtbatch_end_paramsttocttotal_num_batchRDt eval_batchteval_end_params((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_train_multi_devicesA                                         cC@s|dk r |jd|nd|jD}|jd|jDd||f}tj||tjd|dS(sBCheckpoint the model data into file. Parameters ---------- prefix : str Prefix of model name. epoch : int The epoch number of the model. symbol : Symbol The input Symbol. arg_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's weights. aux_params : dict of str to NDArray Model parameter, dict of name to NDArray of net's auxiliary states. Notes ----- - ``prefix-symbol.json`` will be saved for symbol. - ``prefix-epoch.params`` will be saved for parameters. s%s-symbol.jsoncS@s/i|]%\}}|jtd|qS(sarg:%s(t as_in_contextR (RRStv((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys s cS@s/i|]%\}}|jtd|qS(saux:%s(RR (RRSR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys s s%s-%04d.paramssSaved checkpoint to "%s"N(R tsavetitemstupdatetndRgRw(tprefixRRR+Ryt save_dictt param_name((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pytsave_checkpointms c C@stjd|}tjd||f}i}i}xd|jD]V\}}|jdd\}} |dkr||| s c@s+i|]!\}}|kr||qS(((RRSR(Rb(sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys s ( RRR RRRRR+tsettlist_argumentsRRytlist_auxiliary_states(R((RaRbsK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyRs     % cC@s|jdp|jdS(s!Check if name is a data argument.tdataRu(tendswith(R8((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt _is_data_argscC@spg|D]'}t|tr"|n t|^q}d|D}|jj|\}}}|dk sqtd|D}|jj|\} }} | dk st|jj} |j} g| D]} | | kr| ^q}|jj }gt | || D]}|d|kr|^q}d|D}gt ||| D]}|d|krU|^qU}d|D}x_|j D]Q\}}|j r||j kr| r|j |||(q|j ||qWx_|j D]Q\}}|jr7||jkr7| r7|j|||(q|j ||qW||_ ||_| t||fS(s2Initialize weight parameters and auxiliary states.cS@si|]}|j|jqS((RR8(Rtitem((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys #s cS@si|]}|j|jqS((tdtypeR8(RR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys &s icS@s4i|]*\}}}tjd|d||qS(RR(Rtzeros(RRStstt((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys 1s cS@s4i|]*\}}}tjd|d||qS(RR(RR(RRSRR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys 5s N(R!RRt infer_shapeR Rt infer_typeRtkeysRRNRR+RRyRX(Rtinputst overwritetxt input_shapest arg_shapest_t aux_shapest input_dtypest arg_dtypest aux_dtypesRat input_namestkeyR5Rbtparam_name_attrsR+taux_name_attrsRyRSR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt _init_params s<4 %    cC@s|jj}d|d<|S(NR(t__dict__RR (Rtthis((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt __getstate__Hs cC@s|jj|dS(N(RR(Rtstate((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt __setstate__Msc @sfdjD}|jt|jdk rjj|\}}}|dk sntdgjjD]}|j ^q{}||krdSnjj j dddd||}|j jj tj|_dS(s7Initialize the predictor module for running prediction.c@s#i|]}j|j|qS((R+R(RR8(R(sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys Rs sIncomplete input shapesNitgrad_reqtnullt type_dict(R+RtdictRR RRRt arg_arraysRt simple_bindR_tcopy_params_fromRyR( RRRtshapesRRRt pred_shapest pred_exec((RsK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_init_predictorPs"   cC@st|tjtjfr^|d krU|r<tdqUtj|jd}nt|tjtjfst dn|jd|jdkrtdn|j dkr|jddkr|j }n|j dkrtdn|r/t j ||t|jd|jd|d d St j ||t|jd|jdtSnt|t jst d n|S( s$Initialize the iterator given input.s+y must be specified when X is numpy.ndarrayis)y must be ndarray when X is numpy.ndarrays/The numbers of data points and labels not equaliis3Label must be 1D or 2D (with 2nd dimension being 1)tshuffletlast_batch_handlet roll_overs,X must be DataIter, NDArray or numpy.ndarrayN(R!RRRtNDArrayR t ValueErrorRRR)tndimtflattenRt NDArrayItertminRR(tDataIter(RtXtyRe((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt _init_iterbs( "% /cC@s|dkr|St|ttfrt|dkr|ddk r|ddkrut|dtjru|dSt|dtrtj|dn|d}t|dtrtj|dn|d}|j ||dt St dnt|tjst dn|S(s(Initialize the iterator given eval_data.iiiResEval data is NONEs\Eval data must be DataIter, or NDArray/numpy.ndarray/list pair (i.e. tuple/list of length 2)N( R R!ttupleRXR;RRRtarrayRRRR)(RR}t input_datat input_label((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pyt_init_eval_iter{s '&& & cC@so|j|ddt}|r+|jn|j}g|D]}|d^q;}td|jjD}xA|jD]6}t|t r|j ||j t|jD]*\}}||j|d|!jq6Wn|d7}|dk ri||kriPqiqiWg| D]}tj|^q}t|dkr|d}n|rgg|D]}tj|^q}g|D]}tj|^q}t|dkr;|d}nt|dkrZ|d}n|||fS|SdS(sRun the prediction, always only use one device. Parameters ---------- X : mxnet.DataIter num_batch : int or None The number of batch to run. Go though all batches if ``None``. Returns ------- y : numpy.ndarray or a list of numpy.ndarray if the network has multiple outputs. The predicted value of the output. Reics@s$|]\}}||jfVqdS(N(R(RRtvalue((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pys siN( RR R(Rkt provide_dataRR+RR!RRR8RRt batch_sizeRtarg_dictRlR;toutputst provide_labelRRptpadRNtappendtasnumpyR1RRuRt concatenate(RRt num_batcht return_dataRkt data_shapesRt data_namesRRR8t data_arraysRt output_listt data_listt label_listRDRKtpaddedt real_sizeto_listto_ndtjRRRu((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pytpredictsZ    #+    ""% " ""   taccc C@st|tjs$tj|}n|j|d dt}|rO|jn|j}g|D]}|d^q_}t d|j j D} xA|jD]6}t|t r|j | |jsRRRRiN(R!Rt EvalMetricR%RR R(RkRRR+RRRR8RRRRR1RRpRRuRRvRR]tget(RRRRRRkRRRRR8RRDRKR((sK/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/model.pytscores6  #    Rc *C@s|j||dt} |j|}|jrR|j| j|_|jn|j|jd<|j| j | j \}}}t |t j st j|}nt|t|j|j\}}i}|r|jt|nXxUt|D]G\}}x8tt|jD]!}|||t|j|sZ          '      $