σ R;]c@s_dZdgZddlmZddlmZddlmZm Z de fd„ƒYZ dS( sParameter optimizer.tTraineri(t optimizer(t_create_kvstorei(t ParameterDictt ParametercBsqeZdZd dd d„Zd„Zd„Zd„Zed„ƒZ d„Z e d„Z d „Z d „ZRS( sζApplies an `Optimizer` on a set of Parameters. Trainer should be used together with `autograd`. Parameters ---------- params : ParameterDict The set of parameters to optimize. optimizer : str or Optimizer The optimizer to use. See `help `_ on Optimizer for a list of available optimizers. optimizer_params : dict Key-word arguments to be passed to optimizer constructor. For example, `{'learning_rate': 0.1}`. All optimizers accept learning_rate, wd (weight decay), clip_gradient, and lr_scheduler. See each optimizer's constructor for a list of additional supported arguments. kvstore : str or KVStore kvstore type for multi-gpu and distributed training. See help on :any:`mxnet.kvstore.create` for more information. compression_params : dict Specifies type of gradient compression and additional arguments depending on the type of compression being used. For example, 2bit compression requires a threshold. Arguments would then be {'type':'2bit', 'threshold':0.5} See mxnet.KVStore.set_gradient_compression method for more details on gradient compression. Properties ---------- learning_rate : float The current learning rate of the optimizer. Given an Optimizer object optimizer, its learning rate can be accessed as optimizer.learning_rate. tdevicecCst|ttfƒr*t|jƒƒ}nt|ttfƒsXtdt|ƒƒ‚ng|_xF|D]>}t|t ƒs–tdt|ƒƒ‚n|jj |ƒqhW||_ |rΏ|ni}|j ddƒ|_ |jƒ|_|j||ƒt|_||_dS(Ns<First argument must be a list or dict of Parameters, got %s.sDFirst argument must be a list or dict of Parameters, got list of %s.t rescale_gradgπ?(t isinstancetdictRtlisttvaluesttuplet ValueErrorttypet_paramsRtappendt_compression_paramstgett_scalet_check_contextst _contextst_init_optimizertFalset_kv_initializedt_kvstore(tselftparamsRtoptimizer_paramstkvstoretcompression_paramstparam((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyt__init__;s(    cCspd}xc|jD]X}|jƒ}|dksb||ksbtd|jt|ƒt|ƒfƒ‚|}qW|S(Ns–All Parameters must be initialized on the same set of contexts, but Parameter %s is initialized on %s while previous Parameters are initialized on %s.(tNoneRtlist_ctxtAssertionErrortnametstr(RtcontextsRtctx((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyRRs " cCs d„t|jƒDƒ}t|tjƒrV| s>tdƒ‚||_||j_ntj|d|||_g|j D]}tj |jƒ^q{|_ dS(NcSsi|]\}}||“qS(((t.0tiR((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pys ^s sUoptimizer_params must be None if optimizer is an instance of Optimizer instead of strt param_dict( t enumerateRRtoptt OptimizerR"t _optimizerR)tcreateRt get_updatert _updaters(RRRR)t_((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyR]s    cs‡fd†ˆjDƒ}tˆjtˆjƒ|ƒ\}}|rόˆjr_|jˆjƒnd|jkrwt}n|r|j ˆj ƒnxTt ˆjƒD]C\}}|j ƒ}|j ||dƒ|j||d| ƒq W|ˆ_|ˆ_ndˆ_dˆ_tˆ_dS(Ncs,i|]"}|jˆjdƒ|j“qS(i(tdataRR#(R'R(R(sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pys ms tdistitpriority(RRRtlenRRtset_gradient_compressionR Rt set_optimizerR-R*t list_datatinittpullt_update_on_kvstoreR tTrueR(Rt arg_arraysRtupdate_on_kvstoreR(Rt param_arrays((RsS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyt _init_kvstorels&       cCs2t|jtjƒs$tdƒ‚n |jjSdS(NsEOptimizer has to be defined before its learning rate can be accessed.(RR-R+R,t UserWarningt learning_rate(R((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyRB…scCs8t|jtjƒs$tdƒ‚n|jj|ƒdS(s‘Sets a new learning rate of the optimizer. Parameters ---------- lr : float The new learning rate of the optimizer. s@Optimizer has to be defined before its learning rate is mutated.N(RR-R+R,RAtset_learning_rate(Rtlr((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyRCŽsc CsŒ|js|jƒn|j||j_x\t|jƒD]K\}}|jdkrZq9n|s¨xE|jƒD]4}|j smt d|j t |j ƒfƒ‚qmqmWn|jr#|jj||jƒd| ƒ|jr|jj||jƒd| ƒq9q#|jj||jƒd| ƒnx^t|j|jƒ|jƒƒD];\}}}| sd|j rE||||ƒt|_ qEqEWq9WdS(sKMakes one step of parameter update. Should be called after `autograd.compute_gradient` and outside of `record()` scope. Parameters ---------- batch_size : int Batch size of data processed. Gradient will be normalized by `1/batch_size`. Set this to 1 if you normalized loss manually with `loss = mean(loss)`. ignore_stale_grad : bool, optional, default=False If true, ignores Parameters with stale gradient (gradient that has not been updated by `backward` after last step) and skip update. tnullsoGradient of Parameter `%s` on context %s has not been updated by backward since last `step`. This could mean a bug in your model that made it only use a subset of the Parameters (Blocks) for this iteration. If you are intentionally only using a subset, call step with ignore_stale_grad=True to suppress this warning and skip updating of Parameters with stale gradientR4N(RR@RR-RR*Rtgrad_reqR8t _fresh_gradRAR#R$tcontextRtpusht list_gradR;R:tzipR0R( Rt batch_sizetignore_stale_gradR(RR2tupdtarrtgrad((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pytsteps,   &    #1cCss|jdk st‚|jr7|jj|dtƒn8t|dƒ'}|j|j dj dtƒƒWdQXdS(s¦Saves trainer states (e.g. optimizer, momentum) to a file. Parameters ---------- fname : str Path to output states file. tdump_optimizertwbiN( R-R R"R;Rtsave_optimizer_statesR<topentwriteR0t get_states(Rtfnametfout((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyt save_statesΛs  cCs³|js|jƒn|jrD|jj|ƒ|jjj|_nkt|dƒ}|j ƒ}WdQXx1|j D]&}|j |ƒ|j dj|_qrW|j dj|_dS(s§Loads trainer states (e.g. optimizer, momentum) from a file. Parameters ---------- fname : str Path to input states file. trbNi( RR@R;Rtload_optimizer_statest_updaterRR-RUtreadR0t set_states(RRXtftstatestupdater((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyt load_statesΫs    N(t__name__t __module__t__doc__R RRRR@tpropertyRBRCRRQRZRc(((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyRs     . N( Rft__all__tRR+tmodelRt parameterRRtobjectR(((sS/usr/local/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet/gluon/trainer.pyts