ó ùµÈ[c@sšdZddlZddlmZmZdefd„ƒYZdefd„ƒYZdefd „ƒYZd efd „ƒYZ d efd „ƒYZ dS(sScheduling learning rate.iÿÿÿÿN(tcostpit LRSchedulercBs5eZdZddddd„Zd„Zd„ZRS(sBase class of a learning rate scheduler. A scheduler returns a new learning rate based on the number of updates that have been performed. Parameters ---------- base_lr : float, optional The initial learning rate. warmup_steps: int number of warmup steps used before this scheduler starts decay warmup_begin_lr: float if using warmup, the learning rate from which it starts warming up warmup_mode: string warmup can be done in two modes. 'linear' mode gradually increases lr with each step in equal increments 'constant' mode keeps lr at warmup_begin_lr for warmup_steps g{®Gáz„?itlinearcCs ||_t|tƒst‚||_||_||_|j|jkrZtdƒ‚n|jdkrxtdƒ‚n|dkr“tdƒ‚n||_dS(Ns-Base lr has to be higher than warmup_begin_lris$Warmup steps has to be positive or 0Rtconstants1Supports only linear and constant modes of warmup(slinearsconstant( tbase_lrt isinstancetinttAssertionErrort warmup_stepstwarmup_final_lrtwarmup_begin_lrt ValueErrort warmup_mode(tselfRR R R ((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyt__init__)s     cCsƒ||jkst‚|jdkrV|j|jt|ƒt|jƒ}|j|S|jdkrl|jStd|jƒ‚dS(NRRsInvalid warmup mode %s(R RR R R tfloatR (Rt num_updatetincrease((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyt get_warmup_lr9s' cCstdƒ‚dS(sÔReturn a new learning rate. The ``num_update`` is the upper bound of the number of updates applied to every weight. Assume the optimizer has updated *i*-th weight by *k_i* times, namely ``optimizer.update(i, weight_i)`` is called by *k_i* times. Then:: num_update = max([k_i for all i]) Parameters ---------- num_update: int the maximal number of updates applied to a weight. smust override thisN(tNotImplementedError(RR((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyt__call__Ds(t__name__t __module__t__doc__RRR(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRs  tFactorSchedulercBs2eZdZddddddd„Zd„ZRS(s¼Reduce the learning rate by a factor for every *n* steps. It returns a new learning rate by:: base_lr * pow(factor, floor(num_update/step)) Parameters ---------- step : int Changes the learning rate for every n updates. factor : float, optional The factor to change the learning rate. stop_factor_lr : float, optional Stop updating the learning rate if it is less than this value. ig:Œ0âŽyE>g{®Gáz„?iRcCs}tt|ƒj||||ƒ|dkr:tdƒ‚n|dkrUtdƒ‚n||_||_||_d|_dS(Nis3Schedule step must be greater or equal than 1 roundgð?s/Factor must be no more than 1 to make lr reducei(tsuperRRR tsteptfactortstop_factor_lrtcount(RRRRRR R R ((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRfs     cCs±||jkr|j|ƒSx‹||j|jkr©|j|j7_|j|j9_|j|jkr|j|_tjd||jƒqtjd||jƒqW|jS(NsMUpdate[%d]: now learning rate arrived at %0.5e, will not change in the futures)Update[%d]: Change learning rate to %0.5e( R RRRRRRtloggingtinfo(RR((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRrs    (RRRRR(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRVs  tMultiFactorSchedulercBs/eZdZdddddd„Zd„ZRS(s*Reduce the learning rate by given a list of steps. Assume there exists *k* such that:: step[k] <= num_update and num_update < step[k+1] Then calculate the new learning rate by:: base_lr * pow(factor, k+1) Parameters ---------- step: list of int The list of steps to schedule a change factor: float The factor to change the learning rate. warmup_steps: int number of warmup steps used before this scheduler starts decay warmup_begin_lr: float if using warmup, the learning rate from which it starts warming up warmup_mode: string warmup can be done in two modes. 'linear' mode gradually increases lr with each step in equal increments 'constant' mode keeps lr at warmup_begin_lr for warmup_steps ig{®Gáz„?iRc Csôtt|ƒj||||ƒt|tƒr@t|ƒdksFt‚xht|ƒD]Z\}}|dkr’||||dkr’tdƒ‚n|dkrStdƒ‚qSqSW|dkrÌtdƒ‚n||_ d|_ ||_ d|_ dS(Niis0Schedule step must be an increasing integer lists3Schedule step must be greater or equal than 1 roundgð?s/Factor must be no more than 1 to make lr reduce( RR!RRtlisttlenRt enumerateR Rt cur_step_indRR( RRRRR R R tit_step((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRs '$     cCs°||jkr|j|ƒSxŠ|jt|jƒdkr¨||j|jkrž|j|j|_|jd7_|j|j9_tj d||jƒq|jSqW|jS(Nis)Update[%d]: Change learning rate to %0.5e( R RR%R#RRRRRR (RR((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyR®s   (RRRRR(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyR!ƒs  t PolySchedulercBs2eZdZddddddd„Zd„ZRS(s& Reduce the learning rate according to a polynomial of given power. Calculate the new learning rate, after warmup if any, by:: final_lr + (start_lr - final_lr) * (1-nup/max_nup)^pwr if nup < max_nup, 0 otherwise. Parameters ---------- max_update: int maximum number of updates before the decay reaches final learning rate. base_lr: float base learning rate to start from pwr: int power of the decay term as a function of the current number of updates. final_lr: float final learning rate after all steps warmup_steps: int number of warmup steps used before this scheduler starts decay warmup_begin_lr: float if using warmup, the learning rate from which it starts warming up warmup_mode: string warmup can be done in two modes. 'linear' mode gradually increases lr with each step in equal increments 'constant' mode keeps lr at warmup_begin_lr for warmup_steps g{®Gáz„?iiRcCstt|ƒj||||ƒt|tƒs4t‚|dkrOtdƒ‚n||_|j|_ ||_ ||_ |j |j |_ dS(Nis3maximum number of updates must be strictly positive(RR(RRRRR tpowerRt base_lr_origt max_updatetfinal_lrR t max_steps(RR+RtpwrR,R R R ((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRÚs     cCs}||jkr|j|ƒS||jkrv|j|j|jtdt||jƒt|jƒ|jƒ|_ n|j S(Ni( R RR+R,R*tpowRR-R)R(RR((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRæs  8(RRRRR(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyR(¾s  tCosineSchedulercBs/eZdZdddddd„Zd„ZRS(sƒ Reduce the learning rate according to a cosine function Calculate the new learning rate by:: final_lr + (start_lr - final_lr) * (1+cos(pi * nup/max_nup))/2 if nup < max_nup, 0 otherwise. Parameters ---------- max_update: int maximum number of updates before the decay reaches 0 base_lr: float base learning rate final_lr: float final learning rate after all steps warmup_steps: int number of warmup steps used before this scheduler starts decay warmup_begin_lr: float if using warmup, the learning rate from which it starts warming up warmup_mode: string warmup can be done in two modes. 'linear' mode gradually increases lr with each step in equal increments 'constant' mode keeps lr at warmup_begin_lr for warmup_steps g{®Gáz„?iRcCstt|ƒj||||ƒt|tƒs4t‚|dkrOtdƒ‚n||_||_||_ |j|j |_ dS(Nis3maximum number of updates must be strictly positive( RR0RRRRR R*R+R,R R-(RR+RR,R R R ((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRs    cCss||jkr|j|ƒS||jkrl|j|j|jdtt||j|jƒd|_n|jS(Nii( R RR+R,R*RRR-R(RR((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyRs  ;(RRRRR(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyR0îs ( RRtmathRRtobjectRRR!R(R0(((sR/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/lr_scheduler.pyts @-;0