ó ùµÈ[c@@s¥dZddlmZmZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddlmZmZddlmZdd lmZdd lmZd efd „ƒYZd efd„ƒYZejedƒZejedƒZejedƒZd„Zdefd„ƒYZdefd„ƒYZ eedƒdefd„ƒYƒƒZ!eedƒdefd„ƒYƒƒZ"edefd„ƒYƒZ#edefd„ƒYƒZ$edefd „ƒYƒZ%ed!efd"„ƒYƒZ&ed#efd$„ƒYƒZ'ed%e'fd&„ƒYƒZ(ed'efd(„ƒYƒZ)ed)efd*„ƒYƒZ*ed+efd,„ƒYƒZ+dS(-sWeight initializer.i(tabsolute_importtprint_functionN(tsqrti(t string_types(tNDArraytload(trandom(tregistry(tndarraytInitDesccB@seZdZddd„ZRS(s"Descriptor for the initialization pattern. Parameter --------- name : str Name of variable. attrs : dict of str to str Attributes of this variable taken from ``Symbol.attr_dict``. global_init : Initializer Global initializer to fallback to. cC@s7tt|ƒj||ƒ}|p$i|_||_|S(N(tsuperR t__new__tattrst global_init(tclstnameR R tret((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR .s N(t__name__t __module__t__doc__tNoneR (((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR "s t InitializercB@s›eZdZd„Zedd„Zd„Zd„Zd„Z d„Z d„Z d„Z d „Z d „Zd „Zd „Zd „Zd„Zd„ZRS(s!The base class of an initializer.cK@s||_t|_d|_dS(N(t_kwargstFalset_verboseRt _print_func(tselftkwargs((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyt__init__7s  cC@s4||_|dkr'd„}|}n||_|S(smSwitch on/off verbose mode Parameters ---------- verbose : bool switch on/off verbose mode print_func : function A function that computes statistics of initialized arrays. Takes an `NDArray` and returns an `str`. Defaults to mean absolute value str((|x|/size(x)).asscalar()). cS@s&ttj|ƒt|jƒjƒƒS(s%returns |x|/size(x), async execution.(tstrRtnormRtsizetasscalar(tx((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyt asum_statJsN(RRR(Rtverboset print_funcR"((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyt set_verbosity<s     cC@s8|jr4|jr4tjd|||j|ƒƒndS(sóInternal verbose print function Parameters ---------- desc : InitDesc or str name of the array init : str initializer pattern arr : NDArray initialized array sInitialized %s as %s: %sN(RRtloggingtinfo(Rtdesctinittarr((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyt_verbose_printQs cC@s"tj|jjjƒ|jgƒS(sSaves the initializer to string Returns ------- str JSON formatted string that describes the initializer. Examples -------- >>> # Create initializer and retrieve its parameters ... >>> init = mx.init.Normal(0.5) >>> init.dumps() '["normal", {"sigma": 0.5}]' >>> init = mx.init.Xavier(factor_type="in", magnitude=2.34) >>> init.dumps() '["xavier", {"rnd_type": "uniform", "magnitude": 2.34, "factor_type": "in"}]' (tjsontdumpst __class__RtlowerR(R((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR-`scC@smt|tƒs#|j||ƒdS|jdkr>||_n|jjddƒ}|r…t|ƒj||ƒ|j |||ƒnä|j dƒrº|j||ƒ|j |d|ƒn¯|j dƒrï|j ||ƒ|j |d|ƒnz|j dƒr$|j ||ƒ|j |d|ƒnE|j dƒrY|j ||ƒ|j |d|ƒn|j||ƒdS(sÊInitialize an array Parameters ---------- desc : InitDesc Initialization pattern descriptor. arr : NDArray The array to be initialized. NRttweighttbiastgammatbeta(t isinstanceR t _legacy_initR RR tgettcreatet _init_weightR+tendswitht _init_biast _init_gammat _init_betat _init_default(RR(R*R)((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyt__call__us,  cC@sútjdtddƒt|tƒs4tdƒ‚nt|tƒsRtdƒ‚n|jdƒrt|j||ƒn‚|jdƒr¥|j dƒr¥|j ||ƒnQ|jdƒrÖ|j d ƒrÖ|j ||ƒn |j d ƒrø|j ||ƒnþ|j d ƒr|j ||ƒnÜ|j d ƒr<|j||ƒnº|j dƒr^|j||ƒn˜|j d ƒr€|j ||ƒnv|j d ƒr¢|j||ƒnT|j dƒrÄ|j ||ƒn2|j dƒræ|j ||ƒn|j||ƒdS(sÉLegacy initialization method. Parameters ---------- name : str Name of corresponding NDArray. arr : NDArray NDArray to be initialized. sCalling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.t stacklevelisname must be stringsarr must be NDArrayt upsamplingtstn_locR1R2R3R4t moving_meant moving_vartmoving_inv_vart moving_avgN(twarningstwarntDeprecationWarningR5Rt TypeErrorRt startswitht_init_bilinearR:t _init_zerot_init_loc_biasR;R<R=R9t _init_oneR>(RRR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR6žs<  c C@sétjtj|jƒddƒ}|j}tj|ddƒ}d|d|dd|}xsttj|ƒƒD]\}||d}||d|d} dt|||ƒdt| ||ƒ||ísN(RRRRRRR%R+R-R?R6RLRNRMROR;R<R=R9R>(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR5s     ) ,       t initializercC@s t|ƒS(s@Registers a custom initializer. Custom initializers can be created by extending `mx.init.Initializer` and implementing the required functions like `_init_weight` and `_init_bias`. The created initializer must be registered using `mx.init.register` before it can be called by name. Parameters ---------- klass : class A subclass of `mx.init.Initializer` that needs to be registered as a custom initializer. Example ------- >>> # Create and register a custom initializer that ... # initializes weights to 0.1 and biases to 1. ... >>> @mx.init.register ... @alias('myinit') ... class CustomInit(mx.init.Initializer): ... def __init__(self): ... super(CustomInit, self).__init__() ... def _init_weight(self, _, arr): ... arr[:] = 0.1 ... def _init_bias(self, _, arr): ... arr[:] = 1 ... >>> # Module is an instance of 'mxnet.module.Module' ... >>> module.init_params("custominit") >>> # module.init_params("myinit") >>> # module.init_params(CustomInit()) (t _register(tklass((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pytregisterûs!tLoadcB@s&eZdZded„Zd„ZRS(síInitializes variables by loading data from file or dict. **Note** Load will drop ``arg:`` or ``aux:`` from name and initialize the variables that match with the prefix dropped. Parameters ---------- param: str or dict of str->`NDArray` Parameter file or dict mapping name to NDArray. default_init: Initializer Default initializer when name is not found in `param`. verbose: bool Flag for enabling logging of source when initializing. cC@s®t|tƒrt|ƒ}nt|tƒs3t‚i|_xY|jƒD]K\}}|jdƒss|jdƒr‡||j|d>> # Given 'module', an instance of 'mxnet.module.Module', initialize biases to zero ... # and every other parameter to random values with uniform distribution. ... >>> init = mx.initializer.Mixed(['bias', '.*'], [mx.init.Zero(), mx.init.Uniform(0.1)]) >>> module.init_params(init) >>> >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected1_weight [[ 0.0097627 0.01856892 0.04303787]] fullyconnected1_bias [ 0.] cC@sVt|ƒt|ƒkst‚ttg|D]}tj|ƒ^q+|ƒƒ|_dS(N(tlenR_tlisttziptretcompiletmap(Rtpatternst initializerstp((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRkscC@sNx7|jD],\}}|j|ƒr |||ƒdSq Wtddƒ‚dS(Ns5Parameter name %s did not match any pattern. Considers7add a ".*" pattern at the and with default Initializer.(RrtmatchRb(RRR*tprogR)((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR?os  (RRRRR?(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRlNs RStZerocB@s eZdZd„Zd„ZRS(s¹Initializes weights to zero. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to zero. ... >>> init = mx.initializer.Zero() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0. 0. 0.]] cC@stt|ƒjƒdS(N(R RxR(R((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRŠscC@s d|(dS(Ni((RRZR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9s(RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRxws tonestOnecB@s eZdZd„Zd„ZRS(s¶Initializes weights to one. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to one. ... >>> init = mx.initializer.One() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 1. 1. 1.]] cC@stt|ƒjƒdS(N(R RzR(R((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR£scC@s d|(dS(Ni((RRZR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9¦s(RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRzs tConstantcB@s eZdZd„Zd„ZRS(sêInitializes the weights to a given value. The value passed in can be a scalar or a NDarray that matches the shape of the parameter to be set. Parameters ---------- value : float, NDArray Value to set. cC@s&tt|ƒjd|ƒ||_dS(Ntvalue(R R{RR|(RR|((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR´scC@s|j|(dS(N(R|(RRZR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9¸s(RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR{©s  tUniformcB@s#eZdZdd„Zd„ZRS(sInitializes weights with random values uniformly sampled from a given range. Parameters ---------- scale : float, optional The bound on the range of the generated random values. Values are generated from the range [-`scale`, `scale`]. Default scale is 0.07. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values uniformly sampled between -0.1 and 0.1. ... >>> init = mx.init.Uniform(0.1) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0.01360891 -0.02144304 0.08511933]] gìQ¸…ë±?cC@s&tt|ƒjd|ƒ||_dS(Ntscale(R R}RR~(RR~((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRÕscC@s!tj|j |jd|ƒdS(Ntout(RtuniformR~(RRZR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9Ùs(RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR}»s tNormalcB@s#eZdZdd„Zd„ZRS(s Initializes weights with random values sampled from a normal distribution with a mean of zero and standard deviation of `sigma`. Parameters ---------- sigma : float, optional Standard deviation of the normal distribution. Default standard deviation is 0.01. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values sampled from a normal distribution. ... >>> init = mx.init.Normal(0.5) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[-0.3214761 -0.12660924 0.53789419]] g{®Gáz„?cC@s&tt|ƒjd|ƒ||_dS(Ntsigma(R RRR‚(RR‚((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRöscC@stjd|jd|ƒdS(NiR(RtnormalR‚(RRZR*((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9ús(RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRÜs t OrthogonalcB@s&eZdZddd„Zd„ZRS(sžInitialize weight as orthogonal matrix. This initializer implements *Exact solutions to the nonlinear dynamics of learning in deep linear neural networks*, available at https://arxiv.org/abs/1312.6120. Parameters ---------- scale : float optional Scaling factor of weight. rand_type: string optional Use "uniform" or "normal" random number to initialize weight. g9´Èv¾Ÿö?R€cC@s5tt|ƒjd|d|ƒ||_||_dS(NR~t rand_type(R R„RR~R…(RR~R…((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRs c C@sõ|jd}tj|jdƒ}|jdkrYtjddd||fƒjƒ}n6|jdkrtjddd||fƒjƒ}ntjj |d t ƒ\}}}|j|jkrË|}n|}|j |j |jƒ}||(dS( NiiR€gð¿gð?RURƒgt full_matrices( RURRRTR…RR€tasnumpyRƒtlinalgtsvdRR~RY( RRZR*tnouttninttmptutvtres((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9s ''! (RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR„ýstXaviercB@s)eZdZdddd„Zd„ZRS(sèReturns an initializer performing "Xavier" initialization for weights. This initializer is designed to keep the scale of gradients roughly the same in all layers. By default, `rnd_type` is ``'uniform'`` and `factor_type` is ``'avg'``, the initializer fills the weights with random numbers in the range of :math:`[-c, c]`, where :math:`c = \sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}`. :math:`n_{in}` is the number of neurons feeding into weights, and :math:`n_{out}` is the number of neurons the result is fed to. If `rnd_type` is ``'uniform'`` and `factor_type` is ``'in'``, the :math:`c = \sqrt{\frac{3.}{n_{in}}}`. Similarly when `factor_type` is ``'out'``, the :math:`c = \sqrt{\frac{3.}{n_{out}}}`. If `rnd_type` is ``'gaussian'`` and `factor_type` is ``'avg'``, the initializer fills the weights with numbers from normal distribution with a standard deviation of :math:`\sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}`. Parameters ---------- rnd_type: str, optional Random generator type, can be ``'gaussian'`` or ``'uniform'``. factor_type: str, optional Can be ``'avg'``, ``'in'``, or ``'out'``. magnitude: float, optional Scale of random number. R€tavgicC@sJtt|ƒjd|d|d|ƒ||_||_t|ƒ|_dS(Ntrnd_typet factor_typet magnitude(R RRR’R“tfloatR”(RR’R“R”((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRBs   c C@sW|j}d}t|ƒdkr9tdj|ƒƒ‚nt|ƒdkratj|dƒ}n|d||d|}}d}|jdkr¤||d}n<|jdkr¼|}n$|jd krÔ|}n td ƒ‚tj|j|ƒ}|j d krt j | |d |ƒn4|j d krGt j d|d |ƒn td ƒ‚dS(Ngð?isLXavier initializer cannot be applied to vector {0}. It requires at least 2D.iiR‘g@tinRsIncorrect factor typeR€tgaussiansUnknown random type( RURmRbtformatRRRTR“RR”R’RR€Rƒ( RRR*RUthw_scaletfan_intfan_outtfactorR~((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR9Js,     (RRRRR9(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyR"st MSRAPrelucB@seZdZddd„ZRS(s<Initialize the weight according to a MSRA paper. This initializer implements *Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification*, available at https://arxiv.org/abs/1502.01852. This initializer is proposed for initialization related to ReLu activation, it maked some changes on top of Xavier method. Parameters ---------- factor_type: str, optional Can be ``'avg'``, ``'in'``, or ``'out'``. slope: float, optional initial slope of any PReLU (or similar) nonlinearities. R‘gÐ?cC@sIdd|d}tt|ƒjd||ƒi|d6|d6|_dS(Ng@iiR—R“tslope(R RRR(RR“RžR”((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRws(RRRR(((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRdstBilinearcB@s eZdZd„Zd„ZRS(s(Initialize weight for upsampling layers.cC@stt|ƒjƒdS(N(R RŸR(R((sQ/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/initializer.pyRsc C@sétjtj|jƒddƒ}|j}tj|ddƒ}d|d|dd|}xsttj|ƒƒD]\}||d}||d|d} dt|||ƒdt| ||ƒ||sX     Á $/)    $A