ó —Àv]c@@s¥dZddlmZmZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddlmZmZddlmZdd lmZdd lmZd efd „ƒYZd efd„ƒYZejedƒZejedƒZejedƒZd„Zdefd„ƒYZdefd„ƒYZ eedƒdefd„ƒYƒƒZ!eedƒdefd„ƒYƒƒZ"edefd„ƒYƒZ#edefd„ƒYƒZ$edefd „ƒYƒZ%ed!efd"„ƒYƒZ&ed#efd$„ƒYƒZ'ed%e'fd&„ƒYƒZ(ed'efd(„ƒYƒZ)ed)efd*„ƒYƒZ*ed+efd,„ƒYƒZ+dS(-sWeight initializer.i(tabsolute_importtprint_functionN(tsqrti(t string_types(tNDArraytload(trandom(tregistry(tndarraytInitDesccB@seZdZddd„ZRS(s) Descriptor for the initialization pattern. Parameters ---------- name : str Name of variable. attrs : dict of str to str Attributes of this variable taken from ``Symbol.attr_dict``. global_init : Initializer Global initializer to fallback to. cC@s7tt|ƒj||ƒ}|p$i|_||_|S(N(tsuperR t__new__tattrst global_init(tclstnameR R tret((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR /s N(t__name__t __module__t__doc__tNoneR (((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR "s t InitializercB@s­eZdZd„Zedd„Zd„Zd„Zd„Z d„Z d„Z d„Z d „Z d „Zd „Zd „Zd „Zd„Zd„Zd„Zd„ZRS(s!The base class of an initializer.cK@s||_t|_d|_dS(N(t_kwargstFalset_verboseRt _print_func(tselftkwargs((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyt__init__8s  cC@s4||_|dkr'd„}|}n||_|S(spSwitch on/off verbose mode Parameters ---------- verbose : bool switch on/off verbose mode print_func : function A function that computes statistics of initialized arrays. Takes an `NDArray` and returns an `str`. Defaults to mean absolute value str((abs(x)/size(x)).asscalar()). cS@s&ttj|ƒt|jƒjƒƒS(s%returns |x|/size(x), async execution.(tstrRtnormRtsizetasscalar(tx((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyt asum_statKsN(RRR(Rtverboset print_funcR"((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyt set_verbosity=s     cC@s8|jr4|jr4tjd|||j|ƒƒndS(sóInternal verbose print function Parameters ---------- desc : InitDesc or str name of the array init : str initializer pattern arr : NDArray initialized array sInitialized %s as %s: %sN(RRtloggingtinfo(Rtdesctinittarr((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyt_verbose_printRs cC@s"tj|jjjƒ|jgƒS(sSaves the initializer to string Returns ------- str JSON formatted string that describes the initializer. Examples -------- >>> # Create initializer and retrieve its parameters ... >>> init = mx.init.Normal(0.5) >>> init.dumps() '["normal", {"sigma": 0.5}]' >>> init = mx.init.Xavier(factor_type="in", magnitude=2.34) >>> init.dumps() '["xavier", {"rnd_type": "uniform", "magnitude": 2.34, "factor_type": "in"}]' (tjsontdumpst __class__RtlowerR(R((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR-ascC@sAt|tƒs#|j||ƒdS|jdkr>||_n|jjddƒ}|r…t|ƒj||ƒ|j |||ƒn¸|j dƒrº|j||ƒ|j |d|ƒnƒ|j dƒrï|j ||ƒ|j |d|ƒnN|j dƒr$|j ||ƒ|j |d|ƒn|j dƒrY|j ||ƒ|j |d|ƒnä|j dƒrŽ|j||ƒ|j |d|ƒn¯|j d ƒrÃ|j||ƒ|j |d |ƒnz|j d ƒrø|j||ƒ|j |d |ƒnE|j d ƒr-|j||ƒ|j |d |ƒn|j||ƒdS( sÊInitialize an array Parameters ---------- desc : InitDesc Initialization pattern descriptor. arr : NDArray The array to be initialized. NRttweighttbiastgammatbetatmintmaxtweight_quantizet bias_quantize(t isinstanceR t _legacy_initR RR tgettcreatet _init_weightR+tendswitht _init_biast _init_gammat _init_betat _init_zerot _init_onet_init_quantized_weightt_init_quantized_biast _init_default(RR(R*R)((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyt__call__vsD  cC@s>tjdtddƒt|tƒs4tdƒ‚nt|tƒsRtdƒ‚n|jdƒrt|j||ƒnÆ|jdƒr¥|j dƒr¥|j ||ƒn•|jdƒrÖ|j d ƒrÖ|j ||ƒnd|j d ƒrø|j ||ƒnB|j d ƒr|j ||ƒn |j d ƒr<|j||ƒnþ|j dƒr^|j||ƒnÜ|j d ƒr€|j ||ƒnº|j d ƒr¢|j||ƒn˜|j dƒrÄ|j ||ƒnv|j dƒræ|j ||ƒnT|j dƒr|j ||ƒn2|j dƒr*|j||ƒn|j||ƒdS(sÉLegacy initialization method. Parameters ---------- name : str Name of corresponding NDArray. arr : NDArray NDArray to be initialized. sCalling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.t stacklevelisname must be stringsarr must be NDArrayt upsamplingtstn_locR1R2R3R4t moving_meant moving_vartmoving_inv_vart moving_avgR5R6N(twarningstwarntDeprecationWarningR9Rt TypeErrorRt startswitht_init_bilinearR>RBt_init_loc_biasR?R@RAR=RCRF(RRR*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR:«sD  c C@sétjtj|jƒddƒ}|j}tj|ddƒ}d|d|dd|}xsttj|ƒƒD]\}||d}||d|d} dt|||ƒdt| ||ƒ||>> # Create and register a custom initializer that ... # initializes weights to 0.1 and biases to 1. ... >>> @mx.init.register ... @alias('myinit') ... class CustomInit(mx.init.Initializer): ... def __init__(self): ... super(CustomInit, self).__init__() ... def _init_weight(self, _, arr): ... arr[:] = 0.1 ... def _init_bias(self, _, arr): ... arr[:] = 1 ... >>> # Module is an instance of 'mxnet.module.Module' ... >>> module.init_params("custominit") >>> # module.init_params("myinit") >>> # module.init_params(CustomInit()) (t _register(tklass((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pytregisters!tLoadcB@s&eZdZded„Zd„ZRS(síInitializes variables by loading data from file or dict. **Note** Load will drop ``arg:`` or ``aux:`` from name and initialize the variables that match with the prefix dropped. Parameters ---------- param: str or dict of str->`NDArray` Parameter file or dict mapping name to NDArray. default_init: Initializer Default initializer when name is not found in `param`. verbose: bool Flag for enabling logging of source when initializing. cC@s®t|tƒrt|ƒ}nt|tƒs3t‚i|_xY|jƒD]K\}}|jdƒss|jdƒr‡||j|d>> # Given 'module', an instance of 'mxnet.module.Module', initialize biases to zero ... # and every other parameter to random values with uniform distribution. ... >>> init = mx.initializer.Mixed(['bias', '.*'], [mx.init.Zero(), mx.init.Uniform(0.1)]) >>> module.init_params(init) >>> >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected1_weight [[ 0.0097627 0.01856892 0.04303787]] fullyconnected1_bias [ 0.] cC@sVt|ƒt|ƒkst‚ttg|D]}tj|ƒ^q+|ƒƒ|_dS(N(tlenRetlisttziptretcompiletmap(Rtpatternst initializerstp((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRƒscC@sNx7|jD],\}}|j|ƒr |||ƒdSq Wtddƒ‚dS(Ns5Parameter name %s did not match any pattern. Considers7add a ".*" pattern at the and with default Initializer.(R}tmatchRm(RRR*tprogR)((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRG‡s  (RRRRRG(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRwfs RYtZerocB@s eZdZd„Zd„ZRS(s¹Initializes weights to zero. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to zero. ... >>> init = mx.initializer.Zero() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0. 0. 0.]] cC@stt|ƒjƒdS(N(R RƒR(R((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR¢scC@s d|(dS(Ni((RR`R*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=¥s(RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRƒs tonestOnecB@s eZdZd„Zd„ZRS(s¶Initializes weights to one. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to one. ... >>> init = mx.initializer.One() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 1. 1. 1.]] cC@stt|ƒjƒdS(N(R R…R(R((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR»scC@s d|(dS(Ni((RR`R*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=¾s(RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR…¨s tConstantcB@s eZdZd„Zd„ZRS(sêInitializes the weights to a given value. The value passed in can be a scalar or a NDarray that matches the shape of the parameter to be set. Parameters ---------- value : float, NDArray Value to set. cC@s&tt|ƒjd|ƒ||_dS(Ntvalue(R R†RR‡(RR‡((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRÌscC@s|j|(dS(N(R‡(RR`R*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=Ðs(RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR†Ás  tUniformcB@s#eZdZdd„Zd„ZRS(sInitializes weights with random values uniformly sampled from a given range. Parameters ---------- scale : float, optional The bound on the range of the generated random values. Values are generated from the range [-`scale`, `scale`]. Default scale is 0.07. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values uniformly sampled between -0.1 and 0.1. ... >>> init = mx.init.Uniform(0.1) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0.01360891 -0.02144304 0.08511933]] gìQ¸…ë±?cC@s&tt|ƒjd|ƒ||_dS(Ntscale(R RˆRR‰(RR‰((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRíscC@s!tj|j |jd|ƒdS(Ntout(RtuniformR‰(RR`R*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=ñs(RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRˆÓs tNormalcB@s#eZdZdd„Zd„ZRS(s Initializes weights with random values sampled from a normal distribution with a mean of zero and standard deviation of `sigma`. Parameters ---------- sigma : float, optional Standard deviation of the normal distribution. Default standard deviation is 0.01. Example ------- >>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values sampled from a normal distribution. ... >>> init = mx.init.Normal(0.5) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[-0.3214761 -0.12660924 0.53789419]] g{®Gáz„?cC@s&tt|ƒjd|ƒ||_dS(Ntsigma(R RŒRR(RR((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRscC@stjd|jd|ƒdS(NiRŠ(RtnormalR(RR`R*((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=s(RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRŒôs t OrthogonalcB@s&eZdZddd„Zd„ZRS(sžInitialize weight as orthogonal matrix. This initializer implements *Exact solutions to the nonlinear dynamics of learning in deep linear neural networks*, available at https://arxiv.org/abs/1312.6120. Parameters ---------- scale : float optional Scaling factor of weight. rand_type: string optional Use "uniform" or "normal" random number to initialize weight. g9´Èv¾Ÿö?R‹cC@s5tt|ƒjd|d|ƒ||_||_dS(NR‰t rand_type(R RRR‰R(RR‰R((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR&s c C@sõ|jd}tj|jdƒ}|jdkrYtjddd||fƒjƒ}n6|jdkrtjddd||fƒjƒ}ntjj |d t ƒ\}}}|j|jkrË|}n|}|j |j |jƒ}||(dS( NiiR‹gð¿gð?R[RŽgt full_matrices( R[RXRZRRR‹RjRŽtlinalgtsvdRR‰R_( RR`R*tnouttninttmptutvtres((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=+s ''! (RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRstXaviercB@s)eZdZdddd„Zd„ZRS(sèReturns an initializer performing "Xavier" initialization for weights. This initializer is designed to keep the scale of gradients roughly the same in all layers. By default, `rnd_type` is ``'uniform'`` and `factor_type` is ``'avg'``, the initializer fills the weights with random numbers in the range of :math:`[-c, c]`, where :math:`c = \sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}`. :math:`n_{in}` is the number of neurons feeding into weights, and :math:`n_{out}` is the number of neurons the result is fed to. If `rnd_type` is ``'uniform'`` and `factor_type` is ``'in'``, the :math:`c = \sqrt{\frac{3.}{n_{in}}}`. Similarly when `factor_type` is ``'out'``, the :math:`c = \sqrt{\frac{3.}{n_{out}}}`. If `rnd_type` is ``'gaussian'`` and `factor_type` is ``'avg'``, the initializer fills the weights with numbers from normal distribution with a standard deviation of :math:`\sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}`. Parameters ---------- rnd_type: str, optional Random generator type, can be ``'gaussian'`` or ``'uniform'``. factor_type: str, optional Can be ``'avg'``, ``'in'``, or ``'out'``. magnitude: float, optional Scale of random number. R‹tavgicC@sJtt|ƒjd|d|d|ƒ||_||_t|ƒ|_dS(Ntrnd_typet factor_typet magnitude(R RšRRœRtfloatRž(RRœRRž((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRZs   c C@sW|j}d}t|ƒdkr9tdj|ƒƒ‚nt|ƒdkratj|dƒ}n|d||d|}}d}|jdkr¤||d}n<|jdkr¼|}n$|jd krÔ|}n td ƒ‚tj|j|ƒ}|j d krt j | |d |ƒn4|j d krGt j d|d |ƒn td ƒ‚dS(Ngð?isLXavier initializer cannot be applied to vector {0}. It requires at least 2D.iiR›g@tinRŠsIncorrect factor typeR‹tgaussiansUnknown random type( R[RxRmtformatRXRZRRRžRœRR‹RŽ( RRR*R[thw_scaletfan_intfan_outtfactorR‰((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=bs,     (RRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRš:st MSRAPrelucB@seZdZddd„ZRS(s<Initialize the weight according to a MSRA paper. This initializer implements *Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification*, available at https://arxiv.org/abs/1502.01852. This initializer is proposed for initialization related to ReLu activation, it maked some changes on top of Xavier method. Parameters ---------- factor_type: str, optional Can be ``'avg'``, ``'in'``, or ``'out'``. slope: float, optional initial slope of any PReLU (or similar) nonlinearities. R›gÐ?cC@sIdd|d}tt|ƒjd||ƒi|d6|d6|_dS(Ng@iiR¡Rtslope(R R§RR(RRR¨Rž((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyRs(RRRR(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR§|stBilinearcB@s eZdZd„Zd„ZRS(s(Initialize weight for upsampling layers.cC@stt|ƒjƒdS(N(R R©R(R((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR—sc C@sétjtj|jƒddƒ}|j}tj|ddƒ}d|d|dd|}xsttj|ƒƒD]\}||d}||d|d} dt|||ƒdt| ||ƒ||R´Rt pack_weights(RR(R*RºtcelltargsRtarg_desc((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR=ßs (RRRRRR=(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pyR®½s(,Rt __future__RRR{R&ROR,tmathRtnumpyRXtbaseRRRRR0RRRR tobjectRtget_register_funcRotget_alias_functaliastget_create_funcR<RqRrRwRƒR…R†RˆRŒRRšR§R©RªR®(((s2/tmp/pip-install-Qvdv_2/mxnet/mxnet/initializer.pytsX     Ø $/)    $A