ó ùµÈ[c @sçdZddddddddd d d d g Zd dlmZmZd dlmZmZmZddl m Z m Z ddl m Z ddlmZddlmZd„Zd„Zd„Zd&d„Zd„Zde fd„ƒYZdee fd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd „ƒYZdefd!„ƒYZd efd"„ƒYZd efd#„ƒYZ d efd$„ƒYZ!d efd%„ƒYZ"d&S('s5Definition of various recurrent neural network cells.t RecurrentCelltHybridRecurrentCelltRNNCelltLSTMCelltGRUCelltSequentialRNNCelltHybridSequentialRNNCellt DropoutCellt ModifierCellt ZoneoutCellt ResidualCelltBidirectionalCelli(tsymboltndarray(t string_typest numeric_typest_as_listi(tBlockt HybridBlock(t_indent(t tensor_types(t LeakyReLUcCs)tg|D]}|j|ƒ^q gƒS(N(tsumt state_info(tcellst batch_sizetc((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_cells_state_info%scKs)tg|D]}|j|^q gƒS(N(Rt begin_state(RtkwargsR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_cells_begin_state(sc CsŠ|dkr†|tkrht|tƒr0|jn |dj}| |jd|jd|ƒ}WdQXq†|jd|jd|ƒ}n|S(NitfuncR(tNoneR t isinstanceRtcontextRtzeros(tcelltFRtinputsRtctx((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_get_begin_state+s  %$c Cs)|dk stdƒ‚|jdƒ}|jdƒ}d}|dk rW|jdƒn|}t|tjƒrÒt} |tkràt|jƒƒdks¥tdƒ‚t tj |d|d|d dƒƒ}qànt|t j ƒr\t } |j |}|tkrà|dks(||j |ks(t‚tt j |d|d|j |d dƒƒ}qàn„|dks€t|ƒ|ks€t‚t|dtjƒrŸt} nt } |dj d}|tkrà| jd||Œ}|}nt|tƒr||kr| j|d |d |ƒ}n||| |fS( NsVunroll(inputs=None) has been deprecated. Please create input variables outside unroll.tTtNiis|unroll doesn't allow grouped symbol as input. Please convert to list with list(inputs) first or let unroll handle splitting.taxist num_outputst squeeze_axistdim1tdim2(R tAssertionErrortfindR!R tSymboltFalsetlent list_outputstlisttsplitR tNDArraytshapeRtTruetstackRtswapaxes( tlengthR&tlayouttmerget in_layoutR+t batch_axisRtin_axisR%((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_format_sequence5s@ !    % $   c CsŽ|dk st‚t|tƒs9|jd||Œ}n|j|d|dtd|ƒ}|sŠt|j|d|d|dtƒƒ}n|S(NR+tsequence_lengthtuse_sequence_lengthR,R-( R R0R!RR;t SequenceMaskR:RR7(R%tdataR=t valid_lengtht time_axisR?toutputs((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_mask_sequence_variable_length^s cBskeZdZd d d„Zd„Zdd„Zdejd„Z d dd d d„Z d„Z d „Z RS( s²Abstract base class for RNN cells Parameters ---------- prefix : str, optional Prefix for names of `Block`s (this prefix is also used for names of weights if `params` is `None` i.e. if `params` are being created and not reused) params : Parameter or None, default None Container for weight sharing between cells. A new Parameter container is created if `params` is `None`. cCs6tt|ƒjd|d|ƒt|_|jƒdS(Ntprefixtparams(tsuperRt__init__R3t _modifiedtreset(tselfRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROvs cCs:d|_d|_x!|jjƒD]}|jƒq"WdS(s1Reset before re-using the cell for another graph.iÿÿÿÿN(t _init_countert_countert _childrentvaluesRQ(RRR$((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRQ{s  icCs tƒ‚dS(s&shape and layout information of statesN(tNotImplementedError(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR‚scKsš|j stdƒ‚g}xw|j|ƒD]f}|jd7_|dk r]|j|ƒn|}|dd|j|jf|}|j|ƒq,W|S(s=Initial state for this cell. Parameters ---------- func : callable, default symbol.zeros Function for creating initial state. For Symbol API, func can be `symbol.zeros`, `symbol.uniform`, `symbol.var etc`. Use `symbol.var` if you want to directly feed input as states. For NDArray API, func can be `ndarray.zeros`, `ndarray.ones`, etc. batch_size: int, default 0 Only required for NDArray API. Size of the batch ('N' in layout) dimension of input. **kwargs : Additional keyword arguments passed to func. For example `mean`, `std`, `dtype`, etc. Returns ------- states : nested list of Symbol Starting states for the first RNN step. syAfter applying modifier cells (e.g. ZoneoutCell) the base cell cannot be called directly. Call the modifier cell instead.itnames%sbegin_state_%dN(RPR0RRSR tupdatet_prefixtappend(RRRRRtstatestinfotstate((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR†s    tNTCc CsL|jƒt|||tƒ\}}}} t||||| ƒ}|} g} g} xVt|ƒD]H} ||| | ƒ\}} | j|ƒ|dk rb| j| ƒqbqbW|dk r!gt| ŒD]6}|j|j dd|Œd|dt ddƒ^qÇ} t || |||t ƒ} nt|| ||ƒ\} }}}| | fS(s Unrolls an RNN cell across time steps. Parameters ---------- length : int Number of steps to unroll. inputs : Symbol, list of Symbol, or None If `inputs` is a single Symbol (usually the output of Embedding symbol), it should have shape (batch_size, length, ...) if `layout` is 'NTC', or (length, batch_size, ...) if `layout` is 'TNC'. If `inputs` is a list of symbols (usually output of previous unroll), they should all have shape (batch_size, ...). begin_state : nested list of Symbol, optional Input states created by `begin_state()` or output state of another cell. Created from `begin_state()` if `None`. layout : str, optional `layout` of input symbol. Only used if inputs is a single Symbol. merge_outputs : bool, optional If `False`, returns outputs as a list of Symbols. If `True`, concatenates output across time steps and returns a single symbol with shape (batch_size, length, ...) if layout is 'NTC', or (length, batch_size, ...) if layout is 'TNC'. If `None`, output whatever is faster. valid_length : Symbol, NDArray or None `valid_length` specifies the length of the sequences in the batch without padding. This option is especially useful for building sequence-to-sequence models where the input and output sequences would potentially be padded. If `valid_length` is None, all sequences are assumed to have the same length. If `valid_length` is a Symbol or NDArray, it should have shape (batch_size,). The ith element will be the length of the ith sequence in the batch. The last valid state will be return and the padded outputs will be masked with 0. Note that `valid_length` must be smaller or equal to `length`. Returns ------- outputs : list of Symbol or Symbol Symbol (if `merge_outputs` is True) or list of Symbols (if `merge_outputs` is False) corresponding to the output from the RNN from this unrolling. states : list of Symbol The new state of this RNN after this unrolling. The type of this symbol is same as the output of `begin_state()`. R+iRDREN( RQRCR3R(trangeR[R tzipt SequenceLastR;R:RK(RRR=R&RR>t merge_outputsRHR+R%RR\RJt all_statestitoutputtele_listt_((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pytunroll¯s"5 !   F!cKsªi|jd6|jd6|jd6|jd6j|ƒ}|rJ|||St|tƒro|j|d||St|tƒr|j|ddd|j |S|||S(s-Get activation function. Convert if is stringttanhtrelutsigmoidtsoftsigntact_typetleakytslope( RjRkRlRmtgetR!Rt ActivationRt_alpha(RRR%R&t activationRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_get_activationýs    cCs(|jd7_tt|ƒj||ƒS(s¦Unrolls the recurrent cell for one time step. Parameters ---------- inputs : sym.Variable Input symbol, 2D, of shape (batch_size * num_units). states : list of sym.Variable RNN state from previous step or the output of begin_state(). Returns ------- output : Symbol Symbol corresponding to the output from the RNN when unrolling for a single time step. states : list of Symbol The new state of this RNN after this unrolling. The type of this symbol is same as the output of `begin_state()`. This can be used as an input state to the next time step of this RNN. See Also -------- begin_state: This function can provide the states for the first time step. unroll: This function unrolls an RNN for a given number of (>=1) time steps. i(RTRNRtforward(RRR&R\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRv sN( t__name__t __module__t__doc__R RORQRR R#RRiRuRv(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRis   )  M cBs&eZdZddd„Zd„ZRS(s'HybridRecurrentCell supports hybridize.cCs#tt|ƒjd|d|ƒdS(NRLRM(RNRRO(RRRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRO,scOs t‚dS(N(RW(RRR%txtargsR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pythybrid_forward/sN(RwRxRyR ROR|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR*sc BsVeZdZdd d dddd d d„Zdd„Zd„Zd„Zd„ZRS( sãElman RNN recurrent neural network cell. Each call computes the following function: .. math:: h_t = \tanh(w_{ih} * x_t + b_{ih} + w_{hh} * h_{(t-1)} + b_{hh}) where :math:`h_t` is the hidden state at time `t`, and :math:`x_t` is the hidden state of the previous layer at time `t` or :math:`input_t` for the first layer. If nonlinearity='relu', then `ReLU` is used instead of `tanh`. Parameters ---------- hidden_size : int Number of units in output symbol activation : str or Symbol, default 'tanh' Type of activation function. i2h_weight_initializer : str or Initializer Initializer for the input weights matrix, used for the linear transformation of the inputs. h2h_weight_initializer : str or Initializer Initializer for the recurrent weights matrix, used for the linear transformation of the recurrent state. i2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. prefix : str, default 'rnn_' Prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None Container for weight sharing between cells. Created if `None`. Inputs: - **data**: input tensor with shape `(batch_size, input_size)`. - **states**: a list of one initial recurrent state tensor with shape `(batch_size, num_hidden)`. Outputs: - **out**: output tensor with shape `(batch_size, num_hidden)`. - **next_states**: a list of one output recurrent state tensor with the same shape as `states`. RjR#ic Csìtt|ƒjd|d| ƒ||_||_||_|jjdd||fd|dtƒ|_ |jjdd||fd|dtƒ|_ |jjdd|fd|dtƒ|_ |jjd d|fd|dtƒ|_ dS( NRLRMt i2h_weightR9tinittallow_deferred_initt h2h_weightti2h_biasth2h_bias( RNRROt _hidden_sizet _activationt _input_sizeRMRqR:R}R€RR‚( RRt hidden_sizeRtti2h_weight_initializerth2h_weight_initializerti2h_bias_initializerth2h_bias_initializert input_sizeRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRObs       cCsi||jfd6dd6gS(NR9tNCt __layout__(Rƒ(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRwscCsdS(Ntrnn((RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt_aliaszscCs„d}t|dƒr"|d7}n|d7}|jj}dj|drR|dnd|dƒ}|jd|jjd ||jS( Ns{name}({mapping}R„s, {_activation}t)s {0} -> {1}iiRXtmapping(thasattrR}R9tformatR t __class__Rwt__dict__(RRtsR9R‘((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt__repr__}s   *c Cs¾d|j}|jd|d|d|d|jd|dƒ} |jd|dd|d|d|jd|d ƒ} |j| | d|d ƒ} |j|| |jd|d ƒ} | | gfS( Nst%d_RGtweighttbiast num_hiddenRXti2hith2htplus0tout(RTtFullyConnectedRƒt elemwise_addRuR„( RRR%R&R\R}R€RR‚RLR›Rœt i2h_plus_h2hRf((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|ˆs      N( RwRxRyR RORRR—R|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR3s.   c BsYeZdZd d dddd d ddd„ Zdd„Zd„Zd„Zd „ZRS( s| Long-Short Term Memory (LSTM) network cell. Each call computes the following function: .. math:: \begin{array}{ll} i_t = sigmoid(W_{ii} x_t + b_{ii} + W_{hi} h_{(t-1)} + b_{hi}) \\ f_t = sigmoid(W_{if} x_t + b_{if} + W_{hf} h_{(t-1)} + b_{hf}) \\ g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hc} h_{(t-1)} + b_{hg}) \\ o_t = sigmoid(W_{io} x_t + b_{io} + W_{ho} h_{(t-1)} + b_{ho}) \\ c_t = f_t * c_{(t-1)} + i_t * g_t \\ h_t = o_t * \tanh(c_t) \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell state at time `t`, :math:`x_t` is the hidden state of the previous layer at time `t` or :math:`input_t` for the first layer, and :math:`i_t`, :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, and out gates, respectively. Parameters ---------- hidden_size : int Number of units in output symbol. i2h_weight_initializer : str or Initializer Initializer for the input weights matrix, used for the linear transformation of the inputs. h2h_weight_initializer : str or Initializer Initializer for the recurrent weights matrix, used for the linear transformation of the recurrent state. i2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. prefix : str, default 'lstm_' Prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None, default None Container for weight sharing between cells. Created if `None`. activation : str, default 'tanh' Activation type to use. See nd/symbol Activation for supported types. recurrent_activation : str, default 'sigmoid' Activation type to use for the recurrent step. See nd/symbol Activation for supported types. Inputs: - **data**: input tensor with shape `(batch_size, input_size)`. - **states**: a list of two initial recurrent state tensors. Each has shape `(batch_size, num_hidden)`. Outputs: - **out**: output tensor with shape `(batch_size, num_hidden)`. - **next_states**: a list of two output recurrent state tensors. Each has the same shape as `states`. R#iRjRlc Cstt|ƒjd|d|ƒ||_||_|jjddd||fd|dtƒ|_|jjddd||fd|dtƒ|_ |jjd dd|fd|dtƒ|_ |jjd dd|fd|dtƒ|_ | |_ | |_ dS( NRLRMR}R9iR~RR€RR‚(RNRRORƒR…RMRqR:R}R€RR‚R„t_recurrent_activation( RRR†R‡RˆR‰RŠR‹RLRMRttrecurrent_activation((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROÓs"       cCs8i||jfd6dd6i||jfd6dd6gS(NR9RŒR(Rƒ(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRìscCsdS(Ntlstm((RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRðscCs^d}|jj}dj|dr,|dnd|dƒ}|jd|jjd||jS(Ns{name}({mapping})s {0} -> {1}iiRXR‘(R}R9R“R R”RwR•(RRR–R9R‘((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—ós  *c Csèd|j}|jd|d|d|d|jdd|dƒ} |jd|d d|d|d|jdd|d ƒ} |j| | d|d ƒ} |j| d dd|d ƒ} |j|| d |jd|dƒ} |j|| d|jd|dƒ}|j|| d|jd|dƒ}|j|| d|jd|dƒ}|jj |j ||dd|dƒ|j | |d|dƒd|dƒ}|jj ||j |d|jd|dƒd|dƒ}|||gfS(Nst%d_RGR˜R™RšiRXR›iRœRR,tsliceReitfiRitotmul0tmul1R^Rnt activation0Rž( RTRŸRƒR t SliceChannelRuR¢R„t _internalt_plust elemwise_mult_mulRr(RRR%R&R\R}R€RR‚RLR›Rœtgatest slice_gatestin_gatet forget_gatet in_transformtout_gatetnext_ctnext_h((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|ûs*     & . N( RwRxRyR RORRR—R|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR˜s9     cBsSeZdZdddddddd„Zdd„Zd„Zd„Zd„ZRS( s&Gated Rectified Unit (GRU) network cell. Note: this is an implementation of the cuDNN version of GRUs (slight modification compared to Cho et al. 2014). Each call computes the following function: .. math:: \begin{array}{ll} r_t = sigmoid(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ i_t = sigmoid(W_{ii} x_t + b_{ii} + W_hi h_{(t-1)} + b_{hi}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - i_t) * n_t + i_t * h_{(t-1)} \\ \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the hidden state of the previous layer at time `t` or :math:`input_t` for the first layer, and :math:`r_t`, :math:`i_t`, :math:`n_t` are the reset, input, and new gates, respectively. Parameters ---------- hidden_size : int Number of units in output symbol. i2h_weight_initializer : str or Initializer Initializer for the input weights matrix, used for the linear transformation of the inputs. h2h_weight_initializer : str or Initializer Initializer for the recurrent weights matrix, used for the linear transformation of the recurrent state. i2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. prefix : str, default 'gru_' prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None, default None Container for weight sharing between cells. Created if `None`. Inputs: - **data**: input tensor with shape `(batch_size, input_size)`. - **states**: a list of one initial recurrent state tensor with shape `(batch_size, num_hidden)`. Outputs: - **out**: output tensor with shape `(batch_size, num_hidden)`. - **next_states**: a list of one output recurrent state tensor with the same shape as `states`. R#ic Csótt|ƒjd|d|ƒ||_||_|jjddd||fd|dtƒ|_|jjddd||fd|dtƒ|_ |jjd dd|fd|dtƒ|_ |jjd dd|fd|dtƒ|_ dS( NRLRMR}R9iR~RR€RR‚( RNRRORƒR…RMRqR:R}R€RR‚( RRR†R‡RˆR‰RŠR‹RLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROIs     cCsi||jfd6dd6gS(NR9RŒR(Rƒ(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR]scCsdS(Ntgru((RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR`scCs^d}|jj}dj|dr,|dnd|dƒ}|jd|jjd||jS(Ns{name}({mapping})s {0} -> {1}iiRXR‘(R}R9R“R R”RwR•(RRR–R9R‘((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—cs  *c Cs d|j}|d} |jd|d|d|d|jdd|d ƒ} |jd| d|d|d|jdd|d ƒ} |j| d dd|d ƒ\} } } |j| d dd|d ƒ\}}} |j|j| |d|dƒddd|dƒ}|j|j| |d|dƒddd|dƒ}|j|j| |j|| d|dƒd|dƒddd|dƒ}|j|d|dƒ}|jj |j|j ||d|dƒ|d|dƒ|j|| d|dƒd|dƒ}||gfS(Nst%d_iRGR˜R™RšiRXR›RœR,t i2h_slicet h2h_sliceRRnRltr_acttplus1tz_actR¨tplus2Rjth_actt ones_like0tminus0R©tmul20Rž( RTRŸRƒR«RrR R®t ones_likeR¬R­t elemwise_sub(RRR%R&R\R}R€RR‚RLt prev_state_hR›Rœti2h_rti2h_zth2h_rth2h_zt reset_gatet update_gatet next_h_tmptonesR·((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|ks@      ( (   (  N( RwRxRyR RORRR—R|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRs2   cBs}eZdZd d d„Zd„Zd„Zdd„Zd„Zd„Z d dd d d „Z d „Z d „Z d „Z RS(s)Sequentially stacking multiple RNN cells.cCs#tt|ƒjd|d|ƒdS(NRLRM(RNRRO(RRRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRO—scCsnd}|jd|jjddjg|jjƒD]3\}}djd|dt|jƒdƒƒ^q1ƒƒS( Ns{name}( {modstr} )RXtmodstrs s ({i}): {m}Retmi(R“R”RwtjoinRUtitemsRR—(RRR–ReRÏ((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—šs cCs|j|ƒdS(sˆAppends a cell into the stack. Parameters ---------- cell : RecurrentCell The cell to add. N(tregister_child(RRR$((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pytadd sicCst|jjƒ|ƒS(N(RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRªscKs,|j stdƒ‚t|jjƒ|S(NsyAfter applying modifier cells (e.g. ZoneoutCell) the base cell cannot be called directly. Call the modifier cell instead.(RPR0RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR­s  cCsÒ|jd7_g}d}td„|jjƒDƒƒs@t‚x||jjƒD]k}t|tƒ slt‚t|jƒƒ}||||!}||7}|||ƒ\}}|j |ƒqPW|t |gƒfS(Niicss|]}t|tƒ VqdS(N(R!R (t.0R$((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pys ·s( RTtallRURVR0R!R R4RR[R(RRR&R\t next_statestpR$tnR^((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt__call__³s% R_c Cs|jƒt|||dƒ\}}}} t|jƒ} t||||| ƒ}d} g} x¦t|jjƒƒD]\} }t|jƒƒ}|| | |!}| |7} |j |d|d|d|d| | dkrádn|d|ƒ\}}| j |ƒqtW|| fS(NiR&RR>RciRH( RQRCR R4RUR(t enumerateRVRRitextend(RRR=R&RR>RcRHRhR%Rt num_cellsR×RÖReR$RØR\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRiÁs  !" cCs|jt|ƒS(N(RUtstr(RRRe((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt __getitem__ØscCs t|jƒS(N(R4RU(RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt__len__ÛscOs t‚dS(N(RW(RRR{R((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|ÞsN(RwRxRyR ROR—RÓRRRÙRiRÞRßR|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR•s        cBs}eZdZd d d„Zd„Zd„Zdd„Zd„Zd„Z d dd d d „Z d „Z d „Z d „Z RS(s/Sequentially stacking multiple HybridRNN cells.cCs#tt|ƒjd|d|ƒdS(NRLRM(RNRRO(RRRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROåscCsnd}|jd|jjddjg|jjƒD]3\}}djd|dt|jƒdƒƒ^q1ƒƒS( Ns{name}( {modstr} )RXRÎs s ({i}): {m}ReRÏi(R“R”RwRÐRURÑRR—(RRR–ReRÏ((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—ès cCs|j|ƒdS(sˆAppends a cell into the stack. Parameters ---------- cell : RecurrentCell The cell to add. N(RÒ(RRR$((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRÓîsicCst|jjƒ|ƒS(N(RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRøscKs,|j stdƒ‚t|jjƒ|S(NsyAfter applying modifier cells (e.g. ZoneoutCell) the base cell cannot be called directly. Call the modifier cell instead.(RPR0RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRûs  cCs¼|jd7_g}d}td„|jjƒDƒƒs@t‚xf|jjƒD]U}t|jƒƒ}||||!}||7}|||ƒ\}}|j|ƒqPW|t|gƒfS(Niicss|]}t|tƒ VqdS(N(R!R (RÔR$((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pys s( RTRÕRURVR0R4RR[R(RRR&R\RÖR×R$RØR^((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRÙs% R_c Cs|jƒt|||dƒ\}}}} t|jƒ} t||||| ƒ}d} g} x¦t|jjƒƒD]\} }t|jƒƒ}|| | |!}| |7} |j |d|d|d|d| | dkrádn|d|ƒ\}}| j |ƒqtW|| fS(NiR&RR>RciRH( RQRCR R4RUR(RÚRVRRiRÛ(RRR=R&RR>RcRHRhR%RRÜR×RÖReR$RØR\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRis  !" cCs|jt|ƒS(N(RURÝ(RRRe((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRÞ$scCs t|jƒS(N(R4RU(RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRß'scCs|j||ƒS(N(RÙ(RRR%R&R\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|*sN(RwRxRyR ROR—RÓRRRÙRiRÞRßR|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRãs       cBs\eZdZd d d d„Zd„Zdd„Zd„Zd„Zd dd d d„Z RS( s6Applies dropout on input. Parameters ---------- rate : float Percentage of elements to drop out, which is 1 - percentage to retain. axes : tuple of int, default () The axes on which dropout mask is shared. If empty, regular dropout is applied. Inputs: - **data**: input tensor with shape `(batch_size, size)`. - **states**: a list of recurrent state tensors. Outputs: - **out**: output tensor with shape `(batch_size, size)`. - **next_states**: returns input `states` directly. cCsJtt|ƒj||ƒt|tƒs4tdƒ‚||_||_dS(Nsrate must be a number(RNRROR!RR0t_ratet_axes(RRtratetaxesRLRM((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROBs cCs"d}|jd|jj|jS(Ns"{name}(rate={_rate}, axes={_axes})RX(R“R”RwR•(RRR–((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—HsicCsgS(N((RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRMscCsdS(Ntdropout((RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRPsc CsM|jdkrC|jd|d|jd|jdd|jƒ}n||fS(NiRGR×RãRXst%d_fwd(RàtDropoutRáRT(RRR%R&R\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|Ss!R_c CsŠ|jƒt||||ƒ\}}}}t|tƒrY|j|||rR|ngƒStt|ƒj||d|d|d|ddƒS(NRR>RcRH( RQRCR!RR|RNRRiR ( RRR=R&RR>RcRHRhR%((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRiYs !(N( RwRxRyR ROR—RRR|Ri(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR.s     cBsSeZdZd„Zed„ƒZdd„Zejd„Z d„Z d„Z RS(sBase class for modifier cells. A modifier cell takes a base cell, apply modifications on it (e.g. Zoneout), and returns a new cell. After applying modifiers the base cell should no longer be called directly. The modifier cell should be used instead. cCs_|j std|jƒ‚t|_tt|ƒjd|j|jƒddƒ||_ dS(Ns>Cell %s is already modified. One cell cannot be modified twiceRLRM( RPR0RXR:RNRRORLRR t base_cell(RRRæ((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROns   %cCs |jjS(N(RæRM(RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRMvsicCs|jj|ƒS(N(RæR(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRzscKsJ|j stdƒ‚t|j_|jjd||}t|j_|S(NsyAfter applying modifier cells (e.g. DropoutCell) the base cell cannot be called directly. Call the modifier cell instead.R(RPR0R3RæRR:(RRRRtbegin((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR}s     cCs t‚dS(N(RW(RRR%R&R\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|†scCs"d}|jd|jj|jS(Ns{name}({base_cell})RX(R“R”RwR•(RRR–((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—‰s( RwRxRyROtpropertyRMRR R#RR|R—(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRes   cBsAeZdZddd„Zd„Zd„Zd„Zd„ZRS(sApplies Zoneout on base cell.gcCswt|tƒ stdƒ‚t|tƒ sB|j sBtdƒ‚tt|ƒj|ƒ||_||_ d|_ dS(Ns€BidirectionalCell doesn't support zoneout since it doesn't support step. Please add ZoneoutCell to the cells underneath instead.spBidirectional SequentialRNNCell doesn't support zoneout. Please add ZoneoutCell to the cells underneath instead.( R!R R0Rt_bidirectionalRNR ROtzoneout_outputstzoneout_statesR t _prev_output(RRRæRêRë((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRO‘s    cCs"d}|jd|jj|jS(NsF{name}(p_out={zoneout_outputs}, p_state={zoneout_states}, {base_cell})RX(R“R”RwR•(RRR–((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—scCsdS(Ntzoneout((RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR¢scCs tt|ƒjƒd|_dS(N(RNR RQR Rì(RR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRQ¥scsý|j|j|j}}}|||ƒ\}}‡fd†} |j} | dkrhˆj|ƒ} n|dkr’ˆj| ||ƒ|| ƒn|} |dkrägt||ƒD]0\} } ˆj| || ƒ| | ƒ^q´n|}| |_| |fS(Ncsˆjˆj|ƒd|ƒS(NR×(RåRÃ(R×tlike(R%(sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyt¬sg(RæRêRëRìR t zeros_liketwhereRa(RRR%R&R\R$t p_outputstp_statest next_outputRÖtmaskt prev_outputRftnew_stold_s((R%sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|©s  0R (RwRxRyROR—RRQR|(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR s    cBs5eZdZd„Zd„Zddddd„ZRS(s§ Adds residual connection as described in Wu et al, 2016 (https://arxiv.org/abs/1609.08144). Output of the cell is output of the base cell plus input. cCstt|ƒj|ƒdS(N(RNR RO(RRRæ((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROÃscCsA|j||ƒ\}}|j||dd|jƒ}||fS(NRXst%d_fwd(RæR RT(RRR%R&R\Rf((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR|ÇsR_c Cs|jƒt|j_|jj|d|d|d|d|d|ƒ\}}t|j_|dkrst|tƒn|}t ||||ƒ\}} } } |dk rÄt | |||| |ƒ}n|rß| j ||ƒ}n4gt ||ƒD]\} } | j | | ƒ^qï}||fS(NR&RR>RcRH( RQR3RæRPRiR:R R!RRCRKR Ra(RRR=R&RR>RcRHRJR\R+R%RhRetj((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRiÌs     !  4N(RwRxRyROR|R Ri(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR ¼s    cBsVeZdZdd„Zd„Zd„Zdd„Zd„Zd dd d d „Z RS( s¸Bidirectional RNN cell. Parameters ---------- l_cell : RecurrentCell Cell for forward unrolling r_cell : RecurrentCell Cell for backward unrolling tbi_cCsLtt|ƒjddddƒ|j|dƒ|j|dƒ||_dS(NRLtRMtl_celltr_cell(RNR ROR RÒt_output_prefix(RRRüRýt output_prefix((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyROïscCstdƒ‚dS(Ns2Bidirectional cannot be stepped. Please use unroll(RW(RRR&R\((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRÙõscCs6d}|jd|jjd|jdd|jdƒS(Ns+{name}(forward={l_cell}, backward={r_cell})RXRüRý(R“R”RwRU(RRR–((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR—øs icCst|jjƒ|ƒS(N(RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRþscKs,|j stdƒ‚t|jjƒ|S(NsyAfter applying modifier cells (e.g. DropoutCell) the base cell cannot be called directly. Call the modifier cell instead.(RPR0RRURV(RRR((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRs  R_c Cs|jƒt|||tƒ\}}}} |dkrLtt|ƒƒ} nQ|j|jdd|Œd|dtƒ} t |j | ddd|dtƒƒ} t ||||| ƒ}|} |j j ƒ\} } | j|d|d| t| j| ƒƒ d |d |d |ƒ\}}| j|d| d| t| j| ƒƒd |d td |ƒ\}}|dkr}tt|ƒƒ}nW|j|jdd|Œd|dtddƒ}t |j |ddd|dtƒƒ}|dkr4t|tƒ}td|||ƒ\}}}}td|||ƒ\}}}}n|rw|jd||Œ}|j||d d dd|jƒ}nYgtt||ƒƒD]=\}\}}|j||d ddd|j|fƒ^q}|dk rút||||||ƒ}n||} || fS(NR+iRDRER,R-R&RR>RcRHtdimiRXs%soutis%st%d(RQRCR3R R6treversedtSequenceReverseR;R:RR7R(RURVRiR4RR!RtconcatRþRÚRaRK(RRR=R&RR>RcRHR+R%Rtreversed_inputsR\RüRýt l_outputstl_statest r_outputstr_statestreversed_r_outputsRhRJRetl_otr_o((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyRisZ !          ! V   N( RwRxRyRORÙR—RRR Ri(((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyR ås       N(#Ryt__all__RûR R tbaseRRRtblockRRtutilsRRtnnRRRR(R RCRKRRRRRRRRRR R R (((sX/usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet/gluon/rnn/rnn_cell.pyts8       ) Á e~NK7*-)