ó šÄïYc@s dZddlZddlmZddlmZddlmZddl m Z ddl m Z d „Z d „Zd „Zd „Zd efd„ƒYZdS(sFExecutor group is a convenient tool for managing a group of executors.iÿÿÿÿN(t OrderedDicti(tcontext(tndarray(tDataDesc(t_split_input_slicec Cs¢x›t|||ƒD]‡\}}}t|tjƒrD|j|ƒqt|ttfƒr‰x>t||ƒD]\}}|j|ƒqiWqx|D]\}} |dkr‰|j} |jdkpÐ| ||j k} | ry|dkr||j|j !j| ƒq†|j | j krBtj |d|d|jd|j d| ƒq†tj |d|d|jd|j ƒ} | j| ƒq–|j| ƒq|j| ƒqWqWdS(s@Load a list of arrays into a list of arrays specified by slices.itaxistbegintendtoutN( tzipt isinstancetndtNDArraytcopytotlistttupletshapetstarttstopRt slice_axis( tdatattargetst major_axistd_srct d_targetsRtsrctdstt slice_idxtd_dstRtdo_cropt d_dst_copy((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt _load_generals*"  "  cCst|j||ƒdS(sLoad data into sliced arrays.N(RR(tbatchRR((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt _load_dataAscCst|j||ƒdS(sLoad label into sliced arrays.N(Rtlabel(R RR((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt _load_labelFsc Cs¯g}x¢t||ƒD]‘\}}|dkr–t|ƒdkrT|j|dƒq§|jtjd|g|D]}|j|djƒ^qmŒƒq|j|dƒqW|S(smMerge outputs that lives on multiple context into one, so that they look like living on one context. iitdim(R tlentappendR tconcatt as_in_contextR(toutputsRtretsttensorsRttensor((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt_merge_multi_contextKs 3tDataParallelExecutorGroupcBsãeZdZdedddd„Zd„Zd„Zded„Z d„Z ed„Z d„Z dd „Z d „Zed dd „Zed „Zddd„Zed„Zdd„Zd„Zd„Zd„Zd„ZRS(sC A group of executors that lives on a group of devices. This is a helper class used to implement data parallelization. Each mini-batch will be split and run on the devices. Parameters ---------- symbol : Symbol The common symbolic computation graph for all executors. contexts : list A list of contexts. workload : list If not ``None``, could be a list of numbers that specify the workload to be assigned to different context. Larger number indicate heavier workload. data_shapes : list Should be a list of (name, shape) tuples, for the shapes of data. Note the order is important and should be the same as the order that the `DataIter` provide the data. label_shapes : list Should be a list of (name, shape) tuples, for the shapes of label. Note the order is important and should be the same as the order that the `DataIter` provide the label. param_names : list A list of strings, indicating the names of parameters (e.g. weights, filters, etc.) in the computation graph. for_training : bool Indicate whether the executors should be bind for training. When not doing training, the memory for gradients will not be allocated. inputs_need_grad : bool Indicate whether the gradients for the input data should be computed. This is currently not used. It will be useful for implementing composition of modules. shared_group : DataParallelExecutorGroup Defaults to ``None``. This is used in bucketing. When not ``None``, it should be a executor group corresponding to a different bucket. In other words, it will correspond to a different symbol with the same set of parameters (e.g. unrolled RNNs with different lengths). In this case the memory regions of the parameters will be shared. logger : Logger Default is `logging`. fixed_param_names: list of str Parameters to be fixed during training. For these parameters, not gradients will be calculated and thus no space will be allocated for the gradient. grad_req : str, list of str, dict of str to str Requirement for gradient accumulation. Can be 'write', 'add', or 'null' (default to 'write'). Can be specified globally (str) or for each argument (list, dict). twritecCs?||_|jƒ|_|jƒ|_||_||_||_||_||_ | |_ d|_ | |_ |j dkrŠg|_ n| |_|jdkr®g|_n|s½d} ng|D]'}t|tƒrß|n t|Œ^qÄ}|dk r4g|D]'}t|tƒr|n t|Œ^q}ng|D]}|j^q;}t| tƒrëi|_x‹|jD]r}||jkr¬||j krœdn| |j|RCR@txRVtkt_RA((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt__init__sˆ             4 7 % ! %                    5cCsêt|ƒdkst‚g|D]}tj|jƒ^q}x¦t||ƒD]•\\}}}|dkrqqMn||}|jdk rÁ||jksâtdd|jd||fƒ‚qM||_t|j|j ƒ|_ qMW|S(sÝDecide the slices for each context according to the workload. Parameters ---------- data_shapes : list list of (name, shape) specifying the shapes for the input data or label. iiÿÿÿÿs(all data must have the same batch size: sbatch_size = %d, but s%s has shape %sN( R%RDRR\tlayoutR RIR?RR9RJ(RaRTRcRRARRRI((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt decide_slicesés%"  % cCs¥g|jD]H\}}gt|jƒD]&\}}|j||j|f^q&^q |_g|jD])}g|jD]}|j|^qu^qe|_|jdk rg|jD]H\}}gt|jƒD]&\}}|j||j|f^qÌ^q°|_ n d|_ gt|j ƒD]>\}}||j krg|jD]}|j |^qB^q|_|jrÇgt|j ƒD]>\}}||j kr}g|jD]}|j|^q¢^q}|_n d|_g|jD]}|d^qÚ}|jrPg|D]D}||j krg|jD]}|j|j j|ƒ^q^q|_n d|_gtt|jƒƒD])}g|jD]}|j|^q^qo|_dS(s'Collect internal arrays from executors.iN(RTt enumerateRKRJtarg_dictRMR@RPRUR?RNR4R2t arg_arraysROR:RQR;tindexRStrangeR%R6RR(RaRARetitetexec_RcRV((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt_collect_arrayss.X9[ ; >   T cCs‚|s|j st‚d|_|j|ƒ|_|dk rR|j|ƒ|_nxºtt|j ƒƒD]£}|j |||jƒ}|dk r­|j |||jƒ}ng}|ré|j |j dt t||ƒ|j|~scss$|]}|jtjƒƒVqdS(N(R RR€(RR‚((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pys sN( R R2ROtsumR%tastypetdtypeR R6RR(RaR{R|RAtblocktweight((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyt get_paramsos " %" cCs‡t||j|jƒ|dkr.|j}n|jdk r_|jr_t||j|jƒnx!|j D]}|j d|ƒqiWdS(s»Split `data_batch` according to workload and run forward on each devices. Parameters ---------- data_batch : DataBatch Or could be any object implementing similar interface. is_train : bool The hint for the backend, indicating whether we are during training phase. Default is `None`, then the value `self.for_training` will be used. Returns ------- tis_trainN( R!RMRXR?R:RNR"R#RYRKtforward(Rat data_batchR‰Rp((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyRŠ„s  cCs©|jdj}g|D]}|j^q}g}xpt|jjƒ||jƒD]P\}}}t|ƒ}|dkrˆ|j||Ts cSsi|]}|j|j“qS((R…RA(RRc((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pys Vs RRCt type_dicttshared_arg_namest shared_exect shared_buffers iýÿÿÿiN(R?RKR8RHRERFR7t simple_bindRCR2R=tintt debug_strtsplit( RaRnRTRURbR©RRHt input_shapest input_typestexecutor((s;build/bdist.linux-armv7l/egg/mxnet/module/executor_group.pyRvHs      2cCs•g}xˆt||ƒD]w\}}t|jƒ}|dkrb|j|j|j|j||s  "