3 ïN•bš ã@sTddlZddlmZddddgZGdd„deƒZdd„ZGd d„deƒZd d„ZdS) éN)Ú DataParallelÚCallbackContextÚexecute_replication_callbacksÚDataParallelWithCallbackÚpatch_replication_callbackc@s eZdZdS)rN)Ú__name__Ú __module__Ú __qualname__©r r úL/home/ec2-user/SageMaker/lama/models/ade20k/segm_lib/nn/modules/replicate.pyrscCsx|d}tt|jƒƒƒ}dd„t|ƒDƒ}xHt|ƒD]<\}}x2t|jƒƒD]"\}}t|dƒrJ|j|||ƒqJWq4WdS)aJ Execute an replication callback `__data_parallel_replicate__` on each module created by original replication. The callback will be invoked with arguments `__data_parallel_replicate__(ctx, copy_id)` Note that, as all modules are isomorphism, we assign each sub-module with a context (shared among multiple copies of this module on different devices). Through this context, different copies can share some information. We guarantee that the callback on the master copy (the first copy) will be called ahead of calling the callback of any slave copies. rcSsg|] }tƒ‘qSr )r)Ú.0Ú_r r r ú *sz1execute_replication_callbacks..Ú__data_parallel_replicate__N)ÚlenÚlistÚmodulesÚrangeÚ enumerateÚhasattrr)rZ master_copyZ nr_modulesZctxsÚiÚmoduleÚjÚmr r r rs  cs eZdZdZ‡fdd„Z‡ZS)ra Data Parallel with a replication callback. An replication callback `__data_parallel_replicate__` of each module will be invoked after being created by original `replicate` function. The callback will be invoked with arguments `__data_parallel_replicate__(ctx, copy_id)` Examples: > sync_bn = SynchronizedBatchNorm1d(10, eps=1e-5, affine=False) > sync_bn = DataParallelWithCallback(sync_bn, device_ids=[0, 1]) # sync_bn.__data_parallel_replicate__ will be invoked. cstt|ƒj||ƒ}t|ƒ|S)N)ÚsuperrÚ replicater)ÚselfrÚ device_idsr)Ú __class__r r r@sz"DataParallelWithCallback.replicate)rrr Ú__doc__rÚ __classcell__r r )rr r2s cs4t|tƒst‚|j‰tjˆƒ‡fdd„ƒ}||_dS)a  Monkey-patch an existing `DataParallel` object. Add the replication callback. Useful when you have customized `DataParallel` implementation. Examples: > sync_bn = SynchronizedBatchNorm1d(10, eps=1e-5, affine=False) > sync_bn = DataParallel(sync_bn, device_ids=[0, 1]) > patch_replication_callback(sync_bn) # this is equivalent to > sync_bn = SynchronizedBatchNorm1d(10, eps=1e-5, affine=False) > sync_bn = DataParallelWithCallback(sync_bn, device_ids=[0, 1]) csˆ||ƒ}t|ƒ|S)N)r)rrr)Ú old_replicater r Ú new_replicateXs z1patch_replication_callback..new_replicateN)Ú isinstancerÚAssertionErrorrÚ functoolsÚwraps)Ú data_parallelr"r )r!r rFs) r%Ztorch.nn.parallel.data_parallelrÚ__all__Úobjectrrrrr r r r Ú s