3 NbE6@sddlZddlZddlmZmZddlZddlZddl Z ddl j Z ddl j j ZddlmZddlmZddlmZmZddlmZddlmZmZddlmZmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%ej&e'Z(dd d Z)de j*e j*dddZ+dddZ,Gdddej-Z.dS)N)DictTuple)DistributedSampler)make_evaluator)make_default_train_dataloadermake_default_val_dataloader)make_discrim_loss)PerceptualLossResNetPL)make_generatormake_discriminator)make_visualizer)add_prefix_to_keys average_dictsset_requires_grad flatten_dictget_has_ddp_rankadamwcKs>|dkrtjj}n |dkr$tjj}ntd|||f|S)NadamrzUnknown optimizer kind )torchoptimAdamAdamW ValueError) parameterskindkwargsZoptimizer_classrF/home/ec2-user/SageMaker/lama/saicinpainting/training/trainers/base.pymake_optimizers   r+?)resultnew_iterate_modelc CsftjTt|j}t|j}x4|jD](}||jj|j||jd|dq,WWdQRXdS)N)alpha)rno_graddictnamed_parameterskeysdatamul_add_)r!r"decayZ res_params new_paramskrrrupdate_running_average"s    r/bilinearc Cs|j\}}}}||}}g} |d kr(dnd} xVt|D]J}tj|d|||jd} tj| ||f|| d} | j| |d}|d}q6Wtj| ddS) Nr1bicubicFr#)device)sizemode align_corners)dim)r1r2) shaperangerrandnr3F interpolateappendcat) Z base_tensorscales scale_mode batch_size_heightwidth cur_height cur_widthr!r6Z cur_sampleZcur_sample_scaledrrrmake_multiscale_noise+s   rHcseZdZddddddddfdd Zd d Zd d Zd dZd)ddZddZddZ ddZ d*ddZ d+ddZ e eejfe eejfdddZeeje eejffd d!d"Zeeje eejffd d#d$Zd%d&Zd'd(ZZS),BaseInpaintingTrainingModuleFdg+?i0u ) predict_onlyvisualize_each_itersaverage_generatorgenerator_avg_betaaverage_generator_start_stepaverage_generator_periodstore_discr_outputs_for_visc stj| | tjd|_t|fjj_|_tsNtjdj|sj jt fjj _ t fjj j_tfjj_tfjj_tfjj_tstjdj jjjdf} | rt| _tjfdd| D_ni_|_|_|_|_ d_!d_"| _#jj jdd d id d krbtj$d d _%jj jd dd idd krtj&d d _'jj j(j)d krt*_+jj jddd idd krt,fjj j-_.nd_.|_/tjddS)Nz(BaseInpaintingTrainingModule init calledz Generator zDiscriminator extra_valcsi|]}tfjj|qSr)rconfig evaluator).0r.)selfrr Wsz9BaseInpaintingTrainingModule.__init__..r#l1 weight_knownrnone) reductionmseweight resnet_plz&BaseInpaintingTrainingModule init done)0super__init__LOGGERinforTr generatoruse_ddprsave_hyperparametersr discriminatorrlosses adversarialadversarial_lossr visualizerrrU val_evaluatortest_evaluatorr)getlistextra_val_titlesnn ModuleDictextra_evaluatorsrNrOrPrQgenerator_averagelast_generator_averaging_steprRL1LossZloss_l1MSELossZloss_mse perceptualr^r loss_plr r_loss_resnet_plrM) rWrTrfrLrMrNrOrPrQrRargsrrS) __class__)rWrrb:sP   z%BaseInpaintingTrainingModule.__init__cCsDt|jj}tt|jjf|jjjdtt|f|jjjdgS)N) optimizer)rprhrr&rrerT optimizers)rWZdiscriminator_paramsrrrconfigure_optimizersusz1BaseInpaintingTrainingModule.configure_optimizerscCsJt|jjj}|jr6t|jj|jj|jjdd|d<t f|jjj}|S)NT) num_replicasrankshuffle ddp_kwargs) r&rTr)trainrftrainer num_nodes num_processes global_rankr)rWr dataloaderrrrtrain_dataloader|s z-BaseInpaintingTrainingModule.train_dataloadercsptf|jjjg}|jjjdk r8|tf|jjjg}n||}|jjjdfrl|fdd|jD7}|S)NrScsg|]}tf|qSr)r)rVr.)rSrr sz?BaseInpaintingTrainingModule.val_dataloader..)rrTr)valZ visual_testrorq)rWresr)rSrval_dataloadersz+BaseInpaintingTrainingModule.val_dataloaderNcCsd|_|j||d|dS)NTr)r5 optimizer_idx)_is_training_step_do_step)rWbatch batch_idxrrrr training_stepsz*BaseInpaintingTrainingModule.training_stepcCsJd}|dkrd}n |dkr d}nd}|j|d}d|_|j||||dS) Nrrr#testrSr7F)r5 extra_val_key)rqrr)rWrrdataloader_idxrr5rrrvalidation_stepsz,BaseInpaintingTrainingModule.validation_stepcCs|jr`|jr`|j|jkr`|j|j|jkr`|jdkrDtj|j |_nt |j|j |j d|j|_t j |drz|djnt j|djjd}dd|djD}|j|ddd|S) N)r,lossTcSsi|]\}}|j|qSr)mean)rVr.vrrrrXszBBaseInpaintingTrainingModule.training_step_end..log_infoF)on_stepon_epoch)trainingrN global_steprPrvrQrucopydeepcopyrer/rOr is_tensorrtensorfloatrequires_grad_itemslog_dict)rWZbatch_parts_outputsZ full_lossrrrrtraining_step_ends   z.BaseInpaintingTrainingModule.training_step_endc s>dd|D}tdd|D}|jdd|jDtjddtjd d d d|D}|jj|d }tj|jd j d}|j d dddt j d|j d|jd|x*t|jD]\}}|jd||qWdd|D}|jj|d } tj| jd j d} | j d dddt j d|j d|jd| x,t| jD]\}}|jd||qNW|jr:x|jjD]\} } d| dfdd|D} | j| d }tj|jd j d}|j d dddt j d| d|j d|jd|x2t|jD]"\}}|jd| d||qWqWdS)NcSsg|]}|D]}|q qSrr)rVZ out_groupstep_outrrrrszEBaseInpaintingTrainingModule.validation_epoch_end..css|]}|dVqdS)rNr)rVrrrr szDBaseInpaintingTrainingModule.validation_epoch_end..cSsi|]\}}|j|qSr)r)rVr.rrrrrXszEBaseInpaintingTrainingModule.validation_epoch_end..zdisplay.max_columnsiz display.widthicSsg|]}d|kr|dqS)val_evaluator_stater)rVsrrrrs)statesr#rallT)axishowinplacez Validation metrics after epoch #z, total z iterations: Zval_cSsg|]}d|kr|dqS)test_evaluator_stater)rVrrrrrszTest metrics after epoch #test_ extra_val__evaluator_statecsg|]}|kr|qSrr)rVr) cur_state_keyrrrsz Extra val z metrics after epoch #rC)rrrpd set_optionrmevaluation_end DataFramestackunstackdropnarcrd current_epochrrlogrnrt)rWoutputsZ averaged_logsZval_evaluator_statesZval_evaluator_resZval_evaluator_res_dfr.rZtest_evaluator_statesZtest_evaluator_resZtest_evaluator_res_dfZcur_eval_titleZ cur_evaluator cur_statesZcur_evaluator_resZcur_evaluator_res_dfr)rrvalidation_epoch_ends:      &z1BaseInpaintingTrainingModule.validation_epoch_endrc Cs|dkr"t|jdt|jdn |dkrBt|jdt|jd||}d}i}|dksb|dkrr|j|\}}n.|dks|dkr|jjjjdkr|j|\}}|j dko||j dks|dkr.|jjjjdkr|j rt j |j|WdQRXd|}|dkr|d|7}|j|j|||d|d} |dkrP| |d7} t|t|| d } |d kr~|jj|| d <nB|dkr|jj|| d <n&|dkr|j|j|| d |d<| S)NrTFr#rrCrS)suffix)rrrrrrr)Nr)rrerhgenerator_lossrTrirjr^discriminator_loss get_ddp_rankrMrRrr%store_discr_outputsrlrr&rrm process_batchrnrt) rWrrr5rr total_lossmetricsZ vis_suffixZmetrics_prefixr!rrrrsD   $        z%BaseInpaintingTrainingModule._do_stepcCs*| r$|j r$|jr$|jdk r$|jS|jS)N)rrNrure)rWZ no_averagerrrget_current_generator sz2BaseInpaintingTrainingModule.get_current_generator)rreturncCs tdS)zUPass data through generator and obtain at leas 'predicted_image' and 'inpainted' keysN)NotImplementedError)rWrrrrforwardsz$BaseInpaintingTrainingModule.forward)rcCs tdS)N)r)rWrrrrrsz+BaseInpaintingTrainingModule.generator_losscCs tdS)N)r)rWrrrrrsz/BaseInpaintingTrainingModule.discriminator_losscCsv|djdd}|j|d\}}|j|d\}}tj||dd|d<tj||dd|d<|d|d|d<dS) Nimager7predicted_imagenearest)r4r5Zdiscr_output_realZdiscr_output_fakeZdiscr_output_diff)r9rhr<r=)rWrout_sizeZdiscr_real_outrCZdiscr_fake_outrrrrs z0BaseInpaintingTrainingModule.store_discr_outputscCs |jj|jjdkr|jjSdS)Nr#)rrrr)rWrrrr"sz)BaseInpaintingTrainingModule.get_ddp_rank)N)rNN)F)__name__ __module__ __qualname__rbrrrrrrrrrrstrrTensorrrrrrr __classcell__rr)r}rrI9s"9   , + $""rI)r)r )r0r1)/rloggingtypingrrpandasrpytorch_lightningZptlrtorch.nnrrtorch.nn.functional functionalr<torch.utils.datarZsaicinpainting.evaluationr%saicinpainting.training.data.datasetsrrZ*saicinpainting.training.losses.adversarialr)saicinpainting.training.losses.perceptualr r Zsaicinpainting.training.modulesr r Z#saicinpainting.training.visualizersr saicinpainting.utilsrrrrr getLoggerrrcrModuler/rHLightningModulerIrrrrs(