B _:@sddlZddlZddlmZmZmZddlmZddlZddl Z ddl m Z m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%dd l&m'Z'm(Z(dd l)m*Z*m+Z+dd l,m-Z-m.Z.m/Z/dd l0m1Z1ddl2Z2e23d Z4ddl5Z6e6j7j8dddkZ9dZ:e;e jgZ?e;e j@e jAgZBddZCddZDieD_EddZFddZGddZHGddde'ZIdS)N)iterkeys iteritemsStringIO)xrange)tictoc)minimizevalueTransformationFactory ComponentUIDBlock ConstraintConstraintListParamVarVarListSet ObjectiveSuffixBinaryNonNegativeIntegers) SolverFactory SolverStatusTerminationCondition ProblemFormat) phextension)SolverManager_PHPyro)SingletonPlugin implements)preprocess_block_constraintspreprocess_block_objectives)InvocationType%transmit_external_function_invocation/transmit_external_function_invocation_to_worker)NormalizedTermDiffConvergencez pyomo.pysp)Fc s|r|j|j}n |j|j}|d}|dk r8|St|_}|d|_|d|_ |d|_ |d|_ |d|_ t |dkstt|_ttd|_ttd|_|}tt|j}t|d |_}t|d d |_t|d dd |_t !|d |j|_"t !|d|j|_#|j rhx<D](} | $|j | %|jq:Wn &di} i|_'x:|D]2} t(|j| dd| } t !| )|| <qWfdd} t*|| d|_+}t,rtdd|j-t.d d dD}nt|j/t.d d d}t |dks*t|d}t !||_0t.t1fdd|Dt2d|_3d|krvt4t4j5d|_6t7r|8ni}t9||d|j3:|:t;d|S)N_interscenario_pluginepsilon cut_scale allow_slack enable_rho enable_cutsr)Zwithin) initializeT)Zdense)Zmutabler.zPHRHO_%sz PHWEIGHT_%scs|||dkS)Nr)bi)_param_sep_srcr/D/tmp/pip-unpacked-wheel-d4p3hk07/pyomo/pysp/plugins/interscenario.py_set_var_valueysz-get_modified_instance.._set_var_value)Zrulecss|]}|dVqdS)r%Nr/).0xr/r/r5 sz(get_modified_instance..)activeZ descend_intor'c3s|]}|dVqdS)r%Nr/)r7r1)r3r/r5r9s)exprZsensedual) direction)idMapz9InterScenario plugin: generated modified problem instance)<contains_bundlesZ_bundle_binding_instance_map_name _instances componentr r(popr)r*r+r,r-lenAssertionErrorrcutlistrr abs_int_varsrabs_binary_vars findRootNodelistrZ_variable_datasr STAGE1VARrseparation_variablesrfixed_variable_valuesweakrefrefrhoweightssetlbsetubfixlocal_stage1_varmapr Zfind_component_onr fixed_variables_constraint PYOMO_4_0Zall_component_datarZcomponent_data_objects original_objsumrseparation_objrZ IMPORT_EXPORTr<"FALLBACK_ON_BRUTE_FORCE_PREPROCESS preprocessr deactivater)ph scenario_treescenario_or_bundleoptionsmodelr0rootNodeZvar_idsZ_S1VidxZ _cuidBufferr1Z_cuidr6_conZ_orig_objective_mapr/)r2r3r4r5get_modified_instance:sp                     rgc Cst|tjkrFydtjt|<t||Sdtjt|<t||Si}|jj}tjt|rttd}tr~|j|ddn ||t r| ni}t |j|d|j |dd}|j j}|j j}|tjkr|tkrd} n|tkrd} nd } | rtd | fnDtr||n |j|x$|jjD]} |j|| || <q0Wtrd|j|ddd n|j|dd n&x$|jjD]} |j|| || <q~W|S) NFTzcore.relax_discrete)inplace)r>) warmstart INFEASIBLE NONOPTIMALzpResolving subproblem model with relaxed second-stage discrete variables failed (%s). Dual values not available.)rhundo)rm)idget_dual_valuesdiscrete_stage2_varsr(rVr rWapplyapply_tor[r\rsolvesolverstatustermination_conditionrok"_acceptable_termination_conditions"_infeasible_termination_conditionsloggerwarningload solutions load_fromrKr<) rtrbdualsrexfrmrfresultssstcstateZvaridr/r/r5rosT        rocstd}tr|j|ddn |||j}||j|j |j |j }|r|j }x2D] }| d|dqdWntr|ni}t||dt||dt} tj| zBy|j|dd} Wn*tdtd| YnXWdtjX| jj} | jj} | t j!krh| t"krhd} trZ|#| n |j$%| n| t&kr|d } d }nd } d }| r|rn"td | ftd| n2t'fdd|j(D}t)|j}t*+||f}| ,||j|r)teez@Exception raised solving the interscenario evaluation subproblemzSolver log: %srjrkz!!!!rlz????z@Solving the interscenario cut separation subproblem failed (%s).c3s*|]"}|t|t|ffVqdS)N)r )r7vid)_parr3r/r5r9=sz+solve_separation_problem..r)rhrm)rm).r rWrqrrr(rXr]rZactivaterMrLr+r)rRrSZunfixr[r\r rrpyutilibmiscsetup_redirectrsrzr{getvaluereset_redirectrtrurvrrwrxr|r}r~rydictrKr mathsqrtcloserT)rtrbfallbackr_blockr+r)rdrf output_bufferrrrranscutobjr/)rr3r5solve_separation_problems                     rc st|||}|j}|j|j|j}|jxD|D]<\} } tfddt| D} | dk r0|| dkq0Wx|D]} g} xt| dD]\} }|j }|j }|j }|j }||||k||| j |k||| j d|k|| |||k| |qW|t| tfddt| dDdkqvWt rt|ni}t||d|r |jj|dd}|jj}|jj}|tjkr|tkrtr||n |j||jtd dtDt|fSdSdS) Nc3sV|]N\}\}}t|td|krd|d|||dVqdS)r'r%N)absmax)r7r1r3r)r4r*r)r/r5r9js zadd_new_cuts..rr'c3s2|]*\}}|dkr|n d|VqdS)g?r'Nr/)r7rval)r4r/r5r9s)r>T)ricss |]\}}|t|fVqdS)N)r )r7_idZ_varr/r/r5r9s)rgr(r)r*rFrUrYraddrGrHZubappendr[r\r_solverrsrtrurvrrwrxrWr|r}r~rr rX)r^r_r`feasibility_cutsincumbent_cutsresolvemrZ_cutlistZcut_objrr;Z _int_binariesrrr0czyrfrrrr/)r4r*r)r5 add_new_cuts]sV            rc Ks>t|||f|}|j}|j}|j}|j} |r8|j} n|jg} td} | } | d| g} g}g}x|D]\}}d}x| D]}||krd}PqW|rt |t |kstxt|D]\}}|||<qWtr|ni}t||dtd|jft}tj|zDy|jj|dd}Wn*tdtd |YnXWdtjXtd ||jf|jj}|jj }|t!j"kr6|t#kr6d}t$r|%|n |j&'||( }|( d|)t*|+|( ||j,r |)t-|j|n |)d| )d q~dsF|t.krd }|)d|)d|j/rt0|j|d}|d kr|j1t2j3kr|t0| |d}nd}| )|td||jfq~d}|)d|)d| )dtd|ftd |q~W|( || | | 4||| fS)NipoptrFT)r>zpreprocessed scenario %s)rz@Exception raised solving the interscenario evaluation subproblemzSolver log: %sz3solved solution from scenario set %s on scenario %sz. r'z????zX zJsolved separation problem for solution from scenario set %s on scenario %sr%z? zrFdg)\$@gn$@gQ?g?g?g?g-C6?g{Gzd?)enableRhoUpdatesenableFeasibilityCutsenableIncumbentCutsr)r*allow_variable_slackiterationIntervalconvergenceRelativeDegredationconvergenceAbsoluteDegredationrecutThresholdrepeated_solution_threshholdrhoScale rhoDampingcutThreshold_minDiffcutThreshold_crossCutiteration0RecutBoundImprovement)selfr/r/r5__init__3s zInterScenarioPlugin.__init__cCsBd|_d|_d|_d|_g|_g|_d|_d|_t|_ g|_ dS)Nr) incumbentrP x_deviationlastConvergenceMetricrrlastRunaverage_solutionr$ convergerunique_scenario_solutions)rr^r/r/r5resetYszInterScenarioPlugin.resetcCs||dS)N)r)rr^r/r/r5pre_ph_initializationes z)InterScenarioPlugin.pre_ph_initializationcCs2|jr.|j}x|jD]}|||dqWdS)Nr)r_scenario_treerI_xbarssetRhoAllScenarios)rr^rcvr/r/r5post_instance_creationis   z*InterScenarioPlugin.post_instance_creationcCs^t|jjdkrtd|jtkr&dnd|_|jrZ|j}x|j D]}| ||dqDWdS)Nr%z5InterScenario plugin only works with 2-stage problemsr'r) rDrZ_stages RuntimeErrorZ_objective_senser _sense_to_minrrIrr)rr^rcrr/r/r5post_ph_initializationps  z*InterScenarioPlugin.post_ph_initializationcs||||d}x|jdkr|jr|d7}td|fg}x@t|jD]2\}tfdd|jD}|rP||dqPW| |dtd||||qWd|_ dS)Nrr'z5InterScenario plugin: PH iteration 0 re-solve pass %sc3s"|]}t|tkrdVqdS)r'N)typetuple)r7r)rr/r5r9sz>InterScenarioPlugin.post_iteration_0_solves..Tz3InterScenario plugin: distributed cuts to scenarios) "_collect_unique_scenario_solutionsr(rPrr enumeraterrYextend_distribute_cutsr)rr^countZ_stale_scenarios_solnZ_was_cutr/)rr5post_iteration_0_solvess$     z+InterScenarioPlugin.post_iteration_0_solvescCs(|j|j||j|j|j|_dS)N)rupdate_current_iterationrrA lastMetricr)rr^r/r/r5post_iteration_0s   z$InterScenarioPlugin.post_iteration_0cCs|js |jr||dS)N)rrr)rr^r/r/r5pre_iteration_k_solvess  z*InterScenarioPlugin.pre_iteration_k_solvesc CsD|j|j||j|j|j}|j}||}d}|||jkrPt dd}|||j kr|||j kr|t d||fd}|j|j |j krt dd}|jdkrt dd}np|jr"|j}x\t|jD]N\}}|j|} |j|} ||jkr| | |jkrt d||| | fd}PqW|r:|j|_ ||||_dS)NFzBInterScenario plugin: triggered by no change in scenario solutionsTzJInterScenario plugin: triggered by convergence degredation (%0.4f; %+0.4f)z2InterScenario plugin: triggered by iteration limitz1InterScenario plugin: triggered to initialize rhozUInterScenario plugin: triggered by variable divergence with rho==0 (%s: %s; [%s, %s]))rrrrrArrrrprintrrrrrPrrIrZ _maximumsZ _minimumsr)r() rr^currlastdeltarunrcrrP_max_minr/r/r5post_iteration_k_solvessL         z+InterScenarioPlugin.post_iteration_k_solvescCsdS)Nr/)rr^r/r/r5post_iteration_ksz$InterScenarioPlugin.post_iteration_kcCs||||dS)N)rr()rr^r/r/r5post_ph_executions  z%InterScenarioPlugin.post_ph_executionc sltd\}}}}||_xtjD]\}fdd|dD}tdtdd|Dtfdd|Djdkrd n d jd d d|Dd dd|Dd|dfq4WfddjD}fddjDtfddt|D} t |} t |} j dkrFd} d} n.j }| |t t | t |} dd| f} | _ td| | | | t d| | | fj r|_tfdd|D}tdd|D}jdkr.jj d|j|t|kr.| dks*| jkr.dS||\}}j}jdkrtdi_xt|D]\}}||j|<qjWnbj}xZt|D]N\}}j|rֈj|d|||j|7<n||j|<qWxBtt|D]2\}}|dkrt|nt|j|fqWj}jrhx&tjD]\}}|||qLWdS)Nz9InterScenario plugin: analyzing scenario dual informationcsg|]}j|qSr/)r get_scenario)r7r8)r^r/r5 sz=InterScenarioPlugin._interscenario_plugin..r'z Solution %2d: generated %2d cuts, cut by %2d other scenarios; objective %10s, scenario cost [%s], cut obj [%s] [generated by %s]css|]}t|tkrdVqdS)r'N)rr)r7rr/r/r5r9sz.c3s"|]}t|tkrdVqdS)r'N)rr)r7r)rr/r5r9sNonez%10.2fz, css|]}d|jVqdS)z%10.2fN)_cost)r7r8r/r/r5r9s css.|]&}t|tkrd|dnd|VqdS)z%5.2frz%5sN)rr)r7r8r/r/r5r9s,cs(g|] }|dD]}j|jqqS)r')rrr)r7sr8)r^r/r5rscs(g|] }|dD]}j|jqqS)r')rr _probability)r7rr8)r^r/r5rsc3s|]\}}||VqdS)Nr/)r7r1r) scenarioProbr/r5r9sz-----%z%+.2f%%rz7 Average scenario cost: %f (%s) Max-min: %f (%0.2f%%)gY@c3s$|]}tfdd|DVqdS)c3s,|]$}t|tkr|djkrdVqdS)rr'N)rrr)r7r8)rr/r5r9szFInterScenarioPlugin._interscenario_plugin...N)rY)r7r)rr/r5r9scss|]}t|VqdS)N)rD)r7rr/r/r5r9 sr)outerz&InterScenario plugin: initializing rho)r_solve_interscenario_solutions_compute_objectivefeasible_objectivesrrrrYjoinrminrrrr_update_incumbentr_update_reported_boundsrrDr_process_dual_informationrrPrrsortedrrIrr)rr^partial_obj_valuesrcuts probabilityZsoln _scenariosZ scenarioCostsZ_avgrrZ_del_avgZ _del_avg_str_prevZcutCountZsubProblemCountZnew_rhologinfoZ_scalerrZ_dampinglrcr/)rr^rrr5r(s          $        *   z)InterScenarioPlugin._interscenario_plugincCs*|j}g|_|j}x|jD]}t|j|j}x@t|j|jD],\}}||s`| |rDt t |||<qDWd}x0|jD]&} || dkr| d |jd}PqW|s|j ||jgfqWd} x6|D].\} } x$|jD]\} }| | kr| d7} PqWqWt d| t|jft| t|jS)NFrr'TzAInterscenario plugin: %s unchanged scenario solutions (out of %s))rrrIrr_xr@ris_variable_binaryis_variable_integerintroundrrrDfloat)rr^Z_old_unique_scenario_solutionsrcrZ _this_solrZ_valfoundZ_solZ _unchangedZ _old_solnZ _old_scenrZ_scenr/r/r5rSs6       z6InterScenarioPlugin._collect_unique_scenario_solutionscCsgggf}g}t|jt}g}|jr2|jj}n|jj}x|D]}||j|j f}|j |j |j |j |jd} |r||jjd|j|j|jtjjddd| |d q@t||j|f|| } x"t|D]\} } | | | qWq@W|rxd} t|}x|D]} | dg|qWx\| |krv|j}||}|j|} x"t|D]\} } | | | |<qPW| d7} qW||fS) N)r)r*r+r,r-invoke_external_functionTz pyomo.pysp.plugins.interscenarior) actionname queue_nameinvocation_typegenerateResponse module_name function_name function_kwds function_argsrr') isinstance_solver_managerrrr?_scenario_bundlesrrrrr)r*rrrqueuer@_phpyro_job_worker_mapr!SingleInvocationkeyrrrDrwait_anyindex get_results)rr^rr distributedZaction_handles subproblemsproblemraZ kwd_optionsZ_tmpr1rnum_results_so_far num_results_ahZ_ah_idr/r/r5r{sZ                z2InterScenarioPlugin._solve_interscenario_solutionsFcsd}tfddjD}|rJ|ttdjt|t|dndt|jt}|j rv|j j }dd}n|j j }dd}g}xD|D]:} g} xt jD]~\\} } d} x || D]}|j| krd } PqW| r| fd djDqjdkr| fd djDqW| sDjsD|dq|t| 7}|r||jjd | j|j| jtjjd d dd| j|fd qt||j | | j|}||td| j|rdndfqWtd|t|fg_jrtdtjfg_|rltdd|D}t|}x<||krj|j}||}|j|||<|d7}q0W|r|j }x|t |D]p\}} ||}|dkrqxN|| D]B}|d|_!t|dt|j"|jkst#|d|j"|j<qWqW|$dS)Nrc3s:|]2}|D](}t|tkr |djkr |dVq qdS)rN)rrr)r7r8r)rr/r5r9s  z7InterScenarioPlugin._distribute_cuts..r'cSs|jS)N)r)r8r/r/r5z6InterScenarioPlugin._distribute_cuts..cSs|gS)Nr/)r8r/r/r5r(r)FTc3s8|]0}t|tkr|djkr|VqdS)rN)rrr)r7r)rnrr/r5r9sc3s6|].}t|tkr|dkr|VqdS)rN)rr)r7r)allCutThresholdrnr/r5r9srz pyomo.pysp.plugins.interscenarior) rrrrrrrrrz!distributed cuts to scenario %s%sz and resolved scenariorjzIInterScenario plugin: added %d feasibility cuts from a library of %s cutsz-InterScenario plugin: added %d incumbent cutscss|]}|dkrdVqdS)Nr'r/)r7r8r/r/r5r9 s)%rrrr rrDrrrrr?rrrrr@rrrrrrr!rrrrrrYrr r!rIrrrEZupdate_variable_statistics)rr^rZ totalCutsZcutObjr"r#Z get_scenariosZresolvesr$rr8rr rrr%r&r'Z_ah_idxrcrr/)r*rnrr5rs                  z$InterScenarioPlugin._distribute_cutscCsng}xdtt|jD]R}d}x>t|D]2\}}|||dkrDd}P|||||7}q&W||qW|S)Ng)rrDrrr)rrrrsoln_idrZscen_or_bundle_idpr/r/r5r"sz&InterScenarioPlugin._compute_objectivec s^ddtjD}|s$tddStdtdd|Dftfdd|Dtdd \}}g}g}g}|j}xt|j d j |j D]^} | | rq| | r|| q|| r|| q|| rd std q|| qWjdksjd j|jkrjrpjrp|spjdd jtfd d|Dtfdd|Df|jj||f_|j|ddj} t| t| n2jd j|jkrdj} t| d}|sjsdSxb|D]Z\} } | |krqj| d jtfdd|Dtfdd|DfqWdS)NcSsg|]}|ddk r|qS)r'Nr/)r7or/r/r5r/sz9InterScenarioPlugin._update_incumbent..zAInterScenario plugin: No scenario solutions are globally feasiblez-InterScenario plugin: Feasible objectives: %scss|]}|dVqdS)r'Nr/)r7r-r/r/r5r97sz8InterScenarioPlugin._update_incumbent..c3s$|]}|dj|dfVqdS)rr'N)r)r7r8)rr/r5r9:sr')rrFZFIXMEc3s|]}|t|fVqdS)N)r )r7r)rr/r5r9Tsc3s|]}|t|fVqdS)N)r )r7r)rr/r5r9Us)innerz0InterScenario plugin: NEW incumbent: %s = %s, %sz,InterScenario plugin: incumbent: %s = %s, %src3s|]}|t|fVqdS)N)r )r7r)rr/r5r9rsc3s|]}|t|fVqdS)N)r )r7r)rr/r5r9ss)rrrrroperator itemgetterrrIrrr@Zis_variable_fixedrrr Zis_variable_semicontinuousrErrr)rrrrrrzinfo) rr^Z feasible_objZbest_idZbest_objZ binary_varsZ integer_varsZcontinuous_varsrcrmsgrr/)rrr5r.sh           z%InterScenarioPlugin._update_incumbentc* s,dgtj}xFtjD]8\}}x.|dD]"}|j|}|||j7<q.WqWt|} |jj} j dkrt fdd| D_ t dd| D} t dd| D} x.t|D] \}} j|d}t dd| D}d}xt|D]\}}|||dkrqx\t |||D]H\}}||t || ||||7<t| |t|| |<q$W||7}qW|rx|D]}|||<qWx..c3s|]}|dVqdS)rNr/)r7r)rr/r5r9sN)rrr)r7)r)rr5r9sz@InterScenarioPlugin._process_dual_information..css|]}|dfVqdS)gNr/)r7rr/r/r5r9scss|]}|dfVqdS)gNr/)r7rr/r/r5r9srcss|]}|dfVqdS)gNr/)r7rr/r/r5r9sg?css|]}|dkr|VqdS)rNr/)r7Z_rhor/r/r5r9sz<%4s: %6s [%7s, %7s] %7s; %6s [%6s, %6s] %6s; RHO %7s : %7s) z---ZDualrrstdevrrrr3Zcomputedfinalcss|]}t|VqdS)N)r)r7r8r/r/r5r9scss|]}|dVqdS)r%Nr/)r7r8r/r/r5r9sr%css|]}t|VqdS)N)r)r7r8r/r/r5r9scss|]}|dVqdS)r%Nr/)r7r8r/r/r5r9sgư>zQ%4d: %6.1f [%7.1f, %7.1f] %7.1f; %6.1f [%6.1f, %6.1f] %6.1f; RHO %7.2f : %%7.2f)rDrrrrrrYrIrrrrrcopysignrrrrr r)*rr^rrZ soln_probr+Z soln_infoZ src_scen_nameZsrc_scenZtotal_soln_probZxbarZmax_dualZ weighted_rhoZsoln_pr8Zavg_dualZp_totalZscen_idr,rdZx_devZvar_infokZ dual_infoZsidZscenario_resultsZsolutionZ_min_rhorrZd_minZd_maxZ_sumZ_sumsqnZd_avgZd_stdevZx_minZx_maxZx_avgZx_stdevr/)rr5rws     (  .   "     z-InterScenarioPlugin._process_dual_informationN)F)__name__ __module__ __qualname__rrZ IPHExtensionrrrrrrrrrrrr(rrrrrrr/r/r/r5r/s& & 1w(= j Ir)Jr/rsixrrrZ six.movesrrNrZpyutilib.misc.timingrrZ pyomo.corerr r r r r rrrrrrrrrZ pyomo.optrrrrZ pyomo.pysprZ%pyomo.solvers.plugins.smanager.phpyrorZpyomo.common.pluginrrZpyomo.repn.standard_repnrr Zpyomo.pysp.phsolverserverutilsr!r"r#Zpyomo.pysp.convergencer$logging getLoggerrzZ pyomo.versionZpyomoversion version_inforWr[setZoptimalZgloballyOptimalZlocallyOptimalrxZ infeasibleZinvalidProblemryrgrorprrrrr/r/r/r5 sD D      pGiG