B _@s`ddlZddlZddlZddlZddlZddlZddlmZddlZ ddl Z ddl m Z ddl m Z mZmZddlTddlmZddlmZddlmZmZdd lmZmZmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-Gddde Z.Gddde#Z/ddZ0ddZ1edddddZ2dS)N) OptionParser)PauseGC) TaskWorkerTaskWorkerServershutdown_pyro_components)*) UndefinedData) pyomo_command)ExtensionPointSingletonPlugin) SolverFactoryTerminationConditionSolutionStatus)IPHSolverServerExtension)ScenarioTreeInstanceFactory) TransmitTypeInvocationType)_PHBase)reset_nonconverged_variablesreset_stage_cost_variablesfind_active_objectiveextract_solve_times)launch_command)iterkeys iteritemsc@seZdZddZddZdS) PHPyroWorkercKsXd|d<dttf|d<tj|f||j|_d|_d|_ i|_ d|_ i|_ dS)NzPH Pyro ServerZ caller_namezPySPWorker_%d@%snameT) osgetpidsocket gethostnamer__init__ WORKERNAMEtypeblocktimeout_phsolverserver_mapZ_bulk_task_collection_modules_imported)selfkwdsr*=/tmp/pip-unpacked-wheel-d4p3hk07/pyomo/pysp/phsolverserver.pyr!1szPHPyroWorker.__init__c Cstjjf|}d}|jdkr*|jt=d}n|jdkrrt|j|j|j<|j |j|j_ |j|_|j|j |}nN|jdkrt d|j j |jdd|_n$t}|j|j |}WdQRX|S)NreleaseT initializeshutdownzReceived shutdown request)r#)pyutilibmiscZBunchactionr&r_PHSolverServerr' object_namer"processprintZ dispatcherZ clear_queuer#Z_worker_shutdownr)r(dataresultZpgcr*r*r+r4As"   zPHPyroWorker.processN)__name__ __module__ __qualname__r!r4r*r*r*r+r/src@seZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)S)*r2cKs*t|d|_d|_tt|_||_dS)NT)rr! _first_solve_uncompressed_scenario_treer r _ph_pluginsr')r(Zmodules_importedr)r*r*r+r!`s   z_PHSolverServer.__init__cCs>|jrtdt|tddt|j|jtjD}|S)Nz8Received request to collect warmstart data for scenario=css|]\}}||jfVqdS)N)value).0symbolZvardatar*r*r+ wsz4_PHSolverServer.collect_warmstart..) _verboser5strdictr _instancesZ_PHInstanceSymbolMapsVarZbySymbol)r( scenario_namer7r*r*r+collect_warmstartqs  z!_PHSolverServer.collect_warmstartcCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzGReceived request to activate PH objective weight terms for scenario(s)=Fz*PH solver server has not been initialized!z$Activating PH objective weight terms) rBr5rClistrrE _initialized RuntimeErrorr"activate_ph_objective_weight_terms)r(r*r*r+rLs  z2_PHSolverServer.activate_ph_objective_weight_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzIReceived request to deactivate PH objective weight terms for scenario(s)=Fz*PH solver server has not been initialized!z&Deactivating PH objective weight terms) rBr5rCrIrrErJrKr$deactivate_ph_objective_weight_terms)r(r*r*r+rMs  z4_PHSolverServer.deactivate_ph_objective_weight_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzIReceived request to activate PH objective proximal terms for scenario(s)=Fz*PH solver server has not been initialized!z&Activating PH objective proximal terms) rBr5rCrIrrErJrKr$activate_ph_objective_proximal_terms)r(r*r*r+rNs  z4_PHSolverServer.activate_ph_objective_proximal_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzKReceived request to deactivate PH objective proximal terms for scenario(s)=Fz*PH solver server has not been initialized!z(Deactivating PH objective proximal terms) rBr5rCrIrrErJrKr&deactivate_ph_objective_proximal_terms)r(r*r*r+rOs  z6_PHSolverServer.deactivate_ph_objective_proximal_termsc Csi}i}|d<xp|dD]d}|j|}i}||<|j|d<|j|d<|j|d<t|j|d<t|j|d<t|j|d<qWi}|d<xF|dD]:}|j |} i} ||<| j | d <| j | d <| j | d <qW|S) NZnodes _variable_ids_standard_variable_ids_variable_indices_integer_binary_semicontinuousZ scenarios_objective_name_objective_sense_rho) _scenario_treeget_noderPrQrRrIrSrTrU get_scenariorVrWrX) r(tree_object_namesr6Z node_data node_name tree_nodeZthis_node_dataZ scenario_datarGscenarioZthis_scenario_datar*r*r+collect_scenario_tree_datas&           z*_PHSolverServer.collect_scenario_tree_datacCsddlm}|rtdtdtd|tdt|td|td||dkrhtd ||dkrtd t|| dkrtd t| td t| |jrtd |jrtdx|jD]}||qW||_ ||_| |_ | |_ | |_ | |_ ||_||_||_||_|jr.td|t||jd}|j|<|dkr`td|dtj|srt|dkstj|stt|||_}|jd||| |jd|_|jdkrtdg}|jr$|j|dkrtd|d|jrtd||j|}|j }n |!||jj}d|j_t"#|j|_$||j_|j%||jjj&|j||jd}|jj'||j dd|jj(dj |_ |)|jrtdx|jD]}|*|qW|+t,t-f}|.||/t01||2t03||jr"|4|jr2tdt5||r|jrd|j6|j7|d|j8dn|j6|j9|j:d|j8d|jrtdx|jD]}|;|qWd|_dS) Nr)PersistentSolverz/Received request to initialize PH solver serverzModel source: zScenario Tree source: z Solver type: zScenario or bundle name: z$Scenario tree bundle specification: zCreate random bundles: zScenario tree random seed: z$Linearize non-binary penalty terms: z4PH solver servers cannot currently be re-initializedz2Invoking pre-initialization PHSolverServer pluginszConstructing solver type=) solver_iozUnknown solver type=z specified)Zdownsample_fractionZbundlesZrandom_bundlesZ random_seedverbosezFUnable to launch PH solver server - scenario tree construction failed.FzBundle=z does not exist.zLoading scenarios for bundle=)compile_scenario_instancesrdT)Zcreate_variable_idsz6Invoking post-instance-creation PHSolverServer pluginsz0Setting instance for persistent solver interface)symbolic_solver_labelsoutput_fixed_variable_boundsz3Invoking post-initialization PHSolverServer plugins)t|r8tdd|jjddD}nt}|jr|j|}i}xT|jD](}|j |}|j |d| ||<q`Wn |j |}|j |d| }|S)Ncss|] }|jVqdS)N)r)r?sr*r*r+rAsz2_PHSolverServer.collect_results..)stages) rTransmitAllStagesTransmitNonLeafStagessetrYrrprrrsr[update_solution_from_instance copy_solution)r(r3Z results_flagsstages_to_loadbundleresultsrGr_r*r*r+collect_resultss           z_PHSolverServer.collect_resultsc (Csddlm} |jdkrtd|jr.d} nd} |jrHtd| |f|jrp|j|std|dSn|j |std |dS||j krtd | |f|j |} ||_ ||_ ||_ ||_||_||_| |_|jr |jrtd xF|jD]} | |qWn,|jrtd x|jD]} | |q$WxBt|jD]4\}}|jrltd |dt||| j|<qDW|jdkr||jdkr|||js|jdkrt|j|jt|j|jnt|j|jt | | rd|j ||jd}nd|j ||j |j |jd}| !r>|j|d<d}t"#| st"$| rzt%dd|jj&ddD}nt%}d}d}|jrr|j'|}t((}| j)|f|}t((|}t*|j+dks|+dj,t-j.ks|j/j0t1j.kr|jr|2td|dd}q|jr0td||j3rTtdt4t|j2dd|j5}|j6j7||j |j8|j d|jrtd|i}x,|jj9D] }|j:|d|;||j<<qWi}|j'|}x|jj9D]}|j|j<}i}xt|jD]j}i}t=||}xL|j>t?ddD]:} i}!x$t| D]\}"}#|@|#|!|"<q$W|!|| jA<qW|||<qW|||j<<qWn|jjB|}|j|}t((}| j)|f|}t((|}t*|j+dks|+dj,t-j.ks|j/j0t1j.kr|jr|2td |dd}n|jrtd!||j3r>td"|t|j2dd|j5}|j6j7||j |j8|j d|j:|d|;}|jrtd#|i}xz|jD]p}i}t=||}xRt|jCt?ddD]<\}$} i}!x$t| D]\}"}#|@|#|!|"<qW|!||$<qW|||<qW|s||f|jD|<i}%|+d}&tE|&d$rZt |&jFtGsZ|&jFdk rZ|&jF|%d$<tH|dd%\|%d&<|%d'<|&j,jI|%d(<|||%f}'nd)}'|jdkr|jrtd*xH|jD]} | J|qWn,|jrtd+x|jD]} | K|qWd|_|'S),Nr)raFz*PH solver server has not been initialized!rr}z(Received request to solve scenario %s=%szRPH solver server requested to solve bundle=%s, but that bundle is not recognized.zVPH solver server requested to solve scenario=%s, but that scenario is not recognized.z,No solver plugin available for scenario%s=%sz5Invoking pre-iteration-0-solve PHSolverServer pluginsz5Invoking pre-iteration-k-solve PHSolverServer pluginszProcessing solver option=z, value=T)load_solutionstee keepfilessuffixes)rrrrfrgr warmstartcss|] }|jVqdS)N)r)r?rr*r*r+rAbsz(_PHSolverServer.solve..rzSolve failed for bundle=z; no solutions generatedz$Successfully solved scenario bundle=zResults for scenario bundle=%s:)num)Z&allow_consistent_values_for_fixed_varsZ#comparison_tolerance_for_fixed_varsZignore_fixed_varsz(Successfully loaded solution for bundle=)r)activezSolve failed for scenario=z&Successfully solved scenario instance=z!Results for scenario instance=%s:z*Successfully loaded solution for scenario=gap)defaultZ solve_timepyomo_solve_timeZsolution_statusr*z6Invoking post-iteration-0-solve PHSolverServer pluginsz6Invoking post-iteration-k-solve PHSolverServer plugins)LrhrarJrKrYrprBr5Z is_bundleZ is_scenariork_teeZ_symbolic_solver_labelsr|Z_solver_optionsZ_solver_suffixesZ _warmstartZ_variable_transmissionr;r=Zpre_iteration_0_solveZpre_iteration_k_solverrCoptionsriZ(form_ph_linearized_objective_constraintsZ'_reset_instance_linearization_variablesZ_preprocess_scenario_instancesrrErrzZwarm_start_capablerrrrrr{timesolvelenZsolutionstatusrZ infeasibleZsolverZtermination_conditionr writerjZ bundle_nameZ_smapZ solutionsZ load_fromZ$_comparison_tolerance_for_fixed_varsrxrr_namegetattrZcomponent_objectsZ Constraintgetr _scenario_mapZ component_mapZ_solver_resultshasattrrrrkeyZpost_iteration_0_solveZpost_iteration_k_solve)(r(r3rrrfrgsolver_optionssolver_suffixesrvariable_transmissionraZ object_typerrrr>Zcommon_solve_kwdsrZfailurerbundle_ef_instanceZsolve_start_timerZ results_smZvariable_valuesr_Z suffix_valuesscenario_instanceZthis_scenario_suffix_values suffix_nameZthis_suffix_mapsuffix constraintZthis_constraint_suffix_mapindexZconstraint_dataZconstraint_nameZauxilliary_valuesZ solution0Zsolve_method_resultr*r*r+rsr                                                        z_PHSolverServer.solvecCsr|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|jj|}|j |qJWdS)NTz,Received request to update xbars for bundle=z.Received request to update xbars for scenario=Fz*PH solver server has not been initialized!) rBrYrpr5rJrKr_tree_node_mapZ_xbarsupdate)r(r3 new_xbarsr]Z node_xbarsr^r*r*r+ update_xbars+s   z_PHSolverServer.update_xbarscCsp|jrtd||jdkr$td||jkr:tddS|jj|}x$t|D]\}}|j| |qPWdS)Nz0Received request to update weights for scenario=Fz*PH solver server has not been initialized!zcERROR: Received request to update weights for instance not in PH solver server instance collection!) rBr5rJrKrErYrr_wr)r(rG new_weightsr_tree_node_nameZtree_node_weightsr*r*r+update_weights=s    z_PHSolverServer.update_weightscCs|jrtd||jdkr$td||jkrPtd|dt|jdS|jj|}x$t |D]\}}|j | |qfWdS)Nz-Received request to update rhos for scenario=Fz*PH solver server has not been initialized!z4ERROR: Received request to update rhos for scenario=z4, which is not in the PH solver server instance set=) rBr5rJrKrErCkeysrYrrrXr)r(rGnew_rhosr_rZtree_node_rhosr*r*r+ update_rhosPs    z_PHSolverServer.update_rhosc Cs|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|jj|j}| |qJWx,t|D] \}}|jj|j }| |qxWdS)NTz;Received request to update tree node statistics for bundle=z=Received request to update tree node statistics for scenario=Fz*PH solver server has not been initialized!) rBrYrpr5rJrKrrZ _minimumsrZ _maximums) r(rGZnew_node_minimumsZnew_node_maximumsrZtree_node_minimumsZthis_tree_node_minimumsZtree_node_maximumsZthis_tree_node_maximumsr*r*r+update_tree_node_statisticsds  z+_PHSolverServer.update_tree_node_statisticscCs|jr>|jdkr*td|d|ntd|d||jdkrPtd|jdkr~|j|}||ttj dnR||j krtd|d|d t |j dS|j |}||ttj ddS) NTz)Received request to define import suffix=z for bundle=z for scenario=Fz*PH solver server has not been initialized!) directionz0ERROR: Received request to define import suffix=z?, which is not in the collection of PH solver server instances=) rBrYrpr5rJrKr{Z add_componentryZIMPORTrErCr)r(r3rrrr*r*r+define_import_suffixs&   $ z$_PHSolverServer.define_import_suffixcstt|}jrTjr8td|d|d|ntd|d|d|jdkrftdd}jrjj|}n jj |}|j krj |}n0|t j krt j |}nt jj|dd}|j |<|} d} t|| s"d | kr|d d \} } t|| s"td |d |dkr0d dkr>id} |tjkr~jrl|jj|f} n|jj |f} n|tjks|tjkrjstdtjj} nh|tjks|tjkrtjj } nB|tjks|tjkrtjj} ntd|ddtjDft|| | dk r>t| |tjkrh| \} } j| fS|tjks|tjks|tjkr}x(| D] \} } j| f|}qW|Stfdd| DSdS)Nz-Received request to invoke external function=z in module=z for bundle=z for scenario=Fz*PH solver server has not been initialized!T) clear_cache.rz Function=z is not present in module=r*z[Received request for bundle invocation type but the scenario tree does not contain bundles.zsz<_PHSolverServer.invoke_external_function..c3s,|]$\}}|j|ffVqdS)N)rY)r? call_name call_object)function function_args function_kwdsr(r*r+rAsz;_PHSolverServer.invoke_external_function..)rrrBrYrpr5rJrKZ_scenario_bundle_maprr'sysmodulesr/r0 import_filersplitZSingleInvocationZPerBundleInvocationZPerBundleChainedInvocationrlrZPerScenarioInvocationZPerScenarioChainedInvocationZPerNodeInvocationZPerNodeChainedInvocationrZ_valuesrD)r(r3invocation_type module_name function_namerrZscenario_tree_objectZ this_moduleZmodule_attrnamesubnameZ call_objectsrrr7r*)rrrr(r+invoke_external_functions                                z(_PHSolverServer.invoke_external_functioncCs|jr>|jdkr"td|n td|tdt||jdkrPtd|jdkr|j|dkrtd|dn||jkrt|d t |||dS) NTz7Received request to restore cached solution for bundle=z9Received request to restore cached solution for scenario=zRestoring from cache id: Fz*PH solver server has not been initialized!zBundle=z does not exist.z3 is not in the PH solver server instance collection) rBrYrpr5rCrJrKrqrErrestoreCachedSolutions)r(r3cache_id release_cacher*r*r+rs    z&_PHSolverServer.restoreCachedSolutionscCs|jr>|jdkr"td|n td|tdt||jdkrPtd|jdkr|j|dkrtd|dn||jkrt|d t ||dS) NTz.Received request to cache solution for bundle=z0Received request to cache solution for scenario=zCaching with id: Fz*PH solver server has not been initialized!zBundle=z does not exist.z3 is not in the PH solver server instance collection) rBrYrpr5rCrJrKrqrErcacheSolutions)r(r3rr*r*r+r3s    z_PHSolverServer.cacheSolutionscCsz|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|j|}|j |qJW| dS)NTz6Received request to update fixed variables for bundle=z8Received request to update fixed variables for scenario=Fz*PH solver server has not been initialized!) rBrYrpr5rJrKrrZZ _fix_queuer_push_fix_queue_to_instances)r(r3fixed_variablesr]Znode_fixed_varsr^r*r*r+update_fixed_variablesMs   z&_PHSolverServer.update_fixed_variablesc Csd}|jdkr^||j|j|j|j|j|j|j|j |j |j |j |j |j|j|j|j|j}n|jdkr|||j|j}nv|jdkrNd}d}d}x|s0y<|d7}||j|j|j|j|j|j|j|j|j }d}Wqtj j!k r,}z0t"d |j|f||krt"d ||Wdd}~XYqXqW|dkrt"d |jn|jd krj|#d}n|jd kr|$d}nl|jdkr|%d}nP|jdkr|&d}n4|jdkr(|j'(dkrx|j?|j@|jA}n|jdkr|B|j|jC|jDd}nl|jdkr|E|j|jCd}nJ|jdkr|F|jG}n0|jdkr|H|jI}ntJdtK|jd|S)Nr-rrrFrTz0Solve failed for object=%s - this was attempt=%dzVAborting PH solver server - the maximum number of solve attempts=%d have been executedz6Successfully recovered from failed solve for object=%srNrOrLrMZ load_rhosrZ load_weightsZ load_xbarsZload_tree_node_statsrrZ!restore_cached_scenario_solutionsZcache_scenario_solutionsr`rHzERROR: Unknown action=z received by PH solver server)Lr1r-r~rr3rrrcrrrrrrrrrrdrerrZ var_configrrrrfrgrrrrr/commonZApplicationErrorr5rNrOrLrMrYrprrErrZ_push_rho_to_instancesrrrrrZ_push_w_to_instancesrrZ_push_xbar_to_instancesrZnew_minsZnew_maxsrrrrrrrrrrrrr`r\rHrGrKrC) r(r6r7Zmax_num_attemptsZattempts_so_farZsuccessful_solveexcrGrr*r*r+r4`s                           z_PHSolverServer.processN)r8r9r:r!rHrLrMrNrOr`r-rrrrrrrrrrrr4r*r*r*r+r2^s. gM%qr2cCst}|jdddddd|jddd d d d d |jdddddd|jdddddd|jdddddgd |jddd ddd|jddd dd dd ||_|S)Nz --verbosezPGenerate verbose output for both initialization and execution. Default is False. store_truerdF)helpr1destrz --profilezjEnable profiling of Python code. The value of this option is the number of functions that are summarized.storeprofileintr)rr1rr#rz --disable-gcz7Disable the python garbage collecter. Default is False. disable_gcz --tracebackzkWhen an exception is thrown, show the entire call stack. Ignored if profiling is enabled. Default is False. tracebackz--user-defined-extensionzVThe name of a python module specifying a user-defined PHSolverServer extension plugin.rtuser_defined_extensionsstringz --pyro-hostz=The hostname to bind on when searching for a Pyro nameserver. pyro_hostz --pyro-portz9The port to bind on when searching for a Pyro nameserver. pyro_port)r add_optionusage)Z usage_stringparserr*r*r+construct_options_parsersRrc Cstt}x|D] }|qWt|jdkrHx|jD] }|tjkrZtd|dnFtd|ttj }tj ddt j |td|tj dd<ddl}|}|dr|d}|dd krt|dd }xd|tj||jD]L\}}t|tr|d krtt}x(|d d D]}t||r"|q"WqWq8Wytt|j|jd Wn(tdt|j|jddYnXdS)Nrz-User-defined PHSolverServer extension module=z already imported - skippingz>Trying to import user-defined PHSolverServer extension module=rzModule successfully loadedz.py/rr T)all)hostportz3PH solver server aborted. Sending shutdown request.)rrZ num_retries)r rdisablerrrrr5rIrminsertr/r0rinspectrfindrstripfindrr getmembersisclass issubclassr rzenablerrrrr) rZph_extension_pointrZthis_extensionZ original_pathrZmodule_to_findrobjr*r*r+exec_phsolverserver1sJ         rZphsolverserverz Pyro-based server for PH solversc Csdddl}ytd}|j|d\}}Wn$tk rH}z|jSd}~XYnXtt|d|j|j|j dS)Nrzphsolverserver [options])argszphsolverserver: )Z error_labelrZ profile_countr) Z pyomo.environr parse_args SystemExitcoderrrrr)rZpyomoZoptions_parserrZ_excr*r*r+mainqs r)N)3gcrrrrruoptparserZpyutilib.commonr/Z pyutilib.miscrZ pyutilib.pyrorrrZ pyomo.coreZ pyomo.optrZ pyomo.commonr Zpyomo.common.pluginr r r r rZpyomo.pysp.phextensionrZ(pyomo.pysp.scenariotree.instance_factoryrZpyomo.pysp.phsolverserverutilsrrZ pyomo.pysp.phrZpyomo.pysp.phutilsrrrrZpyomo.pysp.util.miscrsixrrrr2rrrr*r*r*r+ sJ        /%7@