B u `N@stddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZmZmZddlmZdd lmZdd lmZdd lmZmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2GdddeZ3Gddde)Z4ddZ5ddZ6eddd ddZ7dS)!N) OptionParser)ApplicationError)Bunch) import_file) TaskWorkerTaskWorkerServershutdown_pyro_components)VarSuffix Constraint) UndefinedData) pyomo_command)PauseGC)ExtensionPointSingletonPlugin) SolverFactoryTerminationConditionSolutionStatus)IPHSolverServerExtension)ScenarioTreeInstanceFactory) TransmitTypeInvocationType)_PHBase)reset_nonconverged_variablesreset_stage_cost_variablesextract_solve_times)launch_command)iterkeys iteritemsc@seZdZddZddZdS) PHPyroWorkercKsXd|d<dttf|d<tj|f||j|_d|_d|_ i|_ d|_ i|_ dS)NzPH Pyro ServerZ caller_namezPySPWorker_%d@%snameT) osgetpidsocket gethostnamer__init__ WORKERNAMEtypeblocktimeout_phsolverserver_mapZ_bulk_task_collection_modules_imported)selfkwdsr.=/tmp/pip-unpacked-wheel-n62dbgi3/pyomo/pysp/phsolverserver.pyr%0szPHPyroWorker.__init__c Cstf|}d}|jdkr(|j|j=d}n|jdkrpt|j|j|j<|j|j|j_|j|_|j|j|}nN|jdkrt d|j j |j dd|_ n$t}|j|j|}WdQRX|S)NreleaseT initializeshutdownzReceived shutdown request)r')ractionr* object_name_PHSolverServerr+r&r processprintZ dispatcherZ clear_queuer'Z_worker_shutdownr)r,dataresultZpgcr.r.r/r6@s"     zPHPyroWorker.processN)__name__ __module__ __qualname__r%r6r.r.r.r/r.src@seZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)S)*r5cKs*t|d|_d|_tt|_||_dS)NT)rr% _first_solve_uncompressed_scenario_treerr _ph_pluginsr+)r,Zmodules_importedr-r.r.r/r%_s   z_PHSolverServer.__init__cCs>|jrtdt|tddt|j|jtjD}|S)Nz8Received request to collect warmstart data for scenario=css|]\}}||jfVqdS)N)value).0symbolZvardatar.r.r/ vsz4_PHSolverServer.collect_warmstart..) _verboser7strdictr _instancesZ_PHInstanceSymbolMapsr ZbySymbol)r, scenario_namer9r.r.r/collect_warmstartps  z!_PHSolverServer.collect_warmstartcCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzGReceived request to activate PH objective weight terms for scenario(s)=Fz*PH solver server has not been initialized!z$Activating PH objective weight terms) rDr7rElistrrG _initialized RuntimeErrorr"activate_ph_objective_weight_terms)r,r.r.r/rMs  z2_PHSolverServer.activate_ph_objective_weight_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzIReceived request to deactivate PH objective weight terms for scenario(s)=Fz*PH solver server has not been initialized!z&Deactivating PH objective weight terms) rDr7rErJrrGrKrLr$deactivate_ph_objective_weight_terms)r,r.r.r/rNs  z4_PHSolverServer.deactivate_ph_objective_weight_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzIReceived request to activate PH objective proximal terms for scenario(s)=Fz*PH solver server has not been initialized!z&Activating PH objective proximal terms) rDr7rErJrrGrKrLr$activate_ph_objective_proximal_terms)r,r.r.r/rOs  z4_PHSolverServer.activate_ph_objective_proximal_termscCsN|jr tdttt|j|jdkr2tdt ||jrJtddS)NzKReceived request to deactivate PH objective proximal terms for scenario(s)=Fz*PH solver server has not been initialized!z(Deactivating PH objective proximal terms) rDr7rErJrrGrKrLr&deactivate_ph_objective_proximal_terms)r,r.r.r/rPs  z6_PHSolverServer.deactivate_ph_objective_proximal_termsc Csi}i}|d<xp|dD]d}|j|}i}||<|j|d<|j|d<|j|d<t|j|d<t|j|d<t|j|d<qWi}|d<xF|dD]:}|j |} i} ||<| j | d <| j | d <| j | d <qW|S) NZnodes _variable_ids_standard_variable_ids_variable_indices_integer_binary_semicontinuousZ scenarios_objective_name_objective_sense_rho) _scenario_treeget_noderQrRrSrJrTrUrV get_scenariorWrXrY) r,tree_object_namesr8Z node_data node_name tree_nodeZthis_node_dataZ scenario_datarHscenarioZthis_scenario_datar.r.r/collect_scenario_tree_datas&           z*_PHSolverServer.collect_scenario_tree_datacCsddlm}|rtdtdtd|tdt|td|td||dkrhtd ||dkrtd t|| dkrtd t| td t| |jrtd |jrtdx|jD]}||qW||_ ||_| |_ | |_ | |_ | |_ ||_||_||_||_|jr.td|t||jd}|j|<|dkr`td|dtj|srt|dkstj|stt|||_}|jd||| |jd|_|jdkrtdg}|jr$|j|dkrtd|d|jrtd||j|}|j }n |!||jj}d|j_t"#|j|_$||j_|j%||jjj&|j||jd}|jj'||j dd|jj(dj |_ |)|jrtdx|jD]}|*|qW|+t,t-f}|.||/t01||2t03||jr"|4|jr2tdt5||r|jrd|j6|j7|d|j8dn|j6|j9|j:d|j8d|jrtdx|jD]}|;|qWd|_dS) Nr)PersistentSolverz/Received request to initialize PH solver serverzModel source: zScenario Tree source: z Solver type: zScenario or bundle name: z$Scenario tree bundle specification: zCreate random bundles: zScenario tree random seed: z$Linearize non-binary penalty terms: z4PH solver servers cannot currently be re-initializedz2Invoking pre-initialization PHSolverServer pluginszConstructing solver type=) solver_iozUnknown solver type=z specified)Zdownsample_fractionZbundlesZrandom_bundlesZ random_seedverbosezFUnable to launch PH solver server - scenario tree construction failed.FzBundle=z does not exist.zLoading scenarios for bundle=)compile_scenario_instancesreT)Zcreate_variable_idsz6Invoking post-instance-creation PHSolverServer pluginsz0Setting instance for persistent solver interface)symbolic_solver_labelsoutput_fixed_variable_boundsz3Invoking post-initialization PHSolverServer plugins)compressZ%construct_instances_for_scenario_treeZlinkInInstances _scenariosZ_setup_scenario_instancesZpost_instance_creationZ_create_scenario_ph_parametersr r Z_create_instance_symbol_mapsZadd_ph_objective_weight_termsrrNZadd_ph_objective_proximal_termsrPZ_form_bundle_binding_instances isinstanceZ set_instance_bundle_binding_instance_map_write_fixed_variablesr\instanceZpost_ph_initialization)r,model_location data_locationr4objective_sense solver_typerdscenario_bundle_specificationcreate_random_bundlesscenario_tree_random_seed default_rho!linearize_nonbinary_penalty_termsretain_quadratic_binary_termsbreakpoint_strategyinteger_toleranceoutput_solver_resultsrerfrbplugin object_solverZscenario_instance_factoryZscenarios_to_constructZscenario_bundleZinstance_factoryZ instancesZ symbol_ctypesr.r.r/r1s                             z_PHSolverServer.initializecCsd}t|s>t|r8tdd|jjddD}nt}|jr|j|}i}xT|jD](}|j |}|j |d| ||<q`Wn |j |}|j |d| }|S)Ncss|] }|jVqdS)N)r )rAsr.r.r/rCsz2_PHSolverServer.collect_results..)stages) rTransmitAllStagesTransmitNonLeafStagessetrZrrqrsrtr\update_solution_from_instance copy_solution)r,r4Z results_flagsstages_to_loadbundleresultsrHr`r.r.r/collect_resultss           z_PHSolverServer.collect_resultsc (Csddlm} |jdkrtd|jr.d} nd} |jrHtd| |f|jrp|j|std|dSn|j |std |dS||j krtd | |f|j |} ||_ ||_ ||_ ||_||_||_| |_|jr |jrtd xF|jD]} | |qWn,|jrtd x|jD]} | |q$WxBt|jD]4\}}|jrltd |dt||| j|<qDW|jdkr||jdkr|||js|jdkrt|j|jt|j|jnt|j|jt | | rd|j ||jd}nd|j ||j |j |jd}| !r>|j|d<d}t"#| st"$| rzt%dd|jj&ddD}nt%}d}d}|jrr|j'|}t((}| j)|f|}t((|}t*|j+dks|+dj,t-j.ks|j/j0t1j.kr|jr|2td|dd}q|jr0td||j3rTtdt4t|j2dd|j5}|j6j7||j |j8|j d|jrtd|i}x,|jj9D] }|j:|d|;||j<<qWi}|j'|}x|jj9D]}|j|j<}i}xt|jD]j}i}t=||}xL|j>t?ddD]:} i}!x$t| D]\}"}#|@|#|!|"<q$W|!|| jA<qW|||<qW|||j<<qWn|jjB|}|j|}t((}| j)|f|}t((|}t*|j+dks|+dj,t-j.ks|j/j0t1j.kr|jr|2td |dd}n|jrtd!||j3r>td"|t|j2dd|j5}|j6j7||j |j8|j d|j:|d|;}|jrtd#|i}xz|jD]p}i}t=||}xRt|jCt?ddD]<\}$} i}!x$t| D]\}"}#|@|#|!|"<qW|!||$<qW|||<qW|s||f|jD|<i}%|+d}&tE|&d$rZt |&jFtGsZ|&jFdk rZ|&jF|%d$<tH|dd%\|%d&<|%d'<|&j,jA|%d(<|||%f}'nd)}'|jdkr|jrtd*xH|jD]} | I|qWn,|jrtd+x|jD]} | J|qWd|_|'S),Nr)rbFz*PH solver server has not been initialized!rr}z(Received request to solve scenario %s=%szRPH solver server requested to solve bundle=%s, but that bundle is not recognized.zVPH solver server requested to solve scenario=%s, but that scenario is not recognized.z,No solver plugin available for scenario%s=%sz5Invoking pre-iteration-0-solve PHSolverServer pluginsz5Invoking pre-iteration-k-solve PHSolverServer pluginszProcessing solver option=z, value=T)load_solutionstee keepfilessuffixes)rrrrgrhr warmstartcss|] }|jVqdS)N)r )rArr.r.r/rCasz(_PHSolverServer.solve..rzSolve failed for bundle=z; no solutions generatedz$Successfully solved scenario bundle=zResults for scenario bundle=%s:)num)Z&allow_consistent_values_for_fixed_varsZ#comparison_tolerance_for_fixed_varsZignore_fixed_varsz(Successfully loaded solution for bundle=)r)activezSolve failed for scenario=z&Successfully solved scenario instance=z!Results for scenario instance=%s:z*Successfully loaded solution for scenario=gap)defaultZ solve_timepyomo_solve_timeZsolution_statusr.z6Invoking post-iteration-0-solve PHSolverServer pluginsz6Invoking post-iteration-k-solve PHSolverServer plugins)KrirbrKrLrZrqrDr7Z is_bundleZ is_scenariorl_teeZ_symbolic_solver_labelsr|Z_solver_optionsZ_solver_suffixesZ _warmstartZ_variable_transmissionr=r?Zpre_iteration_0_solveZpre_iteration_k_solverrEoptionsrjZ(form_ph_linearized_objective_constraintsZ'_reset_instance_linearization_variablesZ_preprocess_scenario_instancesrrGrrzZwarm_start_capablerrrrrr{timesolvelenZsolutionstatusrZ infeasibleZsolverZtermination_conditionrwriterkZ bundle_nameZ_smapZ solutionsZ load_fromZ$_comparison_tolerance_for_fixed_varsryrr_namegetattrZcomponent_objectsr getr _scenario_mapZ component_mapZ_solver_resultshasattrrr rZpost_iteration_0_solveZpost_iteration_k_solve)(r,r4rrrgrhsolver_optionssolver_suffixesrvariable_transmissionrbZ object_typerrkeyr@Zcommon_solve_kwdsrZfailurerbundle_ef_instanceZsolve_start_timerZ results_smZvariable_valuesr`Z suffix_valuesscenario_instanceZthis_scenario_suffix_values suffix_nameZthis_suffix_mapsuffix constraintZthis_constraint_suffix_mapindexZconstraint_dataZconstraint_nameZauxilliary_valuesZ solution0Zsolve_method_resultr.r.r/rsr                                                        z_PHSolverServer.solvecCsr|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|jj|}|j |qJWdS)NTz,Received request to update xbars for bundle=z.Received request to update xbars for scenario=Fz*PH solver server has not been initialized!) rDrZrqr7rKrLr_tree_node_mapZ_xbarsupdate)r,r4 new_xbarsr^Z node_xbarsr_r.r.r/ update_xbars*s   z_PHSolverServer.update_xbarscCsp|jrtd||jdkr$td||jkr:tddS|jj|}x$t|D]\}}|j| |qPWdS)Nz0Received request to update weights for scenario=Fz*PH solver server has not been initialized!zcERROR: Received request to update weights for instance not in PH solver server instance collection!) rDr7rKrLrGrZrr_wr)r,rH new_weightsr`tree_node_nameZtree_node_weightsr.r.r/update_weights<s    z_PHSolverServer.update_weightscCs|jrtd||jdkr$td||jkrPtd|dt|jdS|jj|}x$t |D]\}}|j | |qfWdS)Nz-Received request to update rhos for scenario=Fz*PH solver server has not been initialized!z4ERROR: Received request to update rhos for scenario=z4, which is not in the PH solver server instance set=) rDr7rKrLrGrEkeysrZrrrYr)r,rHnew_rhosr`rZtree_node_rhosr.r.r/ update_rhosOs    z_PHSolverServer.update_rhosc Cs|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|jj|j}| |qJWx,t|D] \}}|jj|j }| |qxWdS)NTz;Received request to update tree node statistics for bundle=z=Received request to update tree node statistics for scenario=Fz*PH solver server has not been initialized!) rDrZrqr7rKrLrrZ _minimumsrZ _maximums) r,rHZnew_node_minimumsZnew_node_maximumsrZtree_node_minimumsZthis_tree_node_minimumsZtree_node_maximumsZthis_tree_node_maximumsr.r.r/update_tree_node_statisticscs  z+_PHSolverServer.update_tree_node_statisticscCs|jr>|jdkr*td|d|ntd|d||jdkrPtd|jdkr~|j|}||ttj dnR||j krtd|d|d t |j dS|j |}||ttj ddS) NTz)Received request to define import suffix=z for bundle=z for scenario=Fz*PH solver server has not been initialized!) directionz0ERROR: Received request to define import suffix=z?, which is not in the collection of PH solver server instances=) rDrZrqr7rKrLr{Z add_componentr ZIMPORTrGrEr)r,r4rrrr.r.r/define_import_suffixs&   $ z$_PHSolverServer.define_import_suffixcstt|}jrTjr8td|d|d|ntd|d|d|jdkrftdd}jrjj|}n jj |}|j krj |}n,|t j krt j |}nt |dd}|j |<|} d} t|| sd | kr|d d \} } t|| std |d |dkr*d dkr8id} |tjkrxjrf|jj|f} n|jj |f} n|tjks|tjkrjstdtjj} nh|tjks|tjkrtjj } nB|tjks|tjkrtjj} ntd|ddtjDft|| | dk r8t| |tjkrb| \} } j| fS|tjks|tjks|tjkr}x(| D] \} } j| f|}qW|Stfdd| DSdS)Nz-Received request to invoke external function=z in module=z for bundle=z for scenario=Fz*PH solver server has not been initialized!T) clear_cache.rz Function=z is not present in module=r.z[Received request for bundle invocation type but the scenario tree does not contain bundles.zsz<_PHSolverServer.invoke_external_function..c3s,|]$\}}|j|ffVqdS)N)rZ)rA call_name call_object)function function_args function_kwdsr,r.r/rC sz;_PHSolverServer.invoke_external_function..)rrrDrZrqr7rKrLZ_scenario_bundle_maprr+sysmodulesrrsplitZSingleInvocationZPerBundleInvocationZPerBundleChainedInvocationrmrZPerScenarioInvocationZPerScenarioChainedInvocationZPerNodeInvocationZPerNodeChainedInvocationrZ_valuesrF)r,r4invocation_type module_name function_namerrZscenario_tree_objectZ this_moduleZmodule_attrnamesubnameZ call_objectsrrr9r.)rrrr,r/invoke_external_functions                               z(_PHSolverServer.invoke_external_functioncCs|jr>|jdkr"td|n td|tdt||jdkrPtd|jdkr|j|dkrtd|dn||jkrt|d t |||dS) NTz7Received request to restore cached solution for bundle=z9Received request to restore cached solution for scenario=zRestoring from cache id: Fz*PH solver server has not been initialized!zBundle=z does not exist.z3 is not in the PH solver server instance collection) rDrZrqr7rErKrLrrrGrrestoreCachedSolutions)r,r4cache_id release_cacher.r.r/rs    z&_PHSolverServer.restoreCachedSolutionscCs|jr>|jdkr"td|n td|tdt||jdkrPtd|jdkr|j|dkrtd|dn||jkrt|d t ||dS) NTz.Received request to cache solution for bundle=z0Received request to cache solution for scenario=zCaching with id: Fz*PH solver server has not been initialized!zBundle=z does not exist.z3 is not in the PH solver server instance collection) rDrZrqr7rErKrLrrrGrcacheSolutions)r,r4rr.r.r/r2s    z_PHSolverServer.cacheSolutionscCsz|jr.|jdkr"td|n td||jdkr@tdx,t|D] \}}|j|}|j |qJW| dS)NTz6Received request to update fixed variables for bundle=z8Received request to update fixed variables for scenario=Fz*PH solver server has not been initialized!) rDrZrqr7rKrLrr[Z _fix_queuer_push_fix_queue_to_instances)r,r4fixed_variablesr^Znode_fixed_varsr_r.r.r/update_fixed_variablesLs   z&_PHSolverServer.update_fixed_variablesc Csd}|jdkr^||j|j|j|j|j|j|j|j |j |j |j |j |j|j|j|j|j}n|jdkr|||j|j}nr|jdkrJd}d}d}x|s,y<|d7}||j|j|j|j|j|j|j|j|j }d}Wqtk r(}z0t d |j|f||krt d ||Wdd}~XYqXqW|dkrt d |jn|jd krf|!d}n|jd kr|"d}nl|jdkr|#d}nP|jdkr|$d}n4|jdkr$|j%&dkrx|j?}n|jdkr|@|j|jA|jBd}nl|jdkr|C|j|jAd}nJ|jdkr|D|jE}n0|jdkr|F|jG}ntHdtI|jd|S)Nr1rrrFrTz0Solve failed for object=%s - this was attempt=%dzVAborting PH solver server - the maximum number of solve attempts=%d have been executedz6Successfully recovered from failed solve for object=%srOrPrMrNZ load_rhosrZ load_weightsZ load_xbarsZload_tree_node_statsrrZ!restore_cached_scenario_solutionsZcache_scenario_solutionsrarIzERROR: Unknown action=z received by PH solver server)Jr3r1r~rr4rrrdrrrrrrrrrrerfrr Z var_configrrrrgrhrrrrrr7rOrPrMrNrZrqrrGrrZ_push_rho_to_instancesrrrrrZ_push_w_to_instancesrrZ_push_xbar_to_instancesrZnew_minsZnew_maxsrrrrrrrrrrrrrar]rIrHrLrE) r,r8r9Zmax_num_attemptsZattempts_so_farZsuccessful_solveexcrHrr.r.r/r6_s                           z_PHSolverServer.processN)r:r;r<r%rIrMrNrOrPrar1rrrrrrrrrrrr6r.r.r.r/r5]s. gM%qr5cCst}|jdddddd|jddd d d d d |jdddddd|jdddddd|jdddddgd |jddd ddd|jddd dd dd ||_|S)Nz --verbosezPGenerate verbose output for both initialization and execution. Default is False. store_truereF)helpr3destrz --profilezjEnable profiling of Python code. The value of this option is the number of functions that are summarized.storeprofileintr)rr3rr'rz --disable-gcz7Disable the python garbage collecter. Default is False. disable_gcz --tracebackzkWhen an exception is thrown, show the entire call stack. Ignored if profiling is enabled. Default is False. tracebackz--user-defined-extensionzVThe name of a python module specifying a user-defined PHSolverServer extension plugin.ruuser_defined_extensionsstringz --pyro-hostz=The hostname to bind on when searching for a Pyro nameserver. pyro_hostz --pyro-portz9The port to bind on when searching for a Pyro nameserver. pyro_port)r add_optionusage)Z usage_stringparserr.r.r/construct_options_parsersRrc Cstt}x|D] }|qWt|jdkrBx|jD]}|tjkrZtd|dnBtd|ttj }tj ddt |td|tj dd<ddl }|}| dr|d}|dd kr|dd }xd|tj||jD]L\}}t|tr|d krtt}x(|d d D]}t||r|qWqWq8Wytt|j|jd Wn(tdt|j|jddYnXdS)Nrz-User-defined PHSolverServer extension module=z already imported - skippingz>Trying to import user-defined PHSolverServer extension module=rzModule successfully loadedz.py/rrT)all)hostportz3PH solver server aborted. Sending shutdown request.)rrZ num_retries)rrdisablerrrrr7rJrninsertrinspectrfindrstripfindr getmembersisclass issubclassrrzenablerrrrr) rZph_extension_pointrZthis_extensionZ original_pathrZmodule_to_findr objr.r.r/exec_phsolverserver0sJ        rZphsolverserverz Pyro-based server for PH solversc Csdddl}ytd}|j|d\}}Wn$tk rH}z|jSd}~XYnXtt|d|j|j|j dS)Nrzphsolverserver [options])argszphsolverserver: )Z error_labelrZ profile_countr) Z pyomo.environr parse_args SystemExitcoderrrrr)rZpyomoZoptions_parserrZ_excr.r.r/mainps r)N)8r!r#rrrvoptparserZpyomo.common.errorsrZpyomo.common.collectionsrZ pyutilib.miscrZ pyutilib.pyrorrrZ pyomo.corer r r Z pyomo.optr Z pyomo.commonr Zpyomo.common.gc_managerrZpyomo.common.pluginrrrrrZpyomo.pysp.phextensionrZ(pyomo.pysp.scenariotree.instance_factoryrZpyomo.pysp.phsolverserverutilsrrZ pyomo.pysp.phrZpyomo.pysp.phutilsrrrZpyomo.pysp.util.miscrsixrrrr5rrrr.r.r.r/ sJ           /%7@