B u `w @sdZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl Z ddl mZmZddlmZmZddlmZmZmZmZmZddlmZddlmZdd lmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*ddl+m,Z,ddl-Z-ddl-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3e4dZ5Gddde6Z7e-8e7Gddde9Z:e:j;e:je:j?e:j@e:jAe:jBe:jCe:jDiZEddZFe ddddgZGddZHddZIGd d!d!e9ZJerddlKZKdd"lLmMZMd#d$ZNd%d&ZOeMPeJeNeMQd'eOGd(d)d)eZRGd*d+d+eReZSGd,d-d-eZTGd.d/d/eTeSeZUGd0d1d1eSeZVGd2d3d3eVeZWd4d5ZXieX_YeUeXjYd6<eWeXjYd7<d8d9ZZeZeX_[dS):)InvocationTypeScenarioTreeManagerClientSerialScenarioTreeManagerClientPyroScenarioTreeManagerFactoryN) defaultdict namedtuple)shutdown_pyro_components using_pyro4)dilldill_available) UndefinedData undefined SolverStatusTerminationConditionSolutionStatus) ActionHandle)PySPConfiguredObject)PySPConfigBlocksafe_declare_common_optionsafe_register_common_option_domain_must_be_str_domain_tuple_of_str)load_external_module)ScenarioTreeInstanceFactory)ScenarioTreeActionManagerPyro)ScenarioTreeServerPyro)create_ef_instance) iteritems itervaluesStringIO string_types)xrangez pyomo.pyspc@seZdZddZddZdS)_InvocationTypeMetacCs t||jS)N) isinstance_value)clsobjr'C/tmp/pip-unpacked-wheel-n62dbgi3/pyomo/pysp/scenariotree/manager.py __contains__=sz _InvocationTypeMeta.__contains__cs(ttfddjDdddS)Nc3s|]}t|jr|VqdS)N)r#r$).0r&)r%r'r( Asz/_InvocationTypeMeta.__iter__..cSs|jS)N)index)_r'r'r(Cz._InvocationTypeMeta.__iter__..)key)itersorted__dict__values)r%r')r%r(__iter__?sz_InvocationTypeMeta.__iter__N)__name__ __module__ __qualname__r)r5r'r'r'r(r"<sr"c@seZdZdZGdddeZGdddeZeddZedd Zed d Z ed d Z eddZ eddZ eddZ eddZeddZeddZeddeZeddeZeddeZed d!eZed"d#eZed$d%eZd&d'Zd(S))raControls execution of function invocations with a scenario tree manager. In all cases, the function must accept the process-local scenario tree worker as the first argument. Whether or not additional arguments are required, depends on the invocation type. For the 'Single' invocation type, no additional arguments are required. Otherwise, the function signature is required to accept a second argument representing the worker-local scenario or scenario bundle object. It is implied that the function invocation takes place on the scenario tree worker(s), which is(are) not necessarily the same as the scenario tree manager whose method is provided with the invocation type. For instance, Pyro-based scenario tree managers (e.g., ScenarioTreeManagerClientPyro) must transmit these method invocations to their respective scenario tree workers which live in separate processes. Any scenario tree worker is itself an instance of a ScenarioTreeManager so the same invocation rules apply when using this interface in a worker-local context. The ScenarioTreeManagerClientSerial implementation is its own scenario tree worker, so all function invocations take place locally and on the same object whose method is invoked. If the worker name is not provided (e.g., when the 'invoke_function' method is used), then the following behavior is implied for each invocation type: - Single: The function is executed once per scenario tree worker. Return value will be in the form of a dict mapping worker name to function return value. - PerScenario: The function is executed once per scenario in the scenario tree. Return value will be in the form of a dict mapping scenario name to return value. - PerScenarioChained: The function is executed once per scenario in the scenario tree in a sequential call chain. The result from each function call is passed into the next function call in *arg form after the scenario tree worker and scenario arguments (unless no additional function arguments were initially provided). Return value is in the form of a tuple (or None), and represents the return value from the final call in the chain. - PerBundle: Identical to the PerScenario invocation type except by bundle. - PerBundleChained: Identical to the PerScenarioChained invocation type except by bundle. * NOTE: The remaining invocation types listed below should initialized with any relevant data before being passed into methods that use them. This is done using the __call__ method, which returns a matching invocation type loaded with the provided data. Examples: InvocationType.OnScenario('Scenario1') InvocationType.OnScenarios(['Scenario1', 'Scenario2']) - OnScenario(): The function is executed on the named scenario and its associated scenario tree worker. Return value corresponds exactly to the function return value. - OnScenarios([]): The function is executed on the named scenarios and their associated scenario tree worker(s). Return value will be in the form of a dict mapping scenario name to return value. - OnScenariosChained([]): Same as PerScenarioChained only executed over the given subset of scenarios named. Invocation order is guaranteed to correspond exactly to the iteration order of the given scenario names. - OnBundle(): Identical to the OnScenario invocation type except with a bundle. - OnBundles([]): Identical to the OnScenarios invocation type except by bundle. - OnBundlesChained([]): Identical to the OnScenariosChained invocation type except by bundle. If the scenario tree worker name is provided (e.g., when the 'invoke_function_on_worker' method is used), then the following behaviors change: - Single: The return value corresponds exactly to the function return value (rather than a dict mapping worker_name to return value). - Per*: Function execution takes place only over the scenarios / bundles managed by the named scenario tree worker. - On*: Not necessarily designed for this context, but execution behavior remains the same. An exception will be raised if the named scenario(s) / bundles(s) are not directly managed by the named scenario tree worker. c@sLeZdZddZeddZeddZddZd d Zd d Z d dZ dS)zInvocationType._valuecCs||_||_dS)N)_key_index)selfr0r,r'r'r(__init__szInvocationType._value.__init__cCs|jS)N)r9)r;r'r'r(r0szInvocationType._value.keycCs|jS)N)r:)r;r'r'r(r,szInvocationType._value.indexcCst|j|jfS)N)hashr0r,)r;r'r'r(__hash__szInvocationType._value.__hash__cCs$|j|jko"|j|jko"|j|jkS)N) __class__r0r,)r;otherr'r'r(__eq__s zInvocationType._value.__eq__cCs || S)N)rA)r;r@r'r'r(__ne__szInvocationType._value.__ne__cCs d|jS)NzInvocationType.%s)r0)r;r'r'r(__repr__szInvocationType._value.__repr__N) r6r7r8r<propertyr0r,r>rArBrCr'r'r'r(r$s  r$cs0eZdZfddZeddZddZZS)zInvocationType._value_with_datacs$t|j|||||_d|_dS)N)superr?r<_domain_data)r;r0Zid_domain)r?r'r(r<sz(InvocationType._value_with_data.__init__cCs|jS)N)rG)r;r'r'r(datasz$InvocationType._value_with_data.datacCsT|jdk rtd||j|j|j}|jdks4t|||_|j|jksPt|S)Nz%Must create from InvocationType class)rI ValueErrorr?r0r,rFAssertionErrorrG)r;rIr&r'r'r(__call__s  z(InvocationType._value_with_data.__call__)r6r7r8r<rDrIrL __classcell__r'r')r?r(_value_with_datas  rNSingler PerScenarioPerScenarioChained PerBundlePerBundleChainedSingleInvocationPerScenarioInvocationPerScenarioChainedInvocationPerBundleInvocationPerBundleChainedInvocation OnScenario OnScenarios OnBundle OnBundles OnScenariosChainedOnBundlesChainedcOstdS)N)NotImplementedError)r;argskwdsr'r'r(r<szInvocationType.__init__N)r6r7r8__doc__objectr$rNrOrPrRrTrVrXrZr\r^r`rrbrrdrfrhrjrlr<r'r'r'r(rFs(s                rcCs*|tkr&td|t|ft|}|S)Nz%DEPRECATED: %s has been renamed to %s)_deprecated_invocation_typesloggerwarning)invocation_typer'r'r(_map_deprecated_invocation_types rw _WorkerInittype_namesrIcCs\t|trtd|fddSt|ttfks.tx|D]}t|ts4tq4Wtd|ddSdS)N scenarios)ryrzrI)r#r rxtypelisttuplerK)argnamer'r'r(_ScenarioWorkerInits  rcCst|trBt|ttfkstt|dks.ttd|f||idSt|ttfksVtt|tksftxF|D]>}t|ts~tt||ttfkstt||dksltqlWtd||dSdS)Nrbundles)ryrzrI) r#r r|r}r~rKlenrxdict)rrIrr'r'r(_BundleWorkerInit,s   rc@seZdZdZddZeddZeddZedd Zed d Z ed d Z eddZ eddZ eddZ eddZeddZddZddZd$ddZd%d d!Zd&d"d#ZdS)'ScenarioTreeSolveResultszA container that summarizes the results of solve request to a ScenarioTreeManagerSolver. Results will be organized by scenario name or bundle name, depending on the solve type.cCsL|dks t||_i|_i|_i|_i|_i|_i|_i|_i|_ i|_ dS)N)r{r) rK _solve_type _objective_cost_gap _solve_time_pyomo_solve_time_solver_status_solver_message_termination_condition_solution_status)r; solve_typer'r'r(r<Fs z!ScenarioTreeSolveResults.__init__cCs|jS)z~Return a string indicating the type of objects associated with these solve results ('scenarios' or 'bundles').)r)r;r'r'r(rsz#ScenarioTreeSolveResults.solve_typecCs|jS)zrReturn a dictionary with the objective values for all objects associated with these solve results.)r)r;r'r'r( objectivesz"ScenarioTreeSolveResults.objectivecCs|jS)zxReturn a dictionary with the sum of the stage costs for all objects associated with these solve results.)r)r;r'r'r(costszScenarioTreeSolveResults.costcCs|jS)zsReturn a dictionary with the pyomo solve times for all objects associated with these solve results.)r)r;r'r'r(pyomo_solve_timesz)ScenarioTreeSolveResults.pyomo_solve_timecCs|jS)zaReturn a dictionary with solve times for all objects associated with these solve results.)r)r;r'r'r( solve_timesz#ScenarioTreeSolveResults.solve_timecCs|jS)zkReturn a dictionary with solution gaps for all objects associated with these solve results.)r)r;r'r'r(gapszScenarioTreeSolveResults.gapcCs|jS)zmReturn a dictionary with solver statuses for all objects associated with these solve results.)r)r;r'r'r( solver_statussz&ScenarioTreeSolveResults.solver_statuscCs|jS)zmReturn a dictionary with solver messages for all objects associated with these solve results.)r)r;r'r'r(solver_messagesz'ScenarioTreeSolveResults.solver_messagecCs|jS)z{Return a dictionary with solver termination conditions for all objects associated with these solve results.)r)r;r'r'r(termination_conditionsz.ScenarioTreeSolveResults.termination_conditioncCs|jS)zoReturn a dictionary with solution statuses for all objects associated with these solve results.)r)r;r'r'r(solution_statussz(ScenarioTreeSolveResults.solution_statuscCsVt|tst|j|jkr.td|j|jfx"dD]}t||t||q4WdS)NzCan not update scenario tree manager solver results object with solve type '%s' from another results object with a different solve type '%s') rrrrrrrrr)r#rrKrrJgetattrupdate)r;results attr_namer'r'r(rs   zScenarioTreeSolveResults.updatecCs>||jkrtd|i}xdD]}t|||||<q W|S)zReturn a dictionary that summarizes all results information for an individual object associated with these solve results.zTThis results object does not hold any results for scenario tree object with name: %s) rrrrrrrrr)rKeyErrorr)r; object_namerrr'r'r( results_fors  z$ScenarioTreeSolveResults.results_forFNcCs>tt|t|j}t|dkr0tddStdd|D}|jdkrrttd|f}dt |d d}n2|jd kst ttd |f}dt |d d }|d d 7}|r|dd7}|dd7}t|xZ|D]P}|j|}|j |}|j |} |j |} |j|} dt |d }t|tr:|d7}n|d7}t|trX|d7}n|d7}t| ts| dk r|d7}n|d7}|d7}||||| | | f;}|r&|j|} t| ts| dk r|d7}n|d7}|| f;}|j|} t| ts| dk r|d7}n|d7}|| f;}t|qWtddS)z=Print a summary of the solve results included in this object.rzNo result data availableNcss|]}tt|VqdS)N)rstr)r* _object_namer'r'r(r+sz2ScenarioTreeSolveResults.pprint..rz Bundle Namez%-zs r{z Scenario Namez%-16s %-16s %-14s %-14s %-16s)ZCostZ Objectivez Objective Gapz Solver StatuszTerm. Conditionz %-11sz Solve Timez Pyomo Timez%-16sz%-16.7ez %-16sz %-16.7ez %-14.4ez %-14sz %-14s %-16sz %-11.2f)r}filterr2rkeysrprintmaxrrrKrrrrr#r rgetr)r; output_times filter_names object_names max_name_lenlinerZobjective_valueZ cost_valuerrterm_conditionrrr'r'r(pprintst                            zScenarioTreeSolveResults.pprintcCstt|t|j}t|dkr0tddStdd|D}|jdkrrttd|f}dt |d d}n2|jd kst ttd |f}dt |d d }|d d 7}t|xN|D]F}|j |}|j |}dt |d }|d 7}||||f;}t|qWtddS)z=Print a summary of the solve results included in this object.rzNo result data availableNcss|]}tt|VqdS)N)rr)r*rr'r'r(r+Hsz9ScenarioTreeSolveResults.pprint_status..rz Bundle Namez%-zs r{z Scenario Namez %-14s %-16s)z Solver StatuszTerm. Conditionr) r}rr2rrrrrrrrKrr)r;rrrrrrrr'r'r( pprint_status?s4       z&ScenarioTreeSolveResults.pprint_statuscsvtt|tj}t|dkr0tddStfdd|D}tdd|Drftdtnjdd|D}t |t t|t t fd d|Dt t|}td j t|t||ftfd d|D}td d|Drtd tnjdd|D}t |t t|t t fdd|Dt t|}tdj t|t||fdS)NrzNo result data availablec3s|]}j|VqdS)N)r)r*r)r;r'r(r+ksz@ScenarioTreeSolveResults.print_timing_summary..css|]}t|tVqdS)N)r#r )r*xr'r'r(r+mszOAt least one of the %s had an undefined solve time - skipping timing statisticscSsg|] }t|qSr')float)r*rr'r'r( rszAScenarioTreeSolveResults.print_timing_summary..c3s|]}t|dVqdS)g@N)pow)r*r)meanr'r(r+tszWSolve time statistics for %s - Min: %0.2f Avg: %0.2f Max: %0.2f StdDev: %0.2f (seconds)c3s|]}j|VqdS)N)r)r*r)r;r'r(r+scss|]}t|tVqdS)N)r#r )r*rr'r'r(r+szUAt least one of the %s had an undefined pyomo solve time - skipping timing statisticscSsg|] }t|qSr')r)r*rr'r'r(rsc3s|]}t|dVqdS)g@N)r)r*r)rr'r(r+sz]Pyomo solve time statistics for %s - Min: %0.2f Avg: %0.2f Max: %0.2f StdDev: %0.2f (seconds))r}rr2rrrranyZ object_typesumrmathsqrtrminr)r;rrZ solve_timesZstd_devZpyomo_solve_timesr')rr;r(print_timing_summary`sJ          z-ScenarioTreeSolveResults.print_timing_summary)FN)N)N)r6r7r8rqr<rDrrrrrrrrrrrrrrrr'r'r'r(r@s">           J !r)SerializerBasecCsDddi}||jx*dD]"}tdd||D||<qW|S)Nr?z@pyomo.pysp.scenario_tree.manager_solver.ScenarioTreeSolveResults)rrrcss|]\}}|t|fVqdS)N)r)r*r0valr'r'r(r+sz3ScenarioTreeSolveResults_to_dict..)rr3ritems)r&rIrr'r'r( ScenarioTreeSolveResults_to_dicts   rcCst|d}d|kst|j|x$|jD]}tt|j||j|<q,Wx$|jD]}tt|j||j|<qRWx$|j D]}tt |j ||j |<qxW|S)Nrr?) rrKr3rrrrrrrr) classnamedr&rr'r'r( dict_to_ScenarioTreeSolveResultss      rz@pyomo.pysp.scenario_tree.manager_solver.ScenarioTreeSolveResultscs,eZdZed8ddZGdddeZGdddeZGdd d eZGd d d eZ fd d Z ddZ ddZ e ddZe ddZe ddZe ddZddZddZddZd d!Zdejd"dd#d#fd$d%Zd9d&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5Z d6d7Z!Z"S):ScenarioTreeManagerNcCs|dkrt}|S)N)r)r%optionsr'r'r(_declare_optionssz$ScenarioTreeManager._declare_optionsc@seZdZddZdS)zScenarioTreeManager.AsynccCstdS)z;Wait for the job request to complete and return the result.N)rn)r;r'r'r(completesz"ScenarioTreeManager.Async.completeN)r6r7r8rr'r'r'r(Asyncsrc@s"eZdZdZdddZddZdS)zScenarioTreeManager.AsyncResult)_action_manager_result_action_handle_data_invocation_type _map_resultNcCsd|dk r|dkst|dk r(|dk s(t|dk rH|dks.)rr#rrrrrKrrwait_forrsetrZwait_any get_resultsremover)r;rZahsrr')rr;r(rs4          z(ScenarioTreeManager.AsyncResult.complete)NNN)r6r7r8 __slots__r<rr'r'r'r( AsyncResults  rc@s"eZdZdZdddZddZdS) z$ScenarioTreeManager.AsyncResultChain)_results _return_indexcCs||_||_dS)N)rr)r;r return_indexr'r'r(r<sz-ScenarioTreeManager.AsyncResultChain.__init__cCs`x@tt|jD].}t|j|tjs*t|j||j|<qWd}|jdk r\|j|j}|S)z;Wait for the job request to complete and return the result.N) r!rrr#rrrKrr)r;irr'r'r(r s    z-ScenarioTreeManager.AsyncResultChain.completeN)r)r6r7r8rr<rr'r'r'r(AsyncResultChains rc@s eZdZdZddZddZdS)z'ScenarioTreeManager.AsyncResultCallback)r_donecCs||_d|_dS)NF)rr)r;rr'r'r(r<0sz0ScenarioTreeManager.AsyncResultCallback.__init__cCs|js||_d|_|jS)z;Wait for the job request to complete and return the result.T)rr)r;r'r'r(r4s z0ScenarioTreeManager.AsyncResultCallback.completeN)r6r7r8rr<rr'r'r'r(AsyncResultCallback-srcsX|jtkrtd|jtt|j||d|_d|_i|_i|_d|_ d|_ d|_ dS)Nz'%s is an abstract class for subclassingF) r?rrnrEr<_error_shutdown_scenario_tree_scenario_to_bundle_map_aggregate_user_data_inside_with_block _initialized_objective_sense)r;rorp)r?r'r(r<;s  zScenarioTreeManager.__init__cCsb|jjrtd||j|s$tx8|D]0}||jkrPtd|||j|f||j|<q*WdS)Nz)Initializing scenario bundle with name %szXUnable to form binding instance for bundle %s. Scenario %s already belongs to bundle %s.)_optionsverboserrcontains_bundlerKrrJ)r; bundle_name scenario_list scenario_namer'r'r( _init_bundleRs  z ScenarioTreeManager._init_bundlecCsL|jjrtd||j|s$t|j|}x|jD] }|j|=q8WdS)Nz&Releasing scenario bundle with name %s) rrrrrrK get_bundle_scenario_namesr)r;rbundlerr'r'r(_release_bundlebs  z#ScenarioTreeManager._release_bundlecCs|jS)z@Return the objective sense declared for all subproblems.)r)r;r'r'r(objective_senseqsz#ScenarioTreeManager.objective_sensecCstdS)N)rn)r;r'r'r(modules_importedwsz$ScenarioTreeManager.modules_importedcCs|jS)N)r)r;r'r'r( scenario_tree{sz!ScenarioTreeManager.scenario_treecCs|jS)N)r)r;r'r'r( initializedszScenarioTreeManager.initializedcOst}d}d|_yT|jjr>tdt|j|td|j||}|jjrdtdt|jWn:|j stdtdt j t |YnX|jjs|jjrtdt||S) ztInitialize the scenario tree manager. A scenario tree manager must be initialized before using it. NTzInitializing %s with options:rz%s is successfully initializedzEException encountered. Scenario tree manager attempting to shut down.zOriginal Exception:z(Overall initialization time=%.2f seconds)timerrrrr|r6display_options_initr tracebackprint_exceptionsysexc_infocloser)r;rorpZinit_start_timerr'r'r( initializes2 zScenarioTreeManager.initializecCs d|_|S)NT)r)r;r'r'r( __enter__szScenarioTreeManager.__enter__cGs|ddk rttjdt}t|d|g}tj|d|_y |Wq|t dt | Yq|Xn|dS)NrzFException encountered. Scenario tree manager attempting to shut down. TziException encountered during emergency scenario tree manager shutdown. Printing original exception here: ) rstderrwriterr}rrrrrterrorgetvalue)r;rotmp_argsr'r'r(__exit__s      zScenarioTreeManager.__exit__cCsd|jjrtdt|jj|t|jdr<|jj d|_i|_ i|_ d|_ d|_d|_dS)z;Close the scenario tree manager and any associated objects.zClosing _scenario_instance_factoryNF)rrrrr?r6 _close_implhasattrrrrrrrrr)r;r'r'r(rs  zScenarioTreeManager.closer'Fc Cs|jstd|r|rtdt|}|tjksX|tjksX|tjksX|tjksX|tj krj|j sjtd|j |||||||dS)a Invokes a function on scenario tree constructs managed by this scenario tree manager. The first argument accepted by the function must always be the process-local scenario tree worker object, which may or may not be this object. Args: function: The function or name of the function to be invoked. If the object is a function, then the manager will attempt to transmit it using the dill package. Otherwise, the argument must be a string and the module_name keyword is required. module_name: The name of the module containing the function. This can also be an absolute path to a file that contains the function definition. If this function argument is an actual function, this keyword must be left at its default value of None; otherwise, it is required. invocation_type: Controls how the function is invoked. Refer to the doc string for pyomo.pysp.scenariotree.manager.InvocationType for more information. function_args: Extra arguments passed to the function when it is invoked. These will always be placed after the initial process-local scenario tree worker object as well as any additional arguments governed by the invocation type. function_kwds: Additional keywords to pass to the function when it is invoked. async_call: When set to True, the return value will be an asynchronous object. Invocation results can be obtained at any point by calling the complete() method on this object, which will block until all associated action handles are collected.f oneway_call: When set to True, it will be assumed no return value is expected from this function (async_call is implied). Setting both async_call and oneway_call to True will result in an exception being raised. *Note: The 'oneway_call' and 'async_call' keywords are valid for all scenario tree manager implementations. However, they are designed for use with Pyro-based implementations. Their existence in other implementations is not meant to guarantee asynchronicity, but rather to provide a consistent interface for code to be written around. Returns: If 'oneway_call' is True, this function will always return None. Otherwise, the return value type is governed by the 'invocation_type' keyword, which will be nested inside an asynchronous object if 'async_call' is set to True. z-The scenario tree manager is not initialized.z$async oneway calls do not make sensez[Received request for bundle invocation type but the scenario tree does not contain bundles.) module_namerv function_args function_kwds async_call oneway_call) r RuntimeErrorrJrwrrTrVrfrhrlrcontains_bundles_invoke_function_impl)r;functionr rvr r rrr'r'r(invoke_functions*K      z#ScenarioTreeManager.invoke_functioncCs2|jstd|r|rtd|j|||||dS)a4Invokes a method on a scenario tree constructs managed by this scenario tree manager client. This may or may not take place on this client itself. Args: method_name: The name of the method to be invoked. method_args: Arguments passed to the method when it is invoked. method_kwds: Keywords to pass to the method when it is invoked. async_call: When set to True, the return value will be an asynchronous object. Invocation results can be obtained at any point by calling the complete() method on this object, which will block until all associated action handles are collected. oneway_call: When set to True, it will be assumed no return value is expected from this method (async_call is implied). Setting both async_call and oneway_call to True will result in an exception being raised. *Note: The 'oneway_call' and 'async_call' keywords are valid for all scenario tree manager client implementations. However, they are designed for use with Pyro-based implementations. Their existence in other implementations is not meant to guarantee asynchronicity, but rather to provide a consistent interface for code to be written around. Returns: If 'oneway_call' is True, this function will always return None. Otherwise, the return corresponds exactly to the method's return value, which will be nested inside an asynchronous object if 'async_call' is set to True. z-The scenario tree manager is not initialized.z$async oneway calls do not make sense) method_args method_kwdsrr)rrrJ_invoke_method_impl)r; method_namerrrrr'r'r( invoke_method0s+z!ScenarioTreeManager.invoke_methodcCsP|drLtd|}|dkr:|drLtd|n|drLtddS)a~Push the fixed queue on the scenario tree nodes onto the actual variables on the scenario instances. * NOTE: This function is poorly named and this functionality will likely be changed in the near future. Ideally, fixing would be done through the scenario tree manager, rather than through the scenario tree. rz * :A  a 5  r!cseZdZedddZfddZeddZdej ddfd d Z fd d Z fd dZ ddZ dddZdddZddZZS)_ScenarioTreeManagerWorkerNcCs&|dkrt}t|dt|d|S)Nrr)rr)r%rr'r'r(rssz+_ScenarioTreeManagerWorker._declare_optionscsF|jtkrtd|jtt|j||d|_i|_i|_d|_dS)Nz'%s is an abstract class for subclassing) r?rYrnrEr< _instances_bundle_binding_instance_map_solve_results preprocessor)r;rorp)r?r'r(r<s  z#_ScenarioTreeManagerWorker.__init__cCstdS)N)rn)r;r'r'r(uncompressed_scenario_treesz5_ScenarioTreeManagerWorker.uncompressed_scenario_treer'c sdkr ittjs*|dk r(tdn|dkr.csg|]}j|qSr')rr)r*r)r;r'r(rsz.)&r#sixr rJrrmodulespyutilibmiscZ import_file__file__endswithr splitrrrrOrPrRrr{rbr`rIrdrjrrKrTrVrrrfrrhrlr) r;rr rvr r Z this_moduleZmodule_attrnamesubnameZ call_objectsrrbr')rr r r;r(_invoke_function_by_workers                                        z5_ScenarioTreeManagerWorker._invoke_function_by_workerc s6tt||||jjr&td|t}|j|s>t ||j ksLt |j |}xn|j D]d}|j |}||jks~t |j||kst |jdk st |j|j|kst |jdks`t q`W|jj|jd|jddt|j|j|jjd}||j |j<t}|jjs|jjr2td|||fdS)Nz/Forming binding instance for scenario bundle %sF)create_variable_idsZmaster_scenario_treeZinitialize_solution_data)Zef_instance_nameZverbose_outputzCTime construct binding instance for scenario bundle %s=%.2f seconds)rErYrrrrrrrrKr[rrr`r _instancerZZ parent_blocklinkInInstancesrrr) r;rrr=rrscenariobundle_ef_instanceend_time)r?r'r(rs>        z'_ScenarioTreeManagerWorker._init_bundlecs|j|st||jkst|j|}|j|}x0|jD]&}|j|}||j|j q.)r_scenario_bundlesrrZr[)r;Z bundle_namesrr'r'r(r fs   z&_ScenarioTreeManagerWorker._close_implcKs<|dkrtd}|j|}|j|j}t|jdrbt|jjt sb|jjdk rbt |jj|j |<nlt|jdrt|jj t s|jj dk rt |jj |j |<n2t|jdr|jj }t |jj |j |<n t|j |<t|dr|j|j|<n t|j|<|jj|j|<|jj|j|<|jj|j|<t|jdkrt|jdks>t|j}|jj|f|||f|j|<|d} t| dr| jdk r| j|j|<n t|j|<| j|j|<d } d } xH|jjD]<} |j | j} | !| | j"| j#7} | | j$| j#7} qW| |j%|<| |j&|<n(t|j%|<t|j&|<t|j|<t|j|<|S) Nr user_timewallclock_timerrrrQrg)'rrrr[rr solverr#rur rrrvrr rstatusrmessagerrrsolutionrK_smap solutions load_fromr\rrr _scenariosr`update_solution_from_instancerZ probabilityrrr)r;rrmanager_resultsrprZbundle_instancer results_sm solution0Zbundle_objective_valueZbundle_cost_valueZbundle_scenarioror'r'r(rqsh                      z7_ScenarioTreeManagerWorker._process_bundle_solve_resultc Ks|dkrtd}|j|}|j}|jr6|jt|jdrpt |jj t sp|jj dk rpt |jj |j |<ndt|jdrt |jjt s|jjdk rt |jj|j |<n*t|jdrt |jj|j |<n t|j |<t|dr|j|j|<n t|j|<|jj|j|<|jj|j|<|jj|j|<t|jdkrt|jdksDt|j}|jj|f|||f|j|j<||d}t|dr|j dk r|j |j |<n t|j |<|j|j!|<|j"|j#|<|j$|j%|<n(t|j#|<t|j%|<t|j |<t|j!|<|S) Nr{rurvrrrrQr)&rrr`rmrrrZ deactivater rwr#rur rrrvrr rrxrryrrrrzrKr{r|r}r\rrrrrrrr) r;rrrrproZscenario_instancerrr'r'r(rsX                     z9_ScenarioTreeManagerWorker._process_scenario_solve_resultc Csd}x|jjD]}t|jr|d7}|jdk rx|jD]v}|j}xjt|jD]\\}\}}|j|\}} ||j kr|jj | || fqL||j krL|jj | || fqLWq6W|qW|S)NrrQ)r _tree_nodesr _fix_queuer]r~rr _variable_idsZVARIABLE_FREEDZfreed_variablesrDZVARIABLE_FIXEDZfixed_variablesr) r;r tree_noderorZ variable_idZ fixed_status new_valueZ variable_namer,r'r'r(rs(       z<_ScenarioTreeManagerWorker._push_fix_queue_to_instances_impl)N)N)N)r6r7r8r rr<rDr^rrOrkrrr rrrrMr'r')r?r(rYqs   w 2  ] NrYcseZdZed"ddZfddZfddZfdd Zed d Z d d Z de j ddddfddZ d#ddZddZddZddZddZddZde j ddddfddZd$d d!ZZS)%rNcCs&|dkrt}t|dt|d|S)N!output_instance_construction_timecompile_scenario_instances)rr)r%rr'r'r(rDsz0ScenarioTreeManagerClientSerial._declare_optionscs(d|_g|_g|_tt|j||dS)Nz*ScenarioTreeManagerClientSerial:MainWorker) _worker_namer _bundle_namesrErr<)r;rorp)r?r'r(r<Ssz(ScenarioTreeManagerClientSerial.__init__cs0tt|||||jks t|j|dS)N)rErrrrKrD)r;rr)r?r'r(r]s z,ScenarioTreeManagerClientSerial._init_bundlecs.tt||||jkst|j|dS)N)rErrrrKr)r;r)r?r'r(rds z/ScenarioTreeManagerClientSerial._release_bundlecCs|jS)N)r)r;r'r'r(r^jsz:ScenarioTreeManagerClientSerial.uncompressed_scenario_treecs~jdk stt}jjr&tdjjjjjjjj jj jjd_ jj sbjjrvtdt|jjrtdt}jj j jjddjjdj_tfdd jjDstd d jjD_jj sjjrtd t|jrt}jjr4td x"jjD]}|j|jq>Wt}jj stjjrtd||tjjr xvjD]l}jjrtdj|xHjjD]<}j||j}t|dkstj|dqWqWtjjrjxPj D]F}jjr@tdj|x"jjD]}j||qJWq Wj!dj"didS)Nz$Constructing scenario tree instances)rr+rrz1Time to construct scenario instances=%.2f secondsz$Linking instances into scenario treeT)rrlrc3s|]}|jjkVqdS)N)r)r*_s)r;r'r(r+sz?ScenarioTreeManagerClientSerial._init_client..cSsg|] }|jqSr')r)r*Z _scenarior'r'r(rsz@ScenarioTreeManagerClientSerial._init_client..z3Time link scenario tree with instances=%.2f secondsz6Construction extensive form instances for all bundles.z.Scenario bundle construction time=%.2f secondszNExecuting user defined aggregategetter callback function defined in module: %srQzFExecuting user defined posinit callback function defined in module: %s)r)#rrKrrrrrZ%construct_instances_for_scenario_treerr+rrZrrnr$rr~rallrrrtrrrr&r3r2r1rrr%r5rr)r;Zbuild_start_timer=rrqcallback_module_keyrorr')r;r(rXrs           z,ScenarioTreeManagerClientSerial._init_clientr'Fc Cs||jkstt} |jjr4tdt|||f|j|||||d} |rPd} |rb|jd| d} t} |jj sz|jjrtd| | | S)Nz\Transmitting external function invocation request for function=%s in module=%s on worker=%s.)r rvr r )rz%Function invocation time=%.2f seconds) rrKrrrrrrkrr) r;rIrr rvr r rrr=rrqr'r'r(rHs(  z?ScenarioTreeManagerClientSerial._invoke_function_on_worker_implc Cs||jkstt}|jjr0td||jf|dkrScenarioTreeManagerClientSerial._get_scenarios_for_worker_implcCs||jkst|jS)N)rrKr)r;rIr'r'r(rV* szZ server_initr server_nameraction_handlesZinstance_factoryrr'r'r(acquire_scenariotreeserversP sZ            zB_ScenarioTreeManagerClientPyroAdvanced.acquire_scenariotreeserverscCsZ|jdk st|jjr*tdt|jj|jr@td|||j_ | g}x&t |j D]}|||qdW||j|x|D]}|j|qW~d}d}|jjrd}d}nd}d}g}| x(|jjD]}||jj|||dqW||r:|j|x|D]}|j|q$W|jd|_i|_i|_ dS) zQRelease the pool of scenario tree servers and destroy the action manager.Nz"Releasing %s scenario tree serverszIUnpausing pyro transmissions in preparation for releasing manager workersZScenarioTreeServerPyro_shutdownFZScenarioTreeServerPyro_resetT)rrr)rrKrrrrrrrZignore_task_errorsrr}rrrD remove_workerrrrrrr)r;rrrIrr action_namerr'r'r(r sP      zB_ScenarioTreeManagerClientPyroAdvanced.release_scenariotreeserverscCs(|jdk st|j|j}d|_|S)z^Pause transmission of action requests. Return whether transmission was already paused.NT)rrKpauser)r; was_pausedr'r'r(r s  z5_ScenarioTreeManagerClientPyroAdvanced.pause_transmitcCs"|jdk st|jd|_dS)zRUnpause transmission of action requests and bulk transmit anything queued.NF)rrKunpauser)r;r'r'r(r s z7_ScenarioTreeManagerClientPyroAdvanced.unpause_transmitc srjdk st|dkr.tjjfddd}jjrFtd||ft|trt |}y|j |dd}Wn4t k rt d|t jttdfYnXt|tk rtd t|tfjj|d |||f|| d }j|||j|<j||jd kr |jj|<nN|jd ks0t|jj|<gj|<x&|jD]} j||j| qNW|S)Ncstj|gS)N)rrr)k)r;r'r(r. r/zC_ScenarioTreeManagerClientPyroAdvanced.add_worker..)r0z;Initializing worker with name %s on scenario tree server %sT)sparsezUnable to serialize options for registered worker name %s (class=%s). The worker options did not seem to match the registered options on the worker class. Did you forget to register them? Message: %srQz.css|]}|jdfVqdS)TN)r)r*ror'r'r(r+x scs(g|] }|jD]}|jr|jqqSr') node_listr)r*ror)need_node_datar'r(r s zMScenarioTreeManagerClientPyro._request_scenario_tree_data..nodescSsg|]}|qSr'r')r*rr'r'r(r sr{Z&_collect_scenario_tree_data_for_clientT)rrFcsg|]}|jr|jqSr')r)r*r)rr'r(r scss|] }| VqdS)Nr')r*rr'r'r(r+ scss|] }| VqdS)Nr')r*rr'r'r(r+ s)rrrrrrr~rrKrrrtrrLrSrrrQrrr) r;r= async_resultsZneed_scenario_datarrZ node_namerror')rr(_request_scenario_tree_datak sV        z9ScenarioTreeManagerClientPyro._request_scenario_tree_datac st}tddjjD}tddjjD}drFtdjrx|D]F}||}xt |dD]\}}||dkst d||<j |} | j |d | j |d | j |d | j |d | j |d | j |dt| j | j| _tddt | j D| _qzWxTt |dD]D\} } || dks^t d|| <j| } | d| _| d| _qDWdrZtd|qZWn\xX|D]N} || }xt |dD]\}}||dkst d||<j |} | j |d | j |d | j |d | j |d | j |d | j |dt| j | j| _tddt | j D| _qWxTt |dD]D\} } || dkst d|| <j| } | d| _| d| _qWdrtd| qWjjdj_tfddjjDs6t tt|sHt tt|sZt drntddsdrtdt|dS)Ncss|]}|jdfVqdS)FN)r)r*rr'r'r(r+ szKScenarioTreeManagerClientPyro._gather_scenario_tree_data..css|]}|jdfVqdS)FN)r)r*ror'r'r(r+ srz)Waiting for scenario tree data collectionrFTr_standard_variable_ids_variable_indices_integer_binary_semicontinuouscss|]\}}||fVqdS)Nr')r*r0rr'r'r(r+ sr{_objective_namerz2Successfully loaded scenario tree data for bundle=css|]\}}||fVqdS)Nr')r*r0rr'r'r(r+ sz4Successfully loaded scenario tree data for scenario=rc3s|]}|jjkVqdS)N)r)r*r)r;r'r(r+ sz2Scenario tree instance data successfully collectedrz/Scenario tree data collection time=%.2f seconds)rrrrr~rrrrrrKZget_noderrrrrrrrZ_derived_variable_idsZ_name_index_to_idr`rrrr) r;rr=Zhave_node_dataZhave_scenario_datarrZtree_node_nameZ node_datarrZ scenario_dataror')r;r(_gather_scenario_tree_data s                          z8ScenarioTreeManagerClientPyro._gather_scenario_tree_datac sVt}|jjrtdt|jjdkr0td|j rRddt |jj D}nddt |jj D}t|j t|jjkstt|jdkstt|jdkstt|j}d}y|j|jdd}Wn8tk rtd |j|jttd fYnX|dk sttd d |jjD}tt}d}x@t|jjD].}t||krXP||d 7<|d 7}qDWt|t|jjkstxJ|D]B}||dkstx(t||D]} || |!qWqW|j"rt|jj#s|$i} xt%|D]\} }||dj&t'fd d ||DsBtt'dd ||Ds^tt'dd ||Dsztd} dkr|dt| } t(dd||Dtdd ||D} n4dkst|dt| } t)dd||D} |j*| | ||j|d}|jj#r2|j+|j|d,| | <n| | |<| j&dkrx| j-D]T}|j.|sdt| |j/|<x0| j0|D]"}|j1|st| |j2|<qzWqNWn>| j&dkstx,| j-D]"}|j1|st| |j2|<qWq W|jj#s|3t}|jj4s|jjr*td|||jj#rB|j+d| dS|j+|j| dSdS)Nz1Transmitting scenario tree worker initializationsrz5No scenario tree server processes have been acquired!cSsg|]}t|j|jqSr')rrscenario_names)r*rr'r'r(r- szRScenarioTreeManagerClientPyro._initialize_scenariotree_workers..cSsg|]}t|jqSr')rr)r*ror'r'r(r1 sT)rzUnable to extract options for registered worker name %s (class=%s). Did you forget to register the worker options into the options object passed into this class? Message: %srQcss|]}|gfVqdS)Nr')r*rr'r'r(r+I szQScenarioTreeManagerClientPyro._initialize_scenariotree_workers..c3s|]}|jkVqdS)N)ry)r* _worker_init) init_typer'r(r+b scss|]}t|jtkVqdS)N)r|rzr~)r*rr'r'r(r+d scss|]}t|jdkVqdS)rQN)rrz)r*rr'r'r(r+f srz:Worker_BundleGroupcSsg|]}|jdqS)r)rz)r*rr'r'r(rl scss(|] }|jd|j|jdfVqdS)rN)rzrI)r*rr'r'r(r+n sr{z:Worker_ScenarioGroupcSsg|]}|jdqS)r)rz)r*rr'r'r(ru s)r)rz-Initialization transmission time=%.2f seconds)r)5rrrrrrrrrrreversedrr{rrKrrrrrrrr6rrrrrr itertoolscycler!rDr0rrr enumerateryrrrrrrrzrrrIrOrrr)r;r=jobsrrZworker_initializationsrZcntrZ_irZcntrrIrrrrrqr')rr( _initialize_scenariotree_workers! s                 z>ScenarioTreeManagerClientPyro._initialize_scenariotree_workerscsjdk stjr`x jjD]}|j|jq"Wtjj}jj rt dt |n$tjj }jj rt dt |jj }|dkr|}n&||kr||krt d|||f|}jj dkrԈjjnd}jj r|dkr |dk stt d||fnt d||fj||djj r:t d}tj}d }tjjrjrjtx`tjjD]N\}} jj rt d j|j| j|tjjfd } | d_qzWjj rt d d }jddjfd dtjjr} | r.tjjs.td }xRtj j!D]@\}} jj rft dj|j| j|tj"d dqBW|r#$d _%fdd} &| } j'|| gddS)NzBundle jobs available: %szScenario jobs available: %srzValue assigned to pyro_required_scenariotreeservers option (%s) is greater than the number of available jobs (%s). Limiting the number of servers to acquire to %sz7Using timeout of %s seconds to acquire up to %s serverszEWaiting to acquire exactly %s servers to distribute work over %s jobs)rzDBroadcasting requests to initialize workers on scenario tree serversFz_Transmitting invocation of user defined aggregategetter callback function defined in module: %s)rvr z:Broadcasting final aggregate data to scenario tree serversTZ assign_datar)rrzXTransmitting invocation of user defined postinit callback function defined in module: %s)rvrcsd_dS)NT)rrr')rr;r'r(_complete_init s zBScenarioTreeManagerClientPyro._init_client.._complete_init)rr)(rrKrrtrrrrrrrrr~rrrrr2rr&rzipr3r4r2rrrRrrrr%r5r6rPrrrrr)r;rZnum_jobsZservers_requiredrZinitialization_handlerNrrrErrrZasync_callbackr')rr;r(rX s                z*ScenarioTreeManagerClientPyro._init_clientcCs|jS)N)r)r;r'r'r(rM- sz0ScenarioTreeManagerClientPyro._worker_names_implcCs |j|S)N)r)r;rr'r'r(rP0 sz;ScenarioTreeManagerClientPyro._get_worker_for_scenario_implcCs |j|S)N)r)r;rr'r'r(rR3 sz9ScenarioTreeManagerClientPyro._get_worker_for_bundle_implr'Fc Cs|r |r t|jdk stt}|jjr>tdt||ft|tj srt sVt d|dk rft dt |}n|dkrt d|jr|s|st dd} d} |tjks|tjks|tjkr|} i} x*|jD] } | | |j| ||||||d<qW|tjkr dd} | st|nV|tjkrN|j||j||||||d} n&|tjkr~|j||j||||||d} n|tjks|tjkrld} d}|tjkr|j} tj}n|tjkst|j} tj}i}x8|jD].}| |} | |krg|| <|| |qW|} i} x2|D]*} | | |j| ||||| |||d<q$Wd d} | st|n|tjks|tj kr |jrt d ||}x`t!t"|jd D]J}|j|} |j#|j|j| |||||d dd $}t"|dkrd}qW|j|jd||||||d} nT|tj%ks8|tj&krZ|jrLt d |d} d}|tj%krn|j} tj%}n|tj&ks~t|j} tj&}t"|jdkstt't(|j}g}|}xt"|dkrV||)| |d} t"|dks| | |dkr|j| ||||||d d} t"|dkr<|j#|j| d $}t"|dkrNd}g}qWnt d|ddtDf|rd} d} |j#|j| | d}|s|$}t}|jj*s|jjrtd|||S)NzOTransmitting external function invocation request for function=%s in module=%s.zEThis dill module must be available when transmitting function objectszPThe module_name keyword must be None when the function argument is not a string.z7A module name is required when a function name is givenaUnable to perform external function invocations. Pyro transmissions are currently paused, but the function invocation is not one-way and not asynchronous.This implies action handles be collected within this method. Pyro transmissions must be un-paused in order for this to take place.)r rvr r rcSstddt|DS)Ncss$|]}|D]}|||fVq qdS)Nr')r*rr0r'r'r(r+~ szXScenarioTreeManagerClientPyro._invoke_function_impl....)rr)rr'r'r(r.} szEScenarioTreeManagerClientPyro._invoke_function_impl..cSstddt|DS)Ncss$|]}|D]}|||fVq qdS)Nr')r*rr0r'r'r(r+ szXScenarioTreeManagerClientPyro._invoke_function_impl....)rr)rr'r'r(r. szNChained invocation type %s cannot be executed when Pyro transmission is pausedrQF)rrr'rz.)rrzCExternal function invocation request transmission time=%.2f seconds)+rKrrrrrrr#rcr r rJr rrrrOrTrPrrrrrbrQrIrfrSrdrhrDrRrVr!rrrrjrlr}rr0r)r;rr rvr r rrr=rrrrIZ_get_worker_funcr worker_maprrrrZobject_names_for_workerrqr'r'r(r= s<                      *                z3ScenarioTreeManagerClientPyro._invoke_function_implc s|r r tjdk stt}jjr2tdjrH|sHsHtddkrTi}t fddj D}|s rd}j j|d} |s| } t} jjsĈjjrtd| || S)Nz?Transmitting method invocation request to scenario tree workersaUnable to perform method invocations. Pyro transmissions are currently paused, but the method invocation is not one-way and not asynchronous.This implies action handles be collected within this method. Pyro transmissions must be un-paused in order for this to take place.c 3s2|]*}jj|| d|fVqdS))rrIrrrorpN)rrr)r*rI)rrrrr;r'r(r+PszDScenarioTreeManagerClientPyro._invoke_method_impl..)rz8Method invocation request transmission time=%.2f seconds)rKrrrrrrrJrrrrrrr) r;rrrrrr=rrrrqr')rrrrr;r(r2s<   z1ScenarioTreeManagerClientPyro._invoke_method_implc Ks<|dkrtd}t|dks t|\}}x|D]}||t|||<q.W|j|dk rrttt|j||j|<n t|j|<|j|dk rtt t|j||j|<n t|j|<|j |dk rtt t|j ||j |<n t|j |<|dk r8|j |j}t|t|kstx$|D]} |j | || qW|S)NrrS)rrrKrrrrr rrrrrrrr` set_solution) r;rrrrpobject_resultsscenario_tree_resultsr0Zbundle_scenariosrr'r'r(rns4       z:ScenarioTreeManagerClientPyro._process_bundle_solve_resultcKs|dkrtd}t|dks t|\}}x|D]}||t|||<q.W|j|dk rrttt|j||j|<n t|j|<|j|dk rtt t|j||j|<n t|j|<|j |dk rtt t|j ||j |<n t|j |<|dk r|j |||S)Nr{rS)rrrKrrrrr rrrrrr`r)r;rrrrprrr0r'r'r(rs,     z}||j}||kr\i||<|j||kr<|j|||j<q s          -  W T YR / O D