B u `@sddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z ddlmZddlmZmZddlmZmZddlmZdd lmZmZmZmZmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*m+Z+dd lm,Z,m-Z-dd l.m/Z/ddl0Z1ddl2m3Z3ej45e6Z7e8dZ9ddZ:ddZ;d ddZd"ddZ?e@dkre?ejAddddS)#N) WriterFactory) ComponentMap)value as_numeric)_VarData) Constraint_ConstraintData) TextLabelerNumericLabeler)InvocationType)locate_annotations$StochasticConstraintBoundsAnnotation"StochasticConstraintBodyAnnotationStochasticObjectiveAnnotation"StochasticVariableBoundsAnnotation)map_variable_stagesmap_constraint_stages build_repns_safe_remove_file_no_negative_zero_deterministic_check_value_deterministic_check_constant)PySPConfigValuePySPConfigBlocksafe_register_common_optionsafe_register_unique_option_domain_must_be_str)ScenarioTreeManagerClientSerialScenarioTreeManagerClientPyro)launch_command) iteritemsz pyomo.pyspc Os|j}g}xp|jdddD]^}i}t|dr:|j|d<|`t|drR|j|d<|`t|drj|j|d<|`|||fqWz6yt||f||St d|j YnXWdxX|D]P\}}t|dr|`t|dr|`t|dr|`x|D]}t ||||qWqWXdS)NT)active descend_into _gen_obj_repn _gen_con_repn_repnzt>|-t?sxt|st,|-|krtd2|-j%t jf|t,|-+|-}?|?@std3|-j%|?j!}@d|?_!|@dkrd4}@|5|-},t|,dkstx|,D]}A|AAd5s&|AAd6r|>dksB|>ddksBt|8d7}8|9<|<|A|:<|=t=t |-jBt |@|-jB|6|-<tC|-_D|AAd5 r|-jE|7|-<tC|-_Fq|AAd7 r|>dks|>ddk r|8d7}8|9<|<|A|:<|=t=t |-jBt |@|-jB|6|-<tC|-_Dn|AAd8 r|>dk s<|>ddk sdk s|>ddk r|8d7}8|9<|<|A|:<|=t=t |-jEt |@|-jE|7|-<tC|-_Fn dstqWq^WWdQRXWdQRXd}Bt2tj$||j%d:d*}Ct2tj$||j%d;d*}D|j;};d<}Ed0}F|D|-t? st| st,|-|k rtd2|-j%tjf|t,|-+|-}?|?@ std3|-j%t|?jGdk st|Gdk r|?jG}Gt|Gdk s(t|5|-},t |G}G|Gjfd>d d t |?j"}Hx|GD]}&t>|&tH spt|&jI r|td}Ix|,D]6}A|Bd7}B|C<|E|A|Mf|D<|Ft=t |I qW q\Wt|H|?_" qWWdQRXWdQRXd}Nt2tj$||j%d@d*6}Ot2tj$||j%dAd*}Pd/}Qd0}R|P|&tH std}IxW|"|#|(|)|N|8|BfS)JNrcss|] }|VqdS)Nr4).0conr4r4r5 wsz:_convert_external_setup_without_cleanup..T)r!r")Z max_allowedFcs|ddS)NrT)getname)x)constraint_name_bufferr4r5z9_convert_external_setup_without_cleanup..)keyzxThe %s annotation was declared with external entries but no active Constraint objects were recovered from those entries.c3s|]}|jfVqdS)N)default)r8r9)stochastic_rhsr4r5r:scs|ddS)NrT)r;)r<)r=r4r5r>r?c3s|]}|jfVqdS)N)rA)r8r9)stochastic_matrixr4r5r:scss|]\}}t|VqdS)N)id)r8r9_r4r4r5r:scss|]\}}t|VqdS)N)rD)r8r9rEr4r4r5r:szcThe DDSIP writer does not currently support stochastic variable bounds. Invalid annotation type: %sz|No stochastic annotations found. DDSIP conversion requires at least one of the following annotation types: - %s - %s - %sr%cSsg|] }t|qSr4)r)r8cr4r4r5 sz;_convert_external_setup_without_cleanup..cSsg|] }t|qSr4)r)r8rFr4r4r5rGsz .lp.setuplp column_order row_ordercSsdS)NTr4)r<r4r4r5r>r?)!enforce_derived_nonanticipativitycss|]\}}}t|VqdS)N)rD)r8symbolvarscenario_tree_idr4r4r5r:scss|]\}}}t|VqdS)N)rD)r8rLrMrNr4r4r5r:scss|]\}}t|VqdS)N)rD)r8symbolsr9r4r4r5r: ssymbolic_solver_labelsr<cs&|t|trt|kr dnS)N) isinstancerrD)r<)_labelerfirststage_var_suffixfirststage_variable_idsr4r5r>Rs z.lpz .lp.symbolslabelerZforce_objective_constantcSsdS)NTr4)r<r4r4r5r>jr?wz%s %s c3s(|] }|D]\}}||fVqqdS)Nr4)r8Z stage_namerOr9)StageToConstraintMapr4r5r:sz.rhs.sc.structz.rhs.scz %s z %.17g z scen %.17g zThe constraint %s has been declared in the %s annotation but it was not identified as a second-stage constraint. To correct this issue, remove the constraint from this annotation.zaOnly linear constraints are accepted for conversion to DDSIP format. Constraint %s is not linear.gZc_e_Zc_l_Zr_l_Zc_u_Zr_u_z.matrix.sc.structz .matrix.scz %s %s zscen cs|S)Nr4)_v)rIr4r5r>1r?zThe coefficient for variable %s has been marked as stochastic in constraint %s using the %s annotation, but the variable does not appear in the canonical constraint expression.z.cost.sc.structz.cost.sczwThe %s annotation was declared with external entries but no active Objective objects were recovered from those entries.zjOnly linear stochastic objectives are accepted for conversion to DDSIP format. Objective %s is not linear.cs|S)Nr4)rZ)rIr4r5r>zr?zThe coefficient for variable %s has been marked as stochastic in objective %s using the %s annotation, but the variable does not appear in the canonical objective expression.ZONE_VAR_CONSTANTZZeroStochasticDataz.lp.detcSsdS)NTr4)r<r4r4r5r>r?)PospathexistsAssertionErrordict scenario_treer&Z findRootNodeZstageslistZcomponent_data_objectsrr r lenZhas_declarationsZexpand_entriessort RuntimeError__name__tuplersetupdaterr ValueErrorr'rr#r$valuesrZconstantZ linear_coefsZquadratic_coefsjoinr,rrrrZ_instance_objectiveZ parent_blockrDr enumeratepopr r openZ_ScenarioTreeSymbolMapsortedZ _variable_idsZbySymbolZis_expression_typeZbyObjectr( writelinesZ probabilitywriterrSrZ is_linear startswithlowerrZ_lowerupperZ_upperZ linear_varsrZfixedziprrA_namer r)]r.r/output_directoryrUrK io_optionsr`r2rootnodeZ firststageZ secondstageZobjective_name_bufferZvariable_name_bufferZall_constraintsZstochastic_rhs_entriesZempty_rhs_annotationZstochastic_matrix_entriesZempty_matrix_annotationZstochastic_constraint_idsZstochastic_objectiveZstochastic_varboundsZ repn_cacheZ block_repnsZrepnZoutput_filenamewriterZ output_fnameZ symbol_mapZStageToVariableMapZsecondstage_variable_idsZsecondstage_constraint_idsZobjective_objectZobjective_blockZobjective_repnfirststage_variable_countsecondstage_variable_countZ column_indexrLrMrNfirststage_constraint_countsecondstage_constraint_countrJZ row_indexrOr9rWZsymbols_filenamefZ st_symbol_maplinesZid_Zlp_labelZconstraint_symbolsZmodified_constraint_lbZmodified_constraint_ubstochastic_rhs_countZ f_rhs_structZf_rhsZscenario_probabilityZrhs_struct_templateZ rhs_templateZ include_boundZconstraint_repnZ body_constantZ con_labelstochastic_matrix_countZ f_mat_structZf_matZmatrix_struct_templateZmatrix_templateZvar_listZ new_coefsZvar_coefiZ_varZcoefZ var_labelstochastic_cost_countZ f_obj_structZf_objZobj_struct_templateZ obj_templateZ sorted_valuesobjZobjective_variablesZinclude_constantZstochastic_objective_labelZ obj_constantZreference_model_nameZdet_output_filenamersrtr4)rYrTrIr=rUrVrCrBr5r)csF                                                                  *                 $                 &   r)Fc+ Cs8ddl}ddl}|dkri}tj|s,t|j} | rBtdtj |d} tj| sft | |j dt t j| |||fd} | jd} | j} | | \}}}}}}}i}tj |d}t|tj | | d}t||||d<tj |d }t|tj | | d }t||||d <xnd D]d}|d }|dkrT|dkr|q,n(|dkrn|dkr|q,n|dks|ttj ||}|||<t|t|d}|dkr|dks|d| | jdksttj | | jd|d}tj|stt|d}t||WdQRXx\| jD]R} tj | | jd|}tj|s^tt|d}t||WdQRXq0WWdQRXq,W|r,tdtdtj|tdtdtd|td|tdtd|td|td|td|td td!||s|rLtd"| td#d$}!ytd%sdd&}!nd$}!Wnd$}!YnX|rtd'xd(D]}tj | | jd|}"xr| jD]h} tj | | jd|}#|!rtd)|#d*|"}$ntj|#|"d$d+ }$|$rtd,|"| j|#fqWqW|r4td-tj | | jd.}%xn| jD]d} tj | | jd.}&|!rtd)|&d*|%}$ntj|&|%d$d+ }$|$rPtd/|%| j|&fqPW|sD|rtd0xR| jD]H} x@d1D]8}tj | | jd|}'tj|'s tt|'qWqWtt| dkrRtj| d&d2n|rRtd3tj |d4}(t|(d})|)d5|)d6t|d7|)d8t|d7|)d9t|d7|)d:t|d7|)d;|d7|)dt|d7|)d?t|d7|)d@WdQRX|(|dA<tj |dB}*t|*d})|)tj|d|d7|)tj|dA|d7d|kst|)tj|d|d7dC|kr|)tj|dC|d7dD|kr"|)tj|dD|d7WdQRX|*|dE<|S)FNrz,DDSIP conversion does not yet handle bundlesZscenario_filesr6)Zinvocation_typeZ function_argszcore.lpz.lpcorezcore.lp.symbolsz .lp.symbolsrO)rhscostmatrixz.sczcost.scz matrix.sczrhs.scrXrzNames .z.structrz DDSIP Conversion CompletezOutput Saved To: zBasic Problem Information:z - Variables:z - First Stage: %dz - Second Stage: %dz - Constraints:z - Stoch. RHS Entries: %dz - Stoch. Matrix Entries: %dz - Objective:z - Stoch. Cost Entries: %dzS Starting scenario structure consistency checks across scenario files stored in %s.zThis may take some time. If this test is prohibitively slow or can not be executed on your system, disable it by activating the disable_consistency_check option.Fzdiff --help > /dev/nullTz, - Checking structure in stochastic files...)z rhs.sc.structzcost.sc.structzmatrix.sc.structzdiff -q  )ZshallowaThe structure indicated in file '%s' does not match that for scenario %s indicated in file '%s'. This suggests one or more locations of stachastic data have not been annotated. If you feel this message is in error, please report this issue to the PySP developers.z> - Checking deterministic sections in the core problem file...z.lp.detaYOne or more deterministic parts of the problem found in file '%s' do not match those for scenario %s found in file %s. This suggests that one or more locations of stochastic data have not been been annotated on the reference Pyomo model. If this seems like a tolerance issue or a developer error, please report this issue to the PySP developers.z%Cleaning temporary per-scenario files) rHzlp.detz lp.symbolsz matrix.sczmatrix.sc.structzcost.sczcost.sc.structzrhs.scz rhs.sc.struct) ignore_errorszHTemporary per-scenario files are retained in scenario_files subdirectoryz ddsip.configz BEGIN z FIRSTCON  z FIRSTVAR zSECCON zSECVAR zPOSTFIX zSCENAR zSTOCRHS z STOCCOST zSTOCMAT z END configz ddsip.stdinrrscript) pyomo.environZ%pyomo.solvers.plugins.smanager.phpyror[r\r]r^r`Zcontains_bundlesrirkmkdirZinvoke_functionthisfiler Z PerScenarioZ scenariosr,rshutilcopy2rnrq copyfileobjprintrelpathsystemfilecmpcmprblistdirrmtreestr)+rwrUscenario_tree_managerrKrxdisable_consistency_checkskeep_scenario_filesverbosepyomor`Zscenario_directorycountsZreference_scenarioZreference_scenario_namer{r|r}r~rrrZ input_filesZlp_dstZlp_srcZ symbols_dstZ symbols_src_typebasenamedstfdstsrcfsrcr/Zhas_diffZreference_struct_filenameZscenario_struct_filenamercZreference_lp_det_filenameZscenario_lp_det_filenameZscenario_filenameZconfig_filenamerscript_filenamer4r4r5convert_externalsf                   ,                  &rc Cs|dkrt}t|dt|dt|dt|dt|dt|dtdtddd d t|d td td dd d t|dtdtddd d t|dtdtddd d t|dtdtddd d t|dt|t||S)N disable_gcprofile tracebackrrQrwrzFThe directory in which all DDSIP files will be stored. Default is '.'.r)domain descriptiondocZ visibilityfirst_stage_suffixZ__DDSIP_FIRSTSTAGEzPThe suffix used to identify first-stage variables. Default: '__DDSIP_FIRSTSTAGE'rKFzAdds nonanticipativity constraints for variables flagged as derived within their respective time stage (except for the final time stage). The default behavior behavior is to treat derived variables as belonging to the final time stage.raLDisables consistency checks that attempt to find issues with the DDSIP conversion. By default, these checks are run after conversion takes place and leave behind a temporary directory with per-scenario output files if the checks fail. This option is not recommended, but can be used if the consistency checks are prohibitively slow.rzKeeps around the per-scenario DDSIP files created for testing whether a conversion is valid (whether or not the validation checks are performed). These files can be useful for debugging purposes.r) rrrrrboolrZregister_optionsr)optionsr4r4r5convertddsip_register_optionssp             rc Cstj|jst|jt}d|ji}|jr6td}|j dkrJt }n|j dkrXt }||2}| t |j|j||j||j|j|jd}WdQRXt}tdtd||dS)NrQserialZpyro)rKrxrrrrRz!Total execution time=%.2f seconds)r[r\r]rwmakedirstimerQcompile_scenario_instancesr^rrrZ initializerrrKrrrr)r start_timerxZ manager_classrfilesZend_timer4r4r5run_convertddsip's4     rc Csddl}t}t||dj}d|d_y8tjdd}||||d_||j |dWn$t k r}z|j Sd}~XYnXt t |d|j|j|jdS)Nrrzpyomo.pysp.convert.ddsip)prog)r0zpyomo.pysp.convert.ddsip: )Z error_labelrZ profile_countr)rrrgetZ _argparseargparseArgumentParserZinitialize_argparseZimport_argparse parse_args SystemExitcoderrrrr)r0rrZ argparse_valZapZ_excr4r4r5mainQs$     r__main__r7)r0)FNFFF)N)N)Br[rsysrrrloggingZ pyomo.optrZpyomo.common.collectionsrZpyomo.core.base.numvaluerrZpyomo.core.base.varrZpyomo.core.base.constraintrrZpyomo.core.baser r Zpyomo.pysp.scenariotree.managerr Zpyomo.pysp.annotationsr r rrrZpyomo.pysp.convert.smpsrrrrrrrZpyomo.pysp.util.configrrrrrrrZpyomo.pysp.util.miscrrrsixr r\abspath__file__r getLoggerr*r6r)rrrrreargvr4r4r4r5 sP    $    -g  W* -