rotary bed notebooks

[jialuw96]

No description provided.

[adowling2]

Please open a PR into Pyomo with sufficient background suggestions for the Pyomo team to add feedback. Make sure the proposed changes are general and not one-off modifications just for this case study.

[jialuw96]

@bpaul4 @Ryan-Hughes-8 Please see this PR for the measurement optimization and MBDoE code based on the RPB mdel.
Could you also mention Anca here? I cannot find her github account.
If there are any problems, please let me know. Thank you!

[bpaul4]

Hi @jialuw96. Both @Ryan-Hughes-8 and I tested the workflow, and were unable to move past Step 3 due to Git being unable to locate your branch:

(rotary) C:\Users\Brandon>pip install git+https://github.com/jialuw96/pyomo/tree/RPB_MBDOE
Collecting git+https://github.com/jialuw96/pyomo/tree/RPB_MBDOE
  Cloning https://github.com/jialuw96/pyomo/tree/RPB_MBDOE to c:\users\brandon\appdata\local\temp\pip-req-build-ooma34ey
  Running command git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey'
  fatal: repository 'https://github.com/jialuw96/pyomo/tree/RPB_MBDOE/' not found
  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey' did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey' did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[lbianchi-lbl]

Hi @jialuw96. Both @Ryan-Hughes-8 and I tested the workflow, and were unable to move past Step 3 due to Git being unable to locate your branch:

(rotary) C:\Users\Brandon>pip install git+https://github.com/jialuw96/pyomo/tree/RPB_MBDOE
Collecting git+https://github.com/jialuw96/pyomo/tree/RPB_MBDOE
  Cloning https://github.com/jialuw96/pyomo/tree/RPB_MBDOE to c:\users\brandon\appdata\local\temp\pip-req-build-ooma34ey
  Running command git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey'
  fatal: repository 'https://github.com/jialuw96/pyomo/tree/RPB_MBDOE/' not found
  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey' did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/jialuw96/pyomo/tree/RPB_MBDOE 'C:\Users\Brandon\AppData\Local\Temp\pip-req-build-ooma34ey' did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

(Sorry for "crashing the PR"---I happened to see this comment in my GH notifications)

Does this pip install command refer to installing pyomo from the head branch of Pyomo/pyomo#3250? If so, I believe the installation URL should be tweaked slightly:

# install from head branch
pip install "git+https://github.com/jialuw96/pyomo@RPB_MBDOE"

# alternatively, install directly from PR URL
pip install "git+https://github.com/Pyomo/pyomo@refs/pull/3250/merge"

[bpaul4]

@lbianchi-lbl the modified installation call worked, thank you!

[jialuw96]

The installation instructions are updated. Other updates for this PR include:

Pyomo modification

• Pyomo.DoE in RPB_MBDoE branch: [https://github.com/jialuw96/pyomo/tree/RPB_MBDOE].
• PR link for the above branch: [https://github.com/Pyomo.DoE adds checking sentences to the type of objects Pyomo/pyomo#3250]

Readme file

• Update installation requirements

Counter-current flow model (Results are represented in Jialu's thesis chapter 4)

• Countercurrent_MBDoE.ipynb: MBDoE analysis applying Pyomo.DoE and direct_kaug to the RPB model. The mentioned modified Pyomo.DoE in RPB_MBDOE can now also run with sequential_finite mode.

[Ryan-Hughes-8]

Hi @jialuw96, I am attempting to follow the workflow in the readme file, but there is an issue in Step 1 of the Run measurement optimization results section. In Counterflow_MO_data_process.ipynb, there are missing files which results in the error below. Is there another file that should be ran first to generate these required files?

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[6], line 3
      1 # For five parameters, run both its upper and lower perturbation 
      2 # Stored ads. and des. results separately
----> 3 lower_ads_c1 = load_pickle(\"./MBDOE_sens_001/Tgin_des_393_001/Tgin_ads_363/Tgads363_Tgdes393_C1low_ads\")
      4 lower_des_c1 = load_pickle(\"./MBDOE_sens_001/Tgin_des_393_001/Tgin_ads_363/Tgads363_Tgdes393_C1low_des\")
      6 lower_ads_hgx = load_pickle(\"./MBDOE_sens_001/Tgin_des_393_001/Tgin_ads_363/Tgads363_Tgdes393_hgxlow_ads\")

Cell In[2], line 4, in load_pickle(name)
      1 def load_pickle(name): 
      2     \"\"\"Used for loading data from pickle file 
      3     \"\"\"
----> 4     file = open(name, \"rb\")
      5     data = pickle.load(file)
      6     file.close()

File c:\\Users\\hughesr\\anaconda3\\envs\\rotary_2\\lib\\site-packages\\IPython\\core\\interactiveshell.py:284, in _modified_open(file, *args, **kwargs)
    277 if file in {0, 1, 2}:
    278     raise ValueError(
    279         f\"IPython won't let you open fd={file} by default \"
    280         \"as it is likely to crash IPython. If you know what you are doing, \"
    281         \"you can use builtins' open.\"
    282     )
--> 284 return io_open(file, *args, **kwargs)

FileNotFoundError: [Errno 2] No such file or directory: './MBDOE_sens_001/Tgin_des_393_001/Tgin_ads_363/Tgads363_Tgdes393_C1low_ads'"

[lbianchi-lbl]

@Ryan-Hughes-8 can you try again and see if the latest set of commits after your original attempt resolve the errors you have experienced?

[Ryan-Hughes-8]

Hi @jialuw96, I got a chance to attempt to go through the steps laid out in the README for running measurement optimization results. I received a few errors and have a couple comments.

1. I needed to specify the python install when creating the conda environment. conda create --name rotary python=3.8
2. In the Counterflow_MO_data_process.ipynb notebook, I had to add Countercurrent_MO_QVs/ to the store paths in cells 12 and 13. Also in the Counterflow_MO_data_process.ipynb notebook, names_idx and total_fim are not defined in cells 26 and 27.
3. The README lists the notebook Counterflow_Jacobian_organize.ipynb but it doesn't exist. Perhaps it was meant to be Counterflow_Jacobian_process.ipynb.
4. Two errors in Countercurrent_MO.py. Line 281, 2_num_total_loc does not exist. Line 497, objective_opt is not defined in the function call.
5. When running the Countercurrent_MBDoE.ipynb notebook, I receive the error below when trying to compute the FIM in cell 7.
6. Counterflow-finite-difference-analysis.ipynb and draw_figures.ipynb run without any issue.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], [line 6]
      [2] doe_object = DesignOfExperiments(parameter_dict, design_gen,
      [3]                                  measure_class, create_model)
      [5] # analyze a square MBDoE problem
----> [6] result = doe_object.compute_FIM(mode=sensi_opt, 
      [7]                                 FIM_store_name = 'dynamic.csv', 
      [8]                                 read_output=None,
      [9]                                 scale_nominal_param_value=True)
     [12] result.result_analysis()

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\contrib\doe\doe.py:395, in DesignOfExperiments.compute_FIM(self, mode, FIM_store_name, specified_prior, tee_opt, scale_nominal_param_value, scale_constant_value, store_output, read_output, extract_single_model, formula, step)
    [393] square_timer.tic(msg=None)
    [394] if self.mode == CalculationMode.sequential_finite:
--> [395]     FIM_analysis = self._sequential_finite(
    [396]         read_output, extract_single_model, store_output
    [397]     )
    [399] elif self.mode == CalculationMode.direct_kaug:
    [400]     FIM_analysis = self._direct_kaug()

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\contrib\doe\doe.py:419, in DesignOfExperiments._sequential_finite(self, read_output, extract_single_model, store_output)
    [415]     jac = self._finite_calculation(output_record)
    [417] # if measurements are not provided
    [418] else:
--> [419]     mod = self._create_block()
    [421]     # dict for storing model outputs
    [422]     output_record = {}

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\contrib\doe\doe.py:657, in DesignOfExperiments._create_block(self)
    [654]         elif var.ctype is Param:
    [655]             var.value = self.scenario_data.scenario[s][par]
--> [657] mod.block = pyo.Block(mod.scenario, rule=block_build)
    [659] # discretize the model
    [660] if self.discretize_model is not None:

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\core\base\block.py:571, in BlockData.__setattr__(self, name, val)
    [566] if name not in self.__dict__:
    [567]     if isinstance(val, Component):
    [568]         #
    [569]         # Pyomo components are added with the add_component method.
    [570]         #
--> [571]         self.add_component(name, val)
    [572]     else:
    [573]         #
    [574]         # Other Python objects are added with the standard __setattr__
    [575]         # method.
    [576]         #
    [577]         super(BlockData, self).__setattr__(name, val)

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\core\base\block.py:1116, in BlockData.add_component(self, name, val)
   [1108]     logger.debug(
   [1109]         "Constructing %s '%s' on %s from data=%s",
   [1110]         val.__class__.__name__,
   (...)
   [1113]         str(data),
   [1114]     )
   [1115] try:
-> [1116]     val.construct(data)
   [1117] except:
   [1118]     err = sys.exc_info()[1]

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\core\base\block.py:2149, in Block.construct(self, data)
   [2144]     if self.index_set().isfinite() and (
   [2145]         self._dense or self._rule is not None
   [2146]     ):
   [2147]         for _idx in self.index_set():
   [2148]             # Trigger population & call the rule
-> [2149]             self._getitem_when_not_present(_idx)
   [2150] else:
   [2151]     # We must check that any pre-existing components are
   [2152]     # constructed.  This catches the case where someone is
   [2153]     # building a Concrete model by building (potentially
   [2154]     # pseudo-abstract) sub-blocks and then adding them to a
   [2155]     # Concrete model block.
   [2156]     _idx = next(iter(UnindexedComponent_set))

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\core\base\block.py:2088, in Block._getitem_when_not_present(self, idx)
   [2085]     data = None
   [2087] try:
-> [2088]     obj = self._rule(_block, idx)
   [2089]     # If the user returns a block, transfer over everything
   [2090]     # they defined into the empty one we created.  We do
   [2091]     # this inside the try block so that any abstract
   [2092]     # components declared by the rule have the opportunity
   [2093]     # to be initialized with data from
   [2094]    # _BlockConstruction.data as they are transferred over.
   [2095]     if obj is not _block and isinstance(obj, BlockData):

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\core\base\initializer.py:316, in IndexedCallInitializer.__call__(self, parent, idx)
    [314]     return self._fcn(parent, *idx)
    [315] else:
--> [316]     return self._fcn(parent, idx)

File c:\Users\hughesr\anaconda3\envs\rotary\lib\site-packages\pyomo\contrib\doe\doe.py:651, in DesignOfExperiments._create_block.<locals>.block_build(b, s)
    [648] var = cuid.find_component_on(b)
    [650] # if it is a variable, fix it to a new value
--> [651] if var.ctype is Var:
    [652]     var.fix(self.scenario_data.scenario[s][par])
    [653] # if it is a param, give it a new value

AttributeError: 'NoneType' object has no attribute 'ctype'

[ksbeattie]

@adowling2 & @Ryan-Hughes-8,

Is @jialuw96 still available to work on this? I've heard she's graduated.

[jialuw96]

Hi @Ryan-Hughes-8 Thank you for the feedbacks! I have made some changes and committed them. Please see the following for answers to your suggestions specifically:

1. Thank you for testing this. I updated the create environment command line in the README file to create the environment with Python 3.8.
2. I added the file path and added comments to the variable names_idx and total_fim.
3. Yes it was meant to be Counterflow_Jacobian_process.ipynb. I also updated the README file.
4. I have fixed the typos; It should be able to run now.
5. Can you change sensi_opt in block 6 to be direct_kaug instead of sequential_finite and try again?

Thank you for testing the code! I am keeping an eye on the repo and will make my best to answer in time.

[bpaul4]

Hi @jialuw96, thank you for your recent modifications. I attempted to run through the README, and while the installation went smoothly I encountered some issues with the notebooks and made some changes locally to resolve the issues where possible. In general, the notebooks should have some comments explaining what the Input cells are doing.

Counterflow_MO_data_process.ipynb
removed f from Input [1], [20]
added return total_fim to end of compute_trace_from_idx function
changed to total fim = compute_trace_from_idx(benchmark) - is this the correct object?
commented out sol = create_initial(names_idx) since it doesn't work - or update to the correct call

Cocurrent_flow_MBDoE.py
changed objective to objective_opt on line 490 - verify if this is correct
failed to run on line 490 calculator.solve with raise ApplicationError(pyomo.common.errors.ApplicationError: No Python bindings available for <class 'pyomo.solvers.plugins.solvers.gurobi_direct.GurobiDirect'> solver plugin

Cocurrent_flow_MBDoE
changed Input [5] to assert isinstance(m.ads.C1, pcb.param.ScalarParam)
added assert to Input [6]
not sure why cells like Input [8] are included, can we remove them?
Input [10] and [12] Objective.type doesn't exist, what is the correct attribute name?
encountered FileNotFoundError: [Errno 2] No such file or directory: './MO_QVs/Var_z3' on Input [14]
couldn't run the rest of the notebook without [14] working

draw_figure.ipynb called draw_figures in the README, notebook runs smoothly with no issues
couldn't open and run Countercurrent_MO.ipynb, it's too big to render
kaug_testcase.ipynb, package RPB_model_kaug_merge doesn't exist

[jialuw96]

Hi @bpaul4 , thank you for the changes; Please see my feedbacks in the following. Overall I found there are some file names you mentioned that I cannot find; To work on the same contents, I am working on the code in this PR on 70b173b, is there any commit in other branches or other PR?

Counterflow_MO_data_process.ipynb
removed f from Input [1], [20]
added return total_fim to end of compute_trace_from_idx function
changed to total fim = compute_trace_from_idx(benchmark) - is this the correct object?

Yes, this is the correct object and is the way to achieve total_fim.

commented out sol = create_initial(names_idx) since it doesn't work - or update to the correct call

That works; That is a debugging function and I just kept it in case.

Cocurrent_flow_MBDoE.py
changed objective to objective_opt on line 490 - verify if this is correct

I am guessing you are mentioning Countercurrent_MO.py since I didn't find Cocurrent_flow_MBDoE.py. If so, yes, this is correct.

failed to run on line 490 calculator.solve with raise ApplicationError(pyomo.common.errors.ApplicationError: No Python bindings available for <class 'pyomo.solvers.plugins.solvers.gurobi_direct.GurobiDirect'> solver plugin

This seems like a gurobi environment issue; There is a more detailed Gurobi installation guidance for this problem (Readme, step 5) here. I am guessing installing GurobiPy can solve this; This can be done by running conda install -c gurobi gurobi.

Cocurrent_flow_MBDoE
not sure why cells like Input [8] are included, can we remove them?

Yes, those were for debugging and are not used any more.

Input [10] and [12] Objective.type doesn't exist, what is the correct attribute name?

It should be m.Obj.

encountered FileNotFoundError: [Errno 2] No such file or directory: './MO_QVs/Var_z3' on Input [14]

It should be './Cocurrent_MO_QVs/Var_z3'.

couldn't open and run Countercurrent_MO.ipynb, it's too big to render

I cannot find this file; Could you check the file name please?

kaug_testcase.ipynb, package RPB_model_kaug_merge doesn't exist

I also cannot find RPB_model_kaug_merge; I am able to download kaug_testcase.ipynb and see the contents in the notebook.

Thank you for your updates!

[bpaul4]

@jialuw96 thank you for your clarifications. I ran through the README again this afternoon.

I was unable to test CountercurrentMBDoE.ipynb or kaug_test.ipynb, encountered an error about having not have enough memory to run the file from - this happened with both the browser and console commands.

Thank you for the note on Cocurrentflow_MBDoE.ipynb. The section "Measurements and their variances" should be updated to:

# Each measurement has a variance; there are no covariances between measurements 
# file contains: [Nm * 1], Nm: # of measurements, each element is the variance of the corresponding measurement
file = open("./Cocurrent_MO_QVs/Var_z3", "rb")
var_data = pickle.load(file)
file.close()

# read a MO solution file
# file contains: [Nm*Nm], sol_data[i,i] is a binary variable showing if measurement i is chosen 
file = open('./Cocurrent_results/MO_results/Sep17_2000_a', 'rb')
sol_data = pickle.load(file)
file.close()

# read file names 
# file contains: [Nm*1], each element is the name of the corresponding measurement 
file = open("./Cocurrent_MO_QVs/name_z3", 'rb')
name_data = pickle.load(file)
file.close()

In the same notebook, the section "Solve a square MBDoE problem with k_aug" fails due to what seems to be Pyomo compatibility issue:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[20], line 6
      2 doe_object = DesignOfExperiments(parameter_dict, design_gen,
      3                                  measure_class, create_model)
      5 # analyze a square MBDoE problem
----> 6 result = doe_object.compute_FIM(mode=sensi_opt, 
      7                                 FIM_store_name = 'dynamic.csv', 
      8                                 read_output=None,
      9                                 scale_nominal_param_value=True)
     12 result.result_analysis()

File ~\.conda\envs\3.10-env\lib\site-packages\pyomo\contrib\doe\doe.py:429, in DesignOfExperiments.compute_FIM(self, mode, FIM_store_name, specified_prior, tee_opt, scale_nominal_param_value, scale_constant_value, store_output, read_output, extract_single_model, formula, step, only_compute_fim_lower)
    424     FIM_analysis = self._sequential_finite(
    425         read_output, extract_single_model, store_output
    426     )
    428 elif self.mode == CalculationMode.direct_kaug:
--> 429     FIM_analysis = self._direct_kaug()
    431 dT = square_timer.toc(msg=None)
    432 self.logger.info("elapsed time: %0.1f seconds" % dT)

File ~\.conda\envs\3.10-env\lib\site-packages\pyomo\contrib\doe\doe.py:547, in DesignOfExperiments._direct_kaug(self)
    545     cuid = pyo.ComponentUID(par)
    546     var = cuid.find_component_on(mod)
--> 547     var.setlb(self.param[par])
    548     var.setub(self.param[par])
    550 # generate parameter name list and value dictionary with index

AttributeError: 'ScalarParam' object has no attribute 'setlb'

I have Pyomo version 6.7.3 (the latest version is 6.8.0).

Would you be able to make the changes discussed here and my prior comment, and test the files on your local machine?

[bpaul4]

Hello @jialuw96, have there been any updates on this?

[adowling2]

@jialuw96 has graduated and moved on from the CCSI2 team. @djlaky and I plan to revise this to use the new Pyomo.DoE interface developed over the summer

[ksbeattie]

@adowling2 this is a pretty big PR (nearly 600 files changed) and has been open for several months now. Might it be possible merge this (or some subset of it) in and not leave such large set of pending changes open on PR/branch?

[adowling2]

I like this plan. Merge it and then we can have a PR to clean it up.

…

Message ID: ***@***.*** com>