Version/1.11.0 (#449)

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
LLNL · Oct 9, 2023 · 093c867 · 093c867
1 parent faf71ed
commit 093c867
Show file tree

Hide file tree

Showing 59 changed files with 371 additions and 144 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,18 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.11.0]
+### Added
+- New reserved variable:
+  - `VLAUNCHER`: The same functionality as the `LAUNCHER` variable, but will substitute shell variables `MERLIN_NODES`, `MERLIN_PROCS`, `MERLIN_CORES`, and `MERLIN_GPUS` for nodes, procs, cores per task, and gpus
+
+### Changed
+- Hardcoded Sphinx v5.3.0 requirement is now removed so we can use latest Sphinx
+
+### Fixed
+- A bug where the filenames in iterative workflows kept appending `.out`, `.partial`, or `.expanded` to the filenames stored in the `merlin_info/` subdirectory
+- A bug where a skewed sample hierarchy was created when a restart was necessary in the `add_merlin_expanded_chain_to_chord` task
+
 ## [1.10.3]
 ### Added
 - The *.conf regex for the recursive-include of the merlin server directory so that pip will add it to the wheel

diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/docs/requirements.in b/docs/requirements.in
@@ -0,0 +1,4 @@
+# This file will list all requirements for the docs so we can freeze a version of them for release.
+# To freeze the versions run:
+#       pip-compile requirements.in
+sphinx
diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -1,56 +1 @@
-#
-# This file is autogenerated by pip-compile with python 3.8
-# To update, run:
-#
-#    pip-compile requirements.in
-#
-alabaster==0.7.12
-    # via sphinx
-babel==2.10.3
-    # via sphinx
-certifi==2023.7.22
-    # via requests
-charset-normalizer==2.1.1
-    # via requests
-docutils==0.17.1
-    # via sphinx
-idna==3.4
-    # via requests
-imagesize==1.4.1
-    # via sphinx
-importlib-metadata==5.0.0
-    # via sphinx
-jinja2==3.0.3
-    # via sphinx
-markupsafe==2.1.1
-    # via jinja2
-packaging==21.3
-    # via sphinx
-pygments==2.15.0
-    # via sphinx
-pyparsing==3.0.9
-    # via packaging
-pytz==2022.5
-    # via babel
-requests==2.31.0
-    # via sphinx
-snowballstemmer==2.2.0
-    # via sphinx
-sphinx==5.3.0
-    # via -r requirements.in
-sphinxcontrib-applehelp==1.0.2
-    # via sphinx
-sphinxcontrib-devhelp==1.0.2
-    # via sphinx
-sphinxcontrib-htmlhelp==2.0.0
-    # via sphinx
-sphinxcontrib-jsmath==1.0.1
-    # via sphinx
-sphinxcontrib-qthelp==1.0.3
-    # via sphinx
-sphinxcontrib-serializinghtml==1.1.5
-    # via sphinx
-urllib3==1.26.12
-    # via requests
-zipp==3.10.0
-    # via importlib-metadata
+sphinx>=5.3.0
diff --git a/docs/source/merlin_variables.rst b/docs/source/merlin_variables.rst
@@ -160,8 +160,9 @@ Reserved variables
 
             $(MERLIN_INFO)/*.expanded.yaml
 
-The ``LAUNCHER`` Variable
-+++++++++++++++++++++++++
+
+The ``LAUNCHER`` and ``VLAUNCHER`` Variables
++++++++++++++++++++++++++++++++++++++++++++++++
 
 ``$(LAUNCHER)`` is a special case of a reserved variable since it's value *can* be changed.
 It serves as an abstraction to launch a job with parallel schedulers like :ref:`slurm<slurm>`,
@@ -187,6 +188,54 @@ We can modify this to use the ``$(LAUNCHER)`` variable like so:
 
 In other words, the ``$(LAUNCHER)`` variable would become ``srun -N 1 -n 3``.
 
+Similarly, the ``$(VLAUNCHER)`` variable behaves similarly to the ``$(LAUNCHER)`` variable.
+The key distinction lies in its source of information. Instead of drawing certain configuration
+options from the ``run`` section of a step, it retrieves specific shell variables. These shell
+variables are automatically generated by Merlin when you include the ``$(VLAUNCHER)`` variable
+in a step command, but they can also be customized by the user. Currently, the following shell
+variables are:
+
+.. list-table:: VLAUNCHER Variables
+   :widths: 25 50 25
+   :header-rows: 1
+
+   * - Variable
+     - Description
+     - Default
+
+   * - ``${MERLIN_NODES}``
+     - The number of nodes
+     - 1
+
+   * - ``${MERLIN_PROCS}``
+     - The number of tasks/procs
+     - 1
+
+   * - ``${MERLIN_CORES}``
+     - The number of cores per task/proc
+     - 1
+
+   * - ``${MERLIN_GPUS}``
+     - The number of gpus per task/proc
+     - 0
+
+Let's say we have the following defined in our yaml file:
+
+.. code:: yaml
+
+    batch:
+        type: flux
+
+    run:
+        cmd: |
+          MERLIN_NODES=4
+          MERLIN_PROCS=2
+          MERLIN_CORES=8
+          MERLIN_GPUS=2
+          $(VLAUNCHER) python script.py
+
+The ``$(VLAUNCHER)`` variable would be substituted to ``flux run -N 4 -n 2 -c 8 -g 2``.
+
 User variables
 -------------------
 Variables defined by a specification file in the ``env`` section, as in this example:

diff --git a/merlin/__init__.py b/merlin/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.10.3"
+__version__ = "1.11.0"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

diff --git a/merlin/ascii_art.py b/merlin/ascii_art.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/celery.py b/merlin/celery.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/__init__.py b/merlin/common/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/__init__.py b/merlin/common/abstracts/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/enums/__init__.py b/merlin/common/abstracts/enums/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/openfilelist.py b/merlin/common/openfilelist.py
@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/opennpylib.py b/merlin/common/opennpylib.py
@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index.py b/merlin/common/sample_index.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index_factory.py b/merlin/common/sample_index_factory.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/__init__.py b/merlin/common/security/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt.py b/merlin/common/security/encrypt.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt_backend_traffic.py b/merlin/common/security/encrypt_backend_traffic.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/tasks.py b/merlin/common/tasks.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -298,27 +298,33 @@ def add_merlin_expanded_chain_to_chord(  # pylint: disable=R0913,R0914
         LOG.debug("chain added to chord")
     else:
         # recurse down the sample_index hierarchy
-        LOG.debug("recursing down sample_index hierarchy")
-        for next_index in sample_index.children.values():
-            next_index.name = os.path.join(sample_index.name, next_index.name)
-            LOG.debug("generating next step")
-            next_step = add_merlin_expanded_chain_to_chord.s(
-                task_type,
-                chain_,
-                samples[next_index.min - min_sample_id : next_index.max - min_sample_id],
-                labels,
-                next_index,
-                adapter_config,
-                next_index.min,
-            )
-            next_step.set(queue=chain_[0].get_task_queue())
-            LOG.debug(f"recursing with range {next_index.min}:{next_index.max}, {next_index.name} {signature(next_step)}")
-            LOG.debug(f"queuing samples[{next_index.min}:{next_index.max}] in for {chain_} in {next_index.name}...")
-            if self.request.is_eager:
-                next_step.delay()
-            else:
-                self.add_to_chord(next_step, lazy=False)
-            LOG.debug(f"queued for samples[{next_index.min}:{next_index.max}] in for {chain_} in {next_index.name}")
+        try:
+            LOG.debug("recursing down sample_index hierarchy")
+            for next_index in sample_index.children.values():
+                next_index_name_before = next_index.name
+                next_index.name = os.path.join(sample_index.name, next_index.name)
+                LOG.debug("generating next step")
+                next_step = add_merlin_expanded_chain_to_chord.s(
+                    task_type,
+                    chain_,
+                    samples[next_index.min - min_sample_id : next_index.max - min_sample_id],
+                    labels,
+                    next_index,
+                    adapter_config,
+                    next_index.min,
+                )
+                next_step.set(queue=chain_[0].get_task_queue())
+                LOG.debug(f"recursing with range {next_index.min}:{next_index.max}, {next_index.name} {signature(next_step)}")
+                LOG.debug(f"queuing samples[{next_index.min}:{next_index.max}] in for {chain_} in {next_index.name}...")
+                if self.request.is_eager:
+                    next_step.delay()
+                else:
+                    self.add_to_chord(next_step, lazy=False)
+                LOG.debug(f"queued for samples[{next_index.min}:{next_index.max}] in for {chain_} in {next_index.name}")
+        except retry_exceptions as e:
+            # Reset the index to what it was before so we don't accidentally create a bunch of extra samples upon restart
+            next_index.name = next_index_name_before
+            raise e
 
     return ReturnCode.OK
 

diff --git a/merlin/common/util_sampling.py b/merlin/common/util_sampling.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/__init__.py b/merlin/config/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/broker.py b/merlin/config/broker.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/celeryconfig.py b/merlin/config/celeryconfig.py
@@ -10,7 +10,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/configfile.py b/merlin/config/configfile.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.3.
+# This file is part of Merlin, Version: 1.11.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #