Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Admin, mapping jobs with TPV: Fix a few speling mistakes #4559

Merged
merged 3 commits into from
Dec 7, 2023
Merged

Admin, mapping jobs with TPV: Fix a few speling mistakes #4559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
93bb828
Admin: Mapping jobs with TPV: Fix a few speling mistakes
torfinnnome Nov 28, 2023
1ff339d
Replace SINGULARITY_CACHEDIR with APPTAINER_CACHEDIR
torfinnnome Dec 7, 2023
f699b5d
Merge branch 'galaxyproject:main' into patch-5
torfinnnome Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion topics/admin/tutorials/connect-to-compute-cluster/tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ At the top of the stack sits Galaxy. Galaxy must now be configured to use the cl
> + env:
> + - name: LC_ALL
> + value: C
> + - name: SINGULARITY_CACHEDIR
> + - name: APPTAINER_CACHEDIR
> + value: /tmp/singularity
> + - name: APPTAINER_TMPDIR
> + value: /tmp
Expand Down
10 changes: 5 additions & 5 deletions topics/admin/tutorials/job-destinations/tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ We want our tool to run with more than one core. To do this, we need to instruct
> - env:
> - - name: LC_ALL
> - value: C
> - - name: SINGULARITY_CACHEDIR
> - - name: APPTAINER_CACHEDIR
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These does not seem a spelling mistake. Can you please explain why are you making this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I should have made a separate PR for this. Throughout the rest of the training material, APPTAINER_ is used (and partly in this tutorial), so I assume it was just a mistake to not previously change this from SINGULARITY_.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If singularity is not used at all then the corresponding value on the next line also does not make sense and more changes are required I believe

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these variables are interpreted by Galaxy, and must be SINGULARITY (mostly job conf, e.g. singularity_enabled)

Others are read by apptainer and can safely be APPTAINER. TMPDIR is one of those that isn't read by galaxy afaict (grepping release_23.1`)

$ ag CACHEDIR
doc/source/admin/container_resolvers.rst
169:For singularity, admins should also take care of the ``APPTAINER_CACHEDIR``

cachedir it seems is too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, and in the Apptainer tutorial, both APPTAINER_CACHEDIR and APPTAINER_TMPDIR are used.

> - value: /tmp/singularity
> - - name: APPTAINER_TMPDIR
> - value: /tmp
Expand Down Expand Up @@ -342,7 +342,7 @@ We want our tool to run with more than one core. To do this, we need to instruct
> + # Ensuring a consistent collation environment is good for reproducibility.
> + LC_ALL: C
> + # The cache directory holds the docker containers that get converted
> + SINGULARITY_CACHEDIR: /tmp/singularity
> + APPTAINER_CACHEDIR: /tmp/singularity
> + # Singularity uses a temporary directory to build the squashfs filesystem
> + APPTAINER_TMPDIR: /tmp
> + slurm:
Expand Down Expand Up @@ -437,7 +437,7 @@ Now that we've configured the resource requirements for a single tool, let's see
> {: data-commit="Add TPV default inherits"}
>
> We have defined a `global` section specifying that all tools and destinations should inherit from a specified `default`. We have then defined a tool named `default`, whose properties
> are implicitly inherited by all tools at runtime. This means that our `testing` tool will also inherit from this default tool, but it explicitly overrides cores
> are implicitly inherited by all tools at runtime. This means that our `testing` tool will also inherit from this default tool, but it explicitly overrides cores.
> We can also explicitly specify an `inherits` clause if we wish to extend a specific tool or destination, as previously shown in the destinations section.
>
> 2. Run the Galaxy playbook. When the new `tpv_rules_local.yml` is copied, TPV will automatically pickup the changes without requiring a restart of Galaxy.
Expand Down Expand Up @@ -508,8 +508,8 @@ on settings that have worked well in the usegalaxy.* federation. The rule file c
> {: data-commit="TPV clamp max cores and mem"}
>
> These changes indicate that the destination will accept jobs that are up to `max_accepted_cores: 24` and `max_accepted_mem: 256`. If the tool requests resources that exceed these limits, the tool will be rejected
> by the destination. However, once accepted, the resources will be forcibly clamped down to 16 and 128 at most because of the `max_cores` and `max_mem` clauses. (E.g. a tool requesting 24 cores would only be submitted with 16 cores at maximum.) Therefore, a trick that can be used here to support
> job resource requirements in the shared database that are much larger than your destination can actually support, is to combine `max_accepted_cores/mem/gpus with `max_cores/mem/gpus` to accept the job and then
> by the destination. However, once accepted, the resources will be forcibly clamped down to 2 and 8 at most because of the `max_cores` and `max_mem` clauses. (E.g. a tool requesting 24 cores would only be submitted with 16 cores at maximum.) Therefore, a trick that can be used here to support
> job resource requirements in the shared database that are much larger than your destination can actually support, is to combine `max_accepted_cores/mem/gpus` with `max_cores/mem/gpus` to accept the job and then
> clamp it down to a supported range. This allows even the largest resource requirement in the shared database to be accomodated.
>
> > <comment-title>Clamping in practice</comment-title>
Expand Down
Loading