You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running reanalyses, PSL would like to have the ability to push and archive data outputs to a public AWS S3 bucket from NOAA HPC resources. Ideally, we'd like to push this data to both HPSS (tarball) and AWS(raw). I'm working with Phil Pegion, Jeff Whitaker, and Ding Liu on setting up this process.
I believe this is a similar request to #2872 , but that issue is more focused on the entire workflow running on AWS or other CSPs. We'd like to be able to do this when running the workflow from NOAA resources.
What are the requirements for the new functionality?
Ability to configure and push data from AWS S3 public buckets. Users should be able to choose if they want to push data to AWS, HPSS, or both. The workflow should be able to handle this new functionality from NOAA HPC resources.
Acceptance Criteria
Workflow can be set up to upload data to remote AWS public bucket
User can choose to archive data at HPSS(tarball) and/or AWS bucket (raw)
Workflow runs on NOAA resources
Suggest a solution (optional)
/ush/python/pygfs/task/archive.py already handles data archiving to HPSS. We would like the ability to choose AWS as an archive here. I think this would align with Walter Kolczynski's suggestion in #2873 .
The text was updated successfully, but these errors were encountered:
What new functionality do you need?
When running reanalyses, PSL would like to have the ability to push and archive data outputs to a public AWS S3 bucket from NOAA HPC resources. Ideally, we'd like to push this data to both HPSS (tarball) and AWS(raw). I'm working with Phil Pegion, Jeff Whitaker, and Ding Liu on setting up this process.
I believe this is a similar request to #2872 , but that issue is more focused on the entire workflow running on AWS or other CSPs. We'd like to be able to do this when running the workflow from NOAA resources.
What are the requirements for the new functionality?
Ability to configure and push data from AWS S3 public buckets. Users should be able to choose if they want to push data to AWS, HPSS, or both. The workflow should be able to handle this new functionality from NOAA HPC resources.
Acceptance Criteria
Suggest a solution (optional)
/ush/python/pygfs/task/archive.py already handles data archiving to HPSS. We would like the ability to choose AWS as an archive here. I think this would align with Walter Kolczynski's suggestion in #2873 .
The text was updated successfully, but these errors were encountered: