RTX Code Release (PI-3): Enhancements to Replace and Route to support enterprise hydrofabric and new NWPS api capabilities in near-real time

.gitattributes

@@ -19,3 +19,5 @@
 *.tgz binary
 *.tar.gz binary
 *.gz binary
+*.gpkg binary
+*.nc binary

.github/workflows/pytest_rnr.yml

@@ -0,0 +1,57 @@
+name: PyTest Suite
+
+on: [pull_request]
+
+jobs:
+  # lint_and_format:
+  #   runs-on: ubuntu-latest
+
+  #   steps:
+  #   - uses: actions/checkout@v4
+  #   - uses: actions/setup-python@v5
+  #     with:
+  #       python-version: "3.11"
+
+  #   - name: Install pre-commit
+  #     run: |
+  #         python -m pip install pre-commit
+
+  #   - name: Run pre-commit hooks on all files
+  #     run: |
+  #         pre-commit run --all-files
+
+  build:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11"]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Set UV_HTTP_TIMEOUT
+      run: echo "UV_HTTP_TIMEOUT=600" >> $GITHUB_ENV
+
+    - name: Install python dependencies and packages
+      run: |
+        python -m pip install --upgrade pip
+        pip install uv
+        uv venv
+        source .venv/bin/activate
+        uv pip install -r Source/RnR/requirements.txt
+
+    - name: Echo package versions
+      run: |
+        source .venv/bin/activate
+        uv pip freeze
+
+    - name: Run unit tests
+      run: |
+        source .venv/bin/activate
+        cd Source/RnR
+        pytest -s

.gitignore

@@ -50,4 +50,26 @@ terraform.rc
 **/buildplan
 **/destroyplan
 
-Dev/env.yaml
+Dev/env.yaml
+
+###################
+# Replace and Route
+###################
+Source/RnR/.venv
+Source/RnR/src/rnr/app/core/config.ini
+Source/RnR/mock_db/rnr_schema_dump_20240612.dump
+Source/RnR/data
+Source/RnR/output
+Source/RnR/dist
+Source/RnR/notebooks/historical_flood_event
+Source/RnR/.Trash*
+
+*.ruff_cache
+*.pytest_cache
+
+*.pyc
+*.egg-info/
+__pycache__/
+*.py[cod]
+.env
+.python-version

Source/RnR/CHANGELOG.md

@@ -0,0 +1,80 @@
+All notable changes to this project will be documented in this file.
+We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
+
+## 1.0.0 2024-08-29
+
+### Added
+
+- completed the pi-3 demo for OWP. this includes
+    - the publish/start endpoint
+    - the full data consumer
+    - the IaC component
+
+
+## 0.0.3 2024-08-20
+
+### Added
+
+- migrated to github
+- added references to t-route and hfsubset
+
+
+## 0.0.2 2024-08-12
+
+### Added
+
+- `/static` libraries for JS and CSS
+- `/data` directiories now are linked to the 
+
+### Deprecated
+
+- Nothing.
+
+### Removed
+
+- Reformatted the working dir to format application under one dir
+
+### Fixed
+
+- Nothing.
+
+
+## 0.0.2 2024-08-2
+
+### Added
+
+- the `consumer/` folder to handle all data services related to the consumer (similar to the tools/ dir from PI-2)
+
+### Deprecated
+
+- Nothing.
+
+### Removed
+
+- Reformatted the working dir to format application under one dir
+
+### Fixed
+
+- Nothing.
+
+
+## 0.0.1 2024-07-17
+
+### Added
+
+- Added microservices to read a DB of RFC information
+- Commented/documented microservices that read the NWPS API
+- Commented/documented microservices that read the DB
+- Added relevant tests and project dependencies
+
+### Deprecated
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+### Fixed
+
+- Nothing.

Source/RnR/Dockerfile.app

@@ -0,0 +1,15 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+COPY src/ /app/src
+
+COPY requirements.txt ./
+COPY pyproject.toml ./
+COPY README.md ./
+
+RUN pip install uv
+RUN uv venv
+RUN uv pip install --no-cache-dir -r requirements.txt
+
+ENV PYTHONPATH=/app:$PYTHONPATH

Source/RnR/Dockerfile.notebook

@@ -0,0 +1,11 @@
+FROM jupyter/minimal-notebook
+
+WORKDIR /app
+
+COPY src /app/src
+
+COPY requirements_notebook.txt .
+COPY requirements.txt .
+
+RUN pip install --no-cache-dir -r requirements.txt
+RUN pip install --no-cache-dir -r requirements_notebook.txt

Source/RnR/LICENSE

@@ -0,0 +1,7 @@
+“Software code created by U.S. Government employees is not subject to copyright
+in the United States (17 U.S.C. §105). The United States/Department of Commerce
+reserve all rights to seek and obtain copyright protection in countries other
+than the United States for Software authored in its entirety by the Department
+of Commerce. To this end, the Department of Commerce hereby grants to Recipient
+a royalty-free, nonexclusive license to use, copy, and create derivative works
+of the Software outside of the United States.”

Source/RnR/README.md

@@ -0,0 +1,120 @@
+# Replace and Route
+
+A near real-time system that leverages the NWM data assimilation and channel routing capabilities to extend official flood forecasts issued by RFCs to all river reaches downstream of RFC forecast points by routing the forecasts through the river network to the next downstream forecast point. 
+
+Key Features of Replace and Route:
+- Deploys a message broker architecture to process incoming RFC forecasts
+- Utilizes v20.1 of the enterprise hydrofabric
+- Routes forecasted flow to downstream RFC locations 
+- Uses docker compose to manage architecture
+- Uses a jupyter-notebook to view and manage code through a container
+
+<img src="docs/photos/API_spec.png" alt="isolated" width="750"/>
+
+## Installation
+
+1. Install a virtual environment and download dependencies
+
+```shell
+pip install uv
+uv venv
+source .venv/bin/activate
+uv pip install -r requirements.txt
+```
+
+2. Make sure docker compose is installed on your system
+- https://docs.docker.com/compose/install/
+
+3. Start the service
+```shell
+docker compose up
+```
+
+4. View swagger docs for the services
+
+Each of the URLS corresponds to one of the services:
+
+- Publisher app:
+  - localhost:8000/docs
+- Front-end:
+  - localhost:8001/frontend/v1/plot
+- T-route
+  - localhost:8004/docs
+- hfsubset
+  - localhost:8008/docs
+- jupyter
+  - localhost:8888/lab
+
+
+## Dependencies
+
+The following containers are utilized 
+1. RabbitMQ
+  - rabbitmq:3.13-management
+2. Mock DB
+  - ghcr.io/taddyb33/hydrovis/mock_database:0.0.1
+3. HFsubset
+  - ghcr.io/taddyb33/hfsubset-legacy:0.0.4
+4. T-Route
+  - ghcr.io/taddyb33/t-route-dev:0.0.2
+5. Redis
+  - redis:7.2.5
+
+## Usage
+
+Follow the installation to get to the API services. 
+
+To download RFC data based on the mock DB LIDS, run the following command once the DB is up:
+
+```shell
+curl -X 'POST' \
+  'http://localhost:8000/api/v1/rfc/build_rfc_geopackages' \
+  -H 'accept: application/json' \
+  -d ''
+```
+
+## How to test the software
+
+Run the following command from the project's home dir
+```sh
+pytest -s
+```
+
+## Known issues
+
+- Since there are only 170 locations in the mock DB, this is just a subset of all locations. So, there will be errors if there is no found LID in the DB (This happens oftern)
+
+## Getting help
+
+If you have questions, concerns, bug reports, etc, please file an issue in this repository's Issue Tracker.
+
+## Getting involved
+
+This section should detail why people should get involved and describe key areas you are
+currently focusing on; e.g., trying to get feedback on features, fixing certain bugs, building
+important pieces, etc.
+
+General instructions on _how_ to contribute should be stated with a link to [CONTRIBUTING](CONTRIBUTING.md).
+
+
+----
+
+## Open source licensing info
+1. [TERMS](TERMS.md)
+2. [LICENSE](LICENSE)
+
+
+----
+
+## Credits and References
+
+Credits to the following developers:
+- Tadd Bindas @taddyb33
+- David Martin @david-w-martin
+
+Also, thank you to all OWP members / collaborators
+- Shawn Crawley
+- Fernando Salas
+- Derek G
+- Nels Frazier (T-Route dev)
+- Mike Johnson (Hydrofabric dev)