Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RasterReader and RasterStackReader implementations #188

Merged
merged 22 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
78f26b9
add `DatasetReader/Writer` protocols
scottstanie Jan 5, 2024
9239724
add working binary stack reader
scottstanie Jan 7, 2024
937695e
add rasterio as dep
scottstanie Jan 9, 2024
d4b27d3
start implementing the 3d reader
scottstanie Jan 9, 2024
b8b666a
Merge branch 'main' into raster-reader
scottstanie Jan 9, 2024
fcacd56
fix docstyle, allow 1 or 3 slices for stack
scottstanie Jan 9, 2024
a71f558
start getting binary file test working
scottstanie Jan 10, 2024
e9dac7f
start hdf5 stack, add binary slices test
scottstanie Jan 10, 2024
9b6a8eb
add similar hdf5 tests
scottstanie Jan 10, 2024
4c4f8e7
add general gdal raster reader
scottstanie Jan 10, 2024
840b959
start test
scottstanie Jan 10, 2024
cdee6ea
add `BaseStackReader` for common features
scottstanie Jan 10, 2024
bf6ec79
add `num_threads` to other readers
scottstanie Jan 10, 2024
e2eca6b
fix bad typing
scottstanie Jan 10, 2024
020e8cd
remove `get_max_block_shape`. switch to `StackReader` protocol for `E…
scottstanie Jan 10, 2024
5fc7cc1
reorg tests into classes
scottstanie Jan 10, 2024
4da7635
reorder locations of background loaders
scottstanie Jan 10, 2024
e849f8b
use `StackReader` in `ps.create_ps`, drop `.iter_blocks`
scottstanie Jan 10, 2024
af009c3
fix loading for new `StackReader`, fix `...` as band index
scottstanie Jan 10, 2024
a896abd
fix tests for EagerLoader refactor
scottstanie Jan 10, 2024
6abbae9
bad math
scottstanie Jan 10, 2024
4e3bbcb
update changelog for new version. Close #174 [skip ci]
scottstanie Jan 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/test-build-push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
pydantic=2.1
pymp-pypi=0.4.5
pyproj=3.3
rasterio=1.3
rich=12.0
ruamel_yaml=0.15
scipy=1.5
Expand Down
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
# [Unreleased](https://github.com/isce-framework/dolphin/compare/v0.8.0...main)
# [Unreleased](https://github.com/isce-framework/dolphin/compare/v0.9.0...main)

# [v0.9.0](https://github.com/isce-framework/dolphin/compare/v0.8.0...v0.9.0)

**Added**
- `DatasetReader` and `StackReader` protocols for reading in data from different sources
- `DatasetReader` is for reading in a single dataset, like one raster image.
- `StackReader` is for reading in a stack of datasets, like a stack of SLCs.
- Implementations of these have been done for flat binary files (`BinaryReader`), HDF5 files (`HDF5Reader`), and GDAL rasters (`RasterReader`).

**Changed**
- The `VRTStack` no longer has an `.iter_blocks` method
- This has been replaced with creating an `EagerLoader` directly and passing it to the `reader` argument

# [v0.8.0](https://github.com/isce-framework/dolphin/compare/v0.7.0...v0.8.0)

Expand Down
1 change: 1 addition & 0 deletions conda-env.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ dependencies:
- pydantic>=2.1
- pymp-pypi>=0.4.5
- pyproj>=3.3
- rasterio>=1.3
- rich>=12.0
- ruamel.yaml>=0.15
- scipy>=1.5 # "scipy 0.16+ is required for linear algebra", numba. 1.5 is the oldest version that supports Python 3.7
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ theme:
plugins:
- search
# plugin suggestions from here: https://mkdocstrings.github.io/recipes/
- autorefs
- gen-files:
scripts:
- docs/gen_ref_pages.py
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ plugins = ["pydantic.mypy"]

[tool.pytest.ini_options]
doctest_optionflags = "NORMALIZE_WHITESPACE NUMBER"
addopts = " --cov=dolphin -n auto --maxprocesses=8 --doctest-modules --randomly-seed=1234 --ignore=scripts --ignore=docs --ignore=data --ignore=pkgs"
# addopts = " --cov=dolphin -n auto --maxprocesses=8 --doctest-modules --randomly-seed=1234 --ignore=scripts --ignore=docs --ignore=data --ignore=pkgs"
addopts = " --doctest-modules --randomly-seed=1234 --ignore=scripts --ignore=docs --ignore=data --ignore=pkgs"
filterwarnings = [
"error",
# DeprecationWarning thrown in pkg_resources for older numba verions and llvmlite
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ opera-utils>=0.1.5
pydantic>=2.1
pymp-pypi>=0.4.5
pyproj>=3.3
rasterio>=1.3
rich>=12.0
ruamel_yaml>=0.15
scipy>=1.5
Expand Down
7 changes: 7 additions & 0 deletions src/dolphin/_background.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@

_DEFAULT_TIMEOUT = 0.5

__all__ = [
"BackgroundWorker",
"BackgroundReader",
"BackgroundWriter",
"DummyProcessPoolExecutor",
]


class BackgroundWorker(abc.ABC):
"""Base class for doing work in a background thread.
Expand Down
Loading