Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start adding S3 support for VRTStack #245

Merged
merged 12 commits into from
May 29, 2024

Conversation

scottstanie
Copy link
Collaborator

@scottstanie scottstanie commented Mar 2, 2024

Basic support for reading from S3 added, with a focus on the VRTStack. Allows us to do

s3_urls = ['s3://my-bucket/myfile.tif', ...]
r = io.VRTStack(s3_urls, outfile='stack_on_s3.vrt')

The PathOrStr type has been updated to include the GeneralPath protocol which describes the interface for our minimal S3Path.
Note that I considered adding some optional dependency like https://github.com/fsspec/universal_pathlib or https://github.com/liormizr/s3path or ... (there seem to be dozens of attempts). I thought we'd hold off until there's a strong need to to more file maniuplations beyond s3path.exists().

TODO: enable or tune some of these options if we see slow performance for block iteration https://developmentseed.org/titiler/advanced/performance_tuning/

@scottstanie scottstanie added the work in progress WIP: Don't merge yet label Mar 2, 2024
@scottstanie scottstanie merged commit e4e9369 into isce-framework:main May 29, 2024
5 checks passed
@scottstanie scottstanie deleted the s3-vrt-stack branch May 29, 2024 02:42
@scottstanie scottstanie removed the work in progress WIP: Don't merge yet label May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant