Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread metadata objects through pipelines #607

Open
sfiruch opened this issue Aug 8, 2024 · 1 comment
Open

Thread metadata objects through pipelines #607

sfiruch opened this issue Aug 8, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@sfiruch
Copy link
Member

sfiruch commented Aug 8, 2024

Scenario

Bob reads .vis files from MWA. These .vis files contain metadata, like provenance ("these visibilities were measured by MWA"). Bob runs the visibilities through imagers.

Actual Outcome

The resulting Karabo image does not contain information where the data came from.

Expected Outcome

The resulting Karabo image contains metadata information ("this image was generated from MWA visibilities").

Proposed Design

  1. All Karabo objects contain an (perhaps initially untyped?) "metadata" property
  2. All functions that create a new object, e.g. creating an image from visibilities, assign the metadata object from its sources.
  3. Code/frameworks that write external files should try to include as much information from the metadata object in the written files.

For future versions, the functions mentioned in #2 should try to update/change/convert metadata for the respective data product.

@sfiruch sfiruch added the enhancement New feature or request label Aug 8, 2024
@sfiruch sfiruch changed the title Thread metadata objects through a pipeline Thread metadata objects through pipelines Aug 8, 2024
@Lukas113
Copy link
Collaborator

Lukas113 commented Aug 8, 2024

I think we should put some thoughts into something like that before introducing:

  • What's the purpose of this (helps in identifying which metadata needs to be passed through & how to handle information from different backends)
  • Defining a versioned and typed metadata format (anything else is not really maintainable and makes things very complicated IMO)
  • API-design to read/write for files & code, which should be easy to use & maintain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants