You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dear all,
this project has grown a lot during the past few years. This process can be seen in many parts of this project which contain code of several different generations. Even the most basic class stream contains a lot of now unused stuff, which hampers improving the code and adapting it to new standards. The amount of dependencies has grown as well and some of them are not well or even not supported any more. There are a lot of interconnections between actually independent parts of this code which should be separated more clearly. Format libraries contain multiple descriptions/identifiers, and many methods don't have reasonable descriptions....
After a very fruitful discussion with Jeremy, I would suggest to start with a new clean copy of the most relevant methods focusing on python 3 support in order to clean up this mess. Some thoughts on that are listed below. Regarding the existing MagPy code, I would have a couple of important improvements coming up with the yearly analysis 2018 of our observatory and the analysis of the IAGA workshop results which I would include into 0.4.7, important updates from Jeremey and Heather, and some windows issues which I would include into 0.4.8. Afterwards I would like to mainly focus on a new clean project using proper semantic descriptions. The complete data cycle of the Conrad Observatory, from acquisition to dissemination, is controlled/supported by MagPy. Therefore my primary interest is to keep that working and gradually replace the most important features for this purpose. Please add you thoughts, comments and interest:
New MagPy: working title "magpy3"
Focus on supporting python >= 3.x
Improve test environment and implement travis/azura or similar
Clearly separate all program blocks (e.g. credentials, acquisition, etc)
and clean up dependencies
Primary focus (for RL):
general read and write methods
all methods required for automatic analysis
all methods necessary for supporting MARTAS at the Conrad Observatory (WIC)
all methods necessary for database communication (required for automatic analysis WIC)
generalize methods: options are provided as a dictionary
config method -> obtain configuration information for automatic jobs (read conf data and return dictionary)
remove unnecessary dependencies
Check for alternative dependencies:
cdflib instead of spacepy
Suggested structure:
stream.py -> timeseries class
flag.py -> flagging class
absolutes.py -> absolute class for DI measurements
/database
mysql.py -> methods for mysql communication
etc
/lib
format definitions (IMPORTANT: each format.py contains only one format definition)
skeleton: each format contains isFormat, readFormat, writeFormat
/plot
mpplot.py -> matplotlib plotting class
/gui
decide on suitable gui: wxpython4?
Underlying packages/dependencies:
use pandas ?
The text was updated successfully, but these errors were encountered:
Thanks for starting this thread Roman, I'm definitely interested in ways to make it easier to reuse functionality from MagPy, and hope to hear from others.
I propose we focus two aspects that are somewhat simpler to start: flags and absolutes. Specifically:
data models that are well documented, and
interfaces for reading and writing these from a database.
web services and a JSON format for these information
As long as there are interface methods (e.g. get_flags(startime, endtime, sensor) -> List[Flag]), there can be multiple implementations as needed; one for the existing magpy2 schema, and others for alternate schemas or formats.
Taking a data-first approach with interfaces and web services also allows us to separate any GUI/CLI interface from a backend services, at least optionally, and makes it easier to reuse the services across multiple applications.
We can focus on some of the automation fundamentals and incrementally add features without affecting the existing code base, setting up a migration path for additional data types and features.
Dear all,
this project has grown a lot during the past few years. This process can be seen in many parts of this project which contain code of several different generations. Even the most basic class stream contains a lot of now unused stuff, which hampers improving the code and adapting it to new standards. The amount of dependencies has grown as well and some of them are not well or even not supported any more. There are a lot of interconnections between actually independent parts of this code which should be separated more clearly. Format libraries contain multiple descriptions/identifiers, and many methods don't have reasonable descriptions....
After a very fruitful discussion with Jeremy, I would suggest to start with a new clean copy of the most relevant methods focusing on python 3 support in order to clean up this mess. Some thoughts on that are listed below. Regarding the existing MagPy code, I would have a couple of important improvements coming up with the yearly analysis 2018 of our observatory and the analysis of the IAGA workshop results which I would include into 0.4.7, important updates from Jeremey and Heather, and some windows issues which I would include into 0.4.8. Afterwards I would like to mainly focus on a new clean project using proper semantic descriptions. The complete data cycle of the Conrad Observatory, from acquisition to dissemination, is controlled/supported by MagPy. Therefore my primary interest is to keep that working and gradually replace the most important features for this purpose. Please add you thoughts, comments and interest:
New MagPy: working title "magpy3"
Focus on supporting python >= 3.x
Improve test environment and implement travis/azura or similar
Clearly separate all program blocks (e.g. credentials, acquisition, etc)
and clean up dependencies
Primary focus (for RL):
config method -> obtain configuration information for automatic jobs (read conf data and return dictionary)
Check for alternative dependencies:
Suggested structure:
/database
/lib
skeleton: each format contains isFormat, readFormat, writeFormat
/plot
/gui
Underlying packages/dependencies:
The text was updated successfully, but these errors were encountered: