Cleaning up MagPy

[leonro]

Dear all,
this project has grown a lot during the past few years. This process can be seen in many parts of this project which contain code of several different generations. Even the most basic class stream contains a lot of now unused stuff, which hampers improving the code and adapting it to new standards. The amount of dependencies has grown as well and some of them are not well or even not supported any more. There are a lot of interconnections between actually independent parts of this code which should be separated more clearly. Format libraries contain multiple descriptions/identifiers, and many methods don't have reasonable descriptions....

After a very fruitful discussion with Jeremy, I would suggest to start with a new clean copy of the most relevant methods focusing on python 3 support in order to clean up this mess. Some thoughts on that are listed below. Regarding the existing MagPy code, I would have a couple of important improvements coming up with the yearly analysis 2018 of our observatory and the analysis of the IAGA workshop results which I would include into 0.4.7, important updates from Jeremey and Heather, and some windows issues which I would include into 0.4.8. Afterwards I would like to mainly focus on a new clean project using proper semantic descriptions. The complete data cycle of the Conrad Observatory, from acquisition to dissemination, is controlled/supported by MagPy. Therefore my primary interest is to keep that working and gradually replace the most important features for this purpose. Please add you thoughts, comments and interest:

New MagPy: working title "magpy3"

Focus on supporting python >= 3.x

Improve test environment and implement travis/azura or similar

Clearly separate all program blocks (e.g. credentials, acquisition, etc)
and clean up dependencies

Primary focus (for RL):

• general read and write methods
• all methods required for automatic analysis
• all methods necessary for supporting MARTAS at the Conrad Observatory (WIC)
• all methods necessary for database communication (required for automatic analysis WIC)
• generalize methods: options are provided as a dictionary
  config method -> obtain configuration information for automatic jobs (read conf data and return dictionary)
• remove unnecessary dependencies

Check for alternative dependencies:

• cdflib instead of spacepy

Suggested structure:

• stream.py -> timeseries class
• flag.py -> flagging class
• absolutes.py -> absolute class for DI measurements

/database

• mysql.py -> methods for mysql communication
• etc
  /lib
• format definitions (IMPORTANT: each format.py contains only one format definition)
  skeleton: each format contains isFormat, readFormat, writeFormat
  /plot
• mpplot.py -> matplotlib plotting class

/gui

• decide on suitable gui: wxpython4?

Underlying packages/dependencies:

• use pandas ?

[jmfee-usgs]

Hello,

Thanks for starting this thread Roman, I'm definitely interested in ways to make it easier to reuse functionality from MagPy, and hope to hear from others.

I propose we focus two aspects that are somewhat simpler to start: flags and absolutes. Specifically:

• data models that are well documented, and
• interfaces for reading and writing these from a database.
• web services and a JSON format for these information

As long as there are interface methods (e.g. get_flags(startime, endtime, sensor) -> List[Flag]), there can be multiple implementations as needed; one for the existing magpy2 schema, and others for alternate schemas or formats.

Taking a data-first approach with interfaces and web services also allows us to separate any GUI/CLI interface from a backend services, at least optionally, and makes it easier to reuse the services across multiple applications.

We can focus on some of the automation fundamentals and incrementally add features without affecting the existing code base, setting up a migration path for additional data types and features.