Allow user-defined runtime workflow

[benbovy]

NOTE: these are preliminary notes.

For some cases like time-independent simulations or runge-kutta time stepping schemes, the runtime workflow that is currently defined in the framework, i.e., initialize+master_clock*(run_step+finalize_step)+finalize may not be the most appropriate. More flexibility would be nice.

Any runtime workflow may actually be defined in a very abstract manner, i.e., as multiple passes through the whole DAG of processes in a model, each pass executing a specific method implemented in each process class (or doing nothing if the process don't implement that method). The implementation of this abstract concept shouldn't be too challenging.

Regarding the API, we might simply use a "signature" string like the one above to define a particular workflow. The most simple workflow would have the signature run, which consist of one pass through the DAG executing the run method of each process (topologically sorted). What is more challenging is when the "signature" contains other information than method names (like master_clock above).

The right place to define such signature is probably either as an argument of xs.Model or when setting a simulation driver.

[benbovy]

A intermediate option between the current, fixed workflow and the flexible workflow defined above would be to keep fixed initialize and finalize stages and, instead of two run_step and finalize_step stages, allow user-defined run step sub-stages (given by ordered levels). We could extend the API proposed in #40 to something like:

@xs.process
class Diffusion(object):

    @xs.run_step(level=0, args='clock,clock_diff')
    def run_one_step_first(self, t, dt):
        # use t or dt

    @xs.run_step(level=1, args=None)
    def then_run_another_step(self):
        # ...

[ben1post]

For my usage of xarray-simlab in the current version of the Phydra package, with GEKKO as a backend solver, this issue is quite relevant as referenced in the issue on the Phydra repo automatically linked above.

Specifically the model processes would be much easier to implement if there were different levels of the initialize simulation stage that can be defined. This would be slightly different from what you proposed in the previous comment @benbovy, but perhaps this would be simpler to implement. Keeping the default level of initialize at 0 should not cause any problems with compatibility I think. If this seems reasonable, I could try working on a pull request. Would be happy to hear your thoughts, thanks!

[benbovy]

Note that the suggestions in the comments above are still very elusive. I'm not sure at all if it is a good idea. At least it deserves carefull thinking to make the right design decisions. I'd like to avoid adding too much complexity and confusion.

I haven't looked in detail at your use case, but I wonder if you couldn't solve it by re-arranging your process classes, maybe by adding some aggregation classes (i.e., using xs.group). If that's not possible, I wonder if #62 (enforce user-defined process ordering) wouldn't better help than the suggestions made here.

[ben1post]

Thank you for the feedback! I am not sure if #62 could solve my issues and commented over there for clarification.

To extend on your previous ideas and hopefully specify my use case: It would be useful if there was an interface to xarray-simlab that by default provides the standard runtime stages (i.e. initialize+master_clock*(run_step+finalize_step)+finalize), but when specified by the user would allow for custom runtime stages e.g. when no master clock is supplied initialize+run+finalize or even more complicated but also applicable to my own use case initialize1+initialize2+run+finalize and multiple initialize stages with the step-wise solve initialize1+initialize2+master_clock*(run_step+finalize_step)+finalize. This would make xarray-simlab an incredibly powerful for developing all kinds of models.

I describe my current (incomplete) solution in more detail over on the phydra repo issue. I am using the construct of clocks = [0,1] at model setup precisely to use run_step() as a second initialize simulation stage, and finalize_step() as a run-step, where the model is solved in one step. This was inspired from your npzd xsimlab model code. This means that I have no run_step() stage available to solve the model in the explicit xsimlab clock time step (which would be possible with the GEKKO backend solver). If you say this is a more complex issue, I will write up the first version of Phydra with current xarray-simlab functionality. I'd be happy to help with development, if you have ideas on how to tackle this.