Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction [ICML 2024 Poster]

1. Getting started

To install the package, clone the repostiory and cd into the project folder: Install the package in a new Julia environment:

julia> ]
(@v1.9) pkg> activate .
pkg> instantiate

2. Running the code

2.1 Single runs

To start a single training, execute the main.jl file, where arguments can be passed via command line. For example, to train a mcPLRNN with 50 latent dimensions for 1000 epochs using 4 threads, while keeping all other training parameters at their default setting, call

$ julia -t4 --project main.jl --model PLRNN --latent_dim 50 --epochs 1000

in your terminal of choice (bash/cmd). The default settings can also be adjusted directly; one can then omit passing any arguments at the call site. The arguments are also listed in in the argtable() function.

2.2 Multiple runs + grid search

To run multiple trainings in parallel e.g. when grid searching hyperparameters, the ubermain.jl file is used. Currently, one has to adjust arguments which are supposed to differ from the default settings, and arguments that are supposed to be grid searched, in the ubermain function itself. This is as simple as adding an Argument to the ArgVec vector, which is passed the hyperparameter name (e.g. latent_dim), the desired value, and and identifier for discernibility and documentation purposes. If value is a vector of values, grid search for these hyperparameters is triggered.

function ubermain(n_runs::Int)
    # load defaults with correct data types
    defaults = parse_args([], argtable())

    # list arguments here
    args = NetworkTopology.ArgVec([
        Argument("experiment", "Lorenz63-Pruning"),
        Argument("model", "PLRNN"),
        Argument("prune_measure", ["magnitude", "geometry", "random"], "prune"),
        Argument("prune_steps", 20),
        Argument("prune_value", 0.2)
    ])

    [...]
end

This will run a grid search over prune_measure using the PLRNN.

The identifier (e.g. "prune" in the snippet above) is only mandatory for arguments subject to grid search. Once Arguments are specified, call the ubermain file with the desired number of parallel worker proccesses (+ amount of threads per worker) and the number of runs per task/setting, e.g.

$ julia -t2 --project ubermain.jl -p 10 -r 5

will queue 5 runs for each setting and use 10 parallel workers with each 2 threads.

2.3 Evaluating models

Evaluating trained models is done via evaluate.jl. Providing all information in the default settings file, including file/model paths, enables evaluation of single models or a whole iterative pruning procedure done on the PLRNN.

Specifics

1. Model architecture

For training the mean-centered PLRNN with identity mapping is used:

• mean-centred PLRNN → PLRNN
• identity mapping → Identity

2. Data Format

Data for the algorithm is expected to be a single trajectory in form of a $T \times N$ matrix (file format: .npy), where $T$ is the total number of time steps and $N$ is the data dimensionality. Examples are provided.

3. Supported graph structures

These graph structres can be introduced in the parameter matrix of the PLRNN by setting small_world to:

• Watts-Strogatz model (watts_strogatz): Hyperparameters $K$, $p$
• Barabási-Albert model (barabasi_albert): Hyperparameters $K$
• Erdős–Rényi model (erdos_renyi): Hyperparameters $K$
• GeoHub model (GeoHub): Hyperparameters $K$
• Custom graph structure of a pruned PLRNN (custom): specify the PLRNN model containing the mask by setting initial_mask_path to the name path of the experiment, initial_mask_run to the run id and initial_mask_ind to the prune interation from which the mask is chosen. To use the models initial params set initial_mask_params to the respective model from which the parameters should be chosen.

4. Pruning

To apply iterative pruning to the PLRNN one needs to set the model to → PLRNN with → Identity mapping. The number of pruning steps is given by prune_steps and the amount of weights that get pruned every step by prune_value. The used pruning method applied can be specified by prune_measure, where one needs to choose between

• Geometry-based pruning (geometry)
• Magnitude-based pruning (magnitude)
• Random pruning (random)

Versions

• Julia 1.9

Citation

If you find the repository and/or paper helpful for your own research, please cite our work.

@InProceedings{pmlr-v235-hemmer24a,
  title = 	 {Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction},
  author =       {Hemmer, Christoph J\"{u}rgen and Brenner, Manuel and Hess, Florian and Durstewitz, Daniel},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  pages = 	 {18174--18204},
  year = 	 {2024},
  editor = 	 {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
  volume = 	 {235},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {21--27 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hemmer24a/hemmer24a.pdf},
  url = 	 {https://proceedings.mlr.press/v235/hemmer24a.html},
  abstract = 	 {In dynamical systems reconstruction (DSR) we seek to infer from time series measurements a generative model of the underlying dynamical process. This is a prime objective in any scientific discipline, where we are particularly interested in parsimonious models with a low parameter load. A common strategy here is parameter pruning, removing all parameters with small weights. However, here we find this strategy does not work for DSR, where even low magnitude parameters can contribute considerably to the system dynamics. On the other hand, it is well known that many natural systems which generate complex dynamics, like the brain or ecological networks, have a sparse topology with comparatively few links. Inspired by this, we show that <em>geometric pruning</em>, where in contrast to magnitude-based pruning weights with a low contribution to an attractor’s geometrical structure are removed, indeed manages to reduce parameter load substantially without significantly hampering DSR quality. We further find that the networks resulting from geometric pruning have a specific type of topology, and that this topology, and not the magnitude of weights, is what is most crucial to performance. We provide an algorithm that automatically generates such topologies which can be used as priors for generative modeling of dynamical systems by RNNs, and compare it to other well studied topologies like small-world or scale-free networks.}
}

Acknowledgements

This work was funded by the German Research Foundation (DFG) through individual grant Du 354/15-1 to DD, and within Germany’s Excellence Strategy EXC 2181/1 - 390900948 (STRUCTURES).