To install the package, clone the repostiory and cd
into the project folder:
Install the package in a new Julia environment:
julia> ]
(@v1.9) pkg> activate .
pkg> instantiate
To start a single training, execute the main.jl
file, where arguments can be passed via command line. For example, to train a mcPLRNN with 50 latent dimensions for 1000 epochs using 4 threads, while keeping all other training parameters at their default setting, call
$ julia -t4 --project main.jl --model PLRNN --latent_dim 50 --epochs 1000
in your terminal of choice (bash/cmd). The default settings can also be adjusted directly; one can then omit passing any arguments at the call site. The arguments are also listed in in the argtable()
function.
To run multiple trainings in parallel e.g. when grid searching hyperparameters, the ubermain.jl
file is used. Currently, one has to adjust arguments which are supposed to differ from the default settings, and arguments that are supposed to be grid searched, in the ubermain
function itself. This is as simple as adding an Argument
to the ArgVec
vector, which is passed the hyperparameter name (e.g. latent_dim
), the desired value, and and identifier for discernibility and documentation purposes. If value is a vector of values, grid search for these hyperparameters is triggered.
function ubermain(n_runs::Int)
# load defaults with correct data types
defaults = parse_args([], argtable())
# list arguments here
args = NetworkTopology.ArgVec([
Argument("experiment", "Lorenz63-Pruning"),
Argument("model", "PLRNN"),
Argument("prune_measure", ["magnitude", "geometry", "random"], "prune"),
Argument("prune_steps", 20),
Argument("prune_value", 0.2)
])
[...]
end
This will run a grid search over prune_measure
using the PLRNN
.
The identifier (e.g. "prune"
in the snippet above) is only mandatory for arguments subject to grid search. Once Arguments are specified, call the ubermain file with the desired number of parallel worker proccesses (+ amount of threads per worker) and the number of runs per task/setting, e.g.
$ julia -t2 --project ubermain.jl -p 10 -r 5
will queue 5 runs for each setting and use 10 parallel workers with each 2 threads.
Evaluating trained models is done via evaluate.jl
. Providing all information in the default settings file, including file/model paths, enables evaluation of single models or a whole iterative pruning procedure done on the PLRNN.
For training the mean-centered PLRNN with identity mapping is used:
Data for the algorithm is expected to be a single trajectory in form of a .npy
), where
These graph structres can be introduced in the parameter matrix of the PLRNN by setting small_world
to:
- Watts-Strogatz model (
watts_strogatz
): Hyperparameters$K$ ,$p$ - Barabási-Albert model (
barabasi_albert
): Hyperparameters$K$ - Erdős–Rényi model (
erdos_renyi
): Hyperparameters$K$ - GeoHub model (
GeoHub
): Hyperparameters$K$ - Custom graph structure of a pruned PLRNN (
custom
): specify the PLRNN model containing the mask by settinginitial_mask_path
to the name path of the experiment,initial_mask_run
to the run id andinitial_mask_ind
to the prune interation from which the mask is chosen. To use the models initial params setinitial_mask_params
to the respective model from which the parameters should be chosen.
To apply iterative pruning to the PLRNN one needs to set the model to → PLRNN
with → Identity
mapping. The number of pruning steps is given by prune_steps
and the amount of weights that get pruned every step by prune_value
. The used pruning method applied can be specified by prune_measure
, where one needs to choose between
- Geometry-based pruning (
geometry
) - Magnitude-based pruning (
magnitude
) - Random pruning (
random
)
-
Julia 1.9
If you find the repository and/or paper helpful for your own research, please cite our work.
@InProceedings{pmlr-v235-hemmer24a,
title = {Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction},
author = {Hemmer, Christoph J\"{u}rgen and Brenner, Manuel and Hess, Florian and Durstewitz, Daniel},
booktitle = {Proceedings of the 41st International Conference on Machine Learning},
pages = {18174--18204},
year = {2024},
editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
volume = {235},
series = {Proceedings of Machine Learning Research},
month = {21--27 Jul},
publisher = {PMLR},
pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hemmer24a/hemmer24a.pdf},
url = {https://proceedings.mlr.press/v235/hemmer24a.html},
abstract = {In dynamical systems reconstruction (DSR) we seek to infer from time series measurements a generative model of the underlying dynamical process. This is a prime objective in any scientific discipline, where we are particularly interested in parsimonious models with a low parameter load. A common strategy here is parameter pruning, removing all parameters with small weights. However, here we find this strategy does not work for DSR, where even low magnitude parameters can contribute considerably to the system dynamics. On the other hand, it is well known that many natural systems which generate complex dynamics, like the brain or ecological networks, have a sparse topology with comparatively few links. Inspired by this, we show that <em>geometric pruning</em>, where in contrast to magnitude-based pruning weights with a low contribution to an attractor’s geometrical structure are removed, indeed manages to reduce parameter load substantially without significantly hampering DSR quality. We further find that the networks resulting from geometric pruning have a specific type of topology, and that this topology, and not the magnitude of weights, is what is most crucial to performance. We provide an algorithm that automatically generates such topologies which can be used as priors for generative modeling of dynamical systems by RNNs, and compare it to other well studied topologies like small-world or scale-free networks.}
}
This work was funded by the German Research Foundation (DFG) through individual grant Du 354/15-1 to DD, and within Germany’s Excellence Strategy EXC 2181/1 - 390900948 (STRUCTURES).