-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated first examples to show model-free case. #1520
base: main
Are you sure you want to change the base?
Conversation
@ben18785 @chonlei @martinjrobins do you think this makes sense? It was a little unclear that you could use PINTS for any old function |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1520 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 97 97
Lines 9577 9577
=========================================
Hits 9577 9577 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On a related topic, I think the readme could also be updated to make it clear that PINTS can also be used for simpler problems. For instance, currently it starts by saying
'PINTS (Probabilistic Inference on Noisy Time-Series) is a framework for optimisation and Bayesian inference on ODE models of noisy time-series, such as arise in electrochemistry and cardiac electrophysiology.'
I think it would be good to make it clear that it can also be used with other models that aren't ODEs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this parabola problem is a really good simple first example. (I did actually think you couldn't use PINTS for 1D optimisation though - I'm pretty sure I encountered an error when I tried to use it for a 1D problem a while ago, and I thought someone else had told me that they'd also experienced this.)
My thoughts
Parabola example:
- I'd use 'PINTS' rather than 'Pints' throughout
- I don't think I was really aware of the ErrorMeasure class. For me, maybe some explanation of when you'd use it vs the logPDF might be helpful. I'm guessing you just use it when you don't have a logPDF?
- I think for beginners it might be helpful to have a bit more of an explanation of the output of PINTS and what it is doing - e.g. it's running the Nelder-Mead optimisation algorithm and this algorithm is evaluating the function at different points. I guess briefly explaining what is shown in each of the columns of the output might be helpful (if this isn't done elsewhere).
- I think plotting the parabola and the minimum that has been found might be useful (or at least printing found_parameters, found_value).
- I found the sentence 'Anything that can be made to fit the ErrorMeasure interface can be optimised this way, including error measures that run simulations, make calls to other software packages, or even interface with other languages.' a bit confusing. I'd maybe change 'error measures' to 'classes' or 'functions'.
Fitting time-series models
- I was wondering actually if this notebook should be split in 2 - so at the bottom of parabola example there is just a hyperlink to this example. Perhaps there could be different first examples depending on what someone is looking to do?
- I also found the following sentence confusing: 'As with the error function above, there are only two main methods: One to tell the optimiser how many parameters the are, and one to run the simulations.' I guess we evaluate the function, but for me this is different from running a simulation. Maybe something like '...one to evaluate the model or run a simulation'?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think a simple example where a logPDF is maximised, or just explaining that this can be done in the same way as for the parabola example, might also be helpful. Alternatively, it might be useful to add a separate notebook with this, if there's not already one. For me starting out I think a simple example where the following is done would have been useful:
- How to define a log-likelihood
- How to find the MLE
- How to define a prior and find the posterior
I'd be happy to try and make a notebook doing the above, if that would be useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also really like this new example. I think it would be worth adding at the very start that, in the first simple example, we're just drawing samples from a distribution, and we're not performing inference until we get to the main one.
Other things I spotted
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @k-shep ! I'll deal with the smaller points shortly.
With regards to the main introduction I think we should discuss in the meeting with the other @pints-team/devs ! We'll need to find a balance between the original USP (inference for time series) and wider applications - or decide to change PINTS' focus.
I'm guessing you did what I did initially, and used a scalar parameter instead of a length-1 vector? |
I can't fully remember, but from looking back at my code I think you're probably correct. I do remember seeing an error raised by PINTS which said that it didn't support the 1D case. I think maybe @I-Bouros experienced something similar. |
That's true from my experience PINTS does not work with optimising or sampling single parameters. |
OK Can you dig out a specific example? The notebooks in this PR show both a 1d optimisation and a 1d sampling problem, so it's definitely not impossible in general :D |
Oh, that's interesting! Seems to be coming directly from CMA-ES. We should add a notice to the CMA-ES optimiser to say it doesn't support 1-d optimisations. Alternatively could try and catch this error and replace it with a more informative one |
After discussion today, seems like we should start the first examples by showing you need an error/logpdf, then move on to show how PINTS lets you define one for time series