Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EnbPI Prediction Intervals not centered correctly #495

Open
jawharmohammed opened this issue Jul 25, 2024 · 1 comment · May be fixed by #524
Open

EnbPI Prediction Intervals not centered correctly #495

jawharmohammed opened this issue Jul 25, 2024 · 1 comment · May be fixed by #524
Labels
Bug Type: bug Discussion in progress Discussion ongoing between the Mapie team and the author.

Comments

@jawharmohammed
Copy link

Hello MAPIE team,

Congratulations on your excellent work! I have identified an error in the computation of the prediction intervals generated by EnbPI in MAPIE.

Describe the bug

The prediction intervals generated by the EnbPI method in MAPIE do not align with the expected behavior described in the pseudo code provided in the original paper.

Expected behavior

The prediction intervals should be centered at:
[
f^{\Phi}_t(x_t) = \Phi \left(f^{\Phi}_i(x_t), i = 1, \ldots, T \right)
]
Expected Centering

However, in the predict function of the EnsembleRegressor class, where the interval bounds are initialized, the bounds are set with the out-of-fold (oof) prediction of a single estimator rather than the aggregation of y_pred_multi as indicated in the pseudocode. This issue is highlighted in the following screenshot:

Incorrect Initialization

Suggested Solution

A simple solution would be to replace this part with something like the following:

Suggested Solution

Thank you for your attention to this matter!

@jawharmohammed jawharmohammed added the Bug Type: bug label Jul 25, 2024
@thibaultcordier
Copy link
Collaborator

Hello @jawharmohammed

Thank you for your feedback and sorry for the late reply.
The bug you identified appears to be a bug (need more time to look into this). We've started discussing this with @vincentblot28 and will look into it further.

We'll get back to you shortly.

Best,

@jawharmohammed jawharmohammed linked a pull request Oct 14, 2024 that will close this issue
9 tasks
@Valentin-Laurent Valentin-Laurent added the Discussion in progress Discussion ongoing between the Mapie team and the author. label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Type: bug Discussion in progress Discussion ongoing between the Mapie team and the author.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants