-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Breakdown of which benchmarks were solved in paper #9
Comments
Hi Team, if along with that analysis, we get a clear classification of the competitions based on the types mentioned. And the medals achievement against the various types. I think there were multiple runs and seeds, but just for the best runs would do. Regards, |
Hi, what do you mean exactly by this? Are you looking for the raw data that went into making Figure 6 in the report? If you're simply looking for which comps are low/medium/high, you can check the splits in experiments/splits/ |
Hi, not sure what the other user meant, I was hoping to get a medal breakdown based on the difficulty tiers of the challenges. In table 2 you report the following measures: Made Submission (%), Valid Submission (%), Above Median (%), Bronze (%), Silver (%), Gold (%), Any Medal (%). However, since you do not report which exact benchmarks the above metrics were earned for, there is no way to know the e.g. Above Median (%) for low complexity problems. I was hoping to be able to create a table with the following columns Made Submission (%), Valid Submission (%), Above Median (%), Bronze (%), Silver (%), Gold (%), Any Medal (%) However, reporting the breakdown based on complexity (e.g. low, medium, high). |
Yes, I am looking to find the raw data for Figure 6. And some analysis of the medals earned by the runs based on the various categories mentioned in Figure 6. ( Like the % medal for Tabular, Tex Classification...) |
Is there no results.txt file anywhere, where the benchmarks that were solved are available? Or logs from the experiments? I was just hoping for an accuracy/medal breakdown on MLE-Bench based on complexity, would be happy to calculate myself if the logs are available. |
We'll likely share the grading reports from our runs later this week or next :) |
Thank you very much! |
Hello,
I noticed that in the paper there is no discussion of exactly which of the benchmarks your solutions were able to solve. I am also curious of the percent breakdown for Low, Medium, and High complexity (e.g. above medium / earning Bronze, Silver, Gold). I would greatly appreciate if this data could be provided.
Thank you,
Sam
The text was updated successfully, but these errors were encountered: