-
-
Notifications
You must be signed in to change notification settings - Fork 357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exporting detailed user/system times (in the JSON output)? #660
Comments
Yes. This should be better documented. Slightly related: I also wanted to document what exactly user and system time mean, as that can be quite confusing (#597).
Absolutely.
Yes, let's do it.
Sounds good. It would be great if we could group those somehow. Something like:
Unfortunately, that would be a breaking change to the existing JSON format. But maybe something we could address in a 2.0 release?
That would be an option, but I'd like to carefully plan this feature (and the corresponding CLI). In the past, we also wanted to add the median run time to the output (#344), so the biggest question is: how should the new (default and possibly extended) layout look like?
Contributions are always very welcome! Thank you. |
Currently, the wall time is reported with detailed statistics both in the TUI (mean + stddev + median + min + max), and in the JSON output (list of all the wall times via the
times
field).However, the
user
andsystem
times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.
As such, would you be open to export more details of the user/system times?
user_times
andsystem_times
arrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic).user_stddev
,user_median
,user_min
,user_max
fields (and likewise for system).What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.
The text was updated successfully, but these errors were encountered: