Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exporting detailed user/system times (in the JSON output)? #660

Open
gendx opened this issue Jun 19, 2023 · 1 comment
Open

Exporting detailed user/system times (in the JSON output)? #660

gendx opened this issue Jun 19, 2023 · 1 comment

Comments

@gendx
Copy link

gendx commented Jun 19, 2023

Currently, the wall time is reported with detailed statistics both in the TUI (mean + stddev + median + min + max), and in the JSON output (list of all the wall times via the times field).

However, the user and system times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).

In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.

As such, would you be open to export more details of the user/system times?

  1. At the minimum, I think that adding user_times and system_times arrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic).
  2. Additionally, the JSON output could also be extended with user_stddev, user_median, user_min, user_max fields (and likewise for system).
  3. Lastly (but it's maybe more disruptive), the TUI could also report these details for user/system, when some CLI flag is enabled.

What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.

@sharkdp
Copy link
Owner

sharkdp commented Jun 19, 2023

However, the user and system times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).

Yes. This should be better documented.

Slightly related: I also wanted to document what exactly user and system time mean, as that can be quite confusing (#597).

In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.

Absolutely.

At the minimum, I think that adding user_times and system_times arrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic).

Yes, let's do it.

Additionally, the JSON output could also be extended with user_stddev, user_median, user_min, user_max fields (and likewise for system).

Sounds good. It would be great if we could group those somehow. Something like:

  • runtime
    • mean
    • stddev
    • median
    • list: […]
  • user time
    • mean
    • stddev
    • median
    • list: […]
  • system time
    • mean
    • stddev
    • median
    • list: […]

Unfortunately, that would be a breaking change to the existing JSON format. But maybe something we could address in a 2.0 release?

Lastly (but it's maybe more disruptive), the TUI could also report these details for user/system, when some CLI flag is enabled.

That would be an option, but I'd like to carefully plan this feature (and the corresponding CLI). In the past, we also wanted to add the median run time to the output (#344), so the biggest question is: how should the new (default and possibly extended) layout look like?

What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.

Contributions are always very welcome! Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants