Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDR screenshots are unimplemented (gpu/gpu-next) #15107

Open
6 tasks done
low-batt opened this issue Oct 16, 2024 · 7 comments
Open
6 tasks done

HDR screenshots are unimplemented (gpu/gpu-next) #15107

low-batt opened this issue Oct 16, 2024 · 7 comments

Comments

@low-batt
Copy link
Contributor

mpv Information

mpv v0.39.0-129-g4c24eae220 Copyright © 2000-2024 mpv/MPlayer/mplayer2 projects
 built on Oct 15 2024 20:49:12
libplacebo version: v7.349.0 (v7.349.0-18-g9e16c86f)
FFmpeg version: n7.1
FFmpeg library versions:
   libavcodec      61.19.100
   libavdevice     61.3.100
   libavfilter     10.4.100
   libavformat     61.7.100
   libavutil       59.39.100
   libswresample   5.3.100
   libswscale      8.3.100

Other Information

- macOS version: macOS 13.6.9 (22G830)
- Source of mpv: My own build of master using mpv-build
- Introduced in version: Unknown

Reproduction Steps

The version of IINA under development used to generate JPEG XL screenshots that used HDR color spaces. I recently tested it and found the screenshots looked muddy.

Running mpv on a Mac that supports EDR with --vo=libmpv and setting other options similar to what IINA does causes the video to be displayed with EDR enabled:

mpv/build/mpv --no-config --gpu-debug --log-file=output.txt --autofit=100% --cocoa-cb-output-csp=bt.2100-pq  --pause --screenshot-format=jxl --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Take a screenshot.
Use jxlinfo to inspect the resulting JPEG XL screenshot.
Notice the screenshot uses sRGB.

Expected Behavior

A HDR screenshot would be generated.

Actual Behavior

The resulting JPEG XL screenshot:
mpv-shot0001

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0001.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 
From log file:
[   6.457][v][cplayer] Starting screenshot: 'mpv-shot0001.jxl'
[   6.457][v][screenshot] input: 3840x2160 [0:0] rgba64 rgb/bt.709/srgb/full/display CL=mpeg1/jpeg
[   6.457][d][cplayer] Run command: script-binding, flags=73, args=[name="osc/__keybinding4"]
[   6.458][v][screenshot] will convert image to rgba64
[   6.458][v][screenshot] using 8 threads for scaling
[   6.461][v][screenshot] Using zimg.
[   6.475][d][screenshot] mapped color params:
[   6.475][d][screenshot]   trc = iec61966-2-1
[   6.475][d][screenshot]   primaries = bt709
[   6.475][d][screenshot]   range = pc
[   6.475][d][screenshot]   colorspace = gbr
[   6.475][d][screenshot]   chroma_location = center
[   7.137][i][cplayer] Screenshot: 'mpv-shot0001.jxl'

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):
mpv-shot0002

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0002.jxl 
JPEG XL image, 3840x2160, lossy, 16-bit RGB
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 

For completeness, with gpu-next:

mpv/build/mpv --no-config --autofit=100% --pause --screenshot-format=jxl --start=6 --target-colorspace-hint --vo=gpu-next ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

EDR is activated and the resulting screenshot is similar to libmpv with screenshot-sw, but slightly different:

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0003.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative

Log File

output.txt
Screenshots.zip

Sample Files

I used a downloaded copy of this YouTube video: The World in HDR in 4K (ULTRA HD)

I carefully read all instruction and confirm that I did the following:

  • I tested with the latest mpv version to validate that the issue is not already fixed.
  • I provided all required information including system and mpv version.
  • I produced the log file with the exact same set of files, parameters, and conditions used in "Reproduction Steps", with the addition of --log-file=output.txt.
  • I produced the log file while the behaviors described in "Actual Behavior" were actively observed.
  • I attached the full, untruncated log file.
  • I attached the backtrace in the case of a crash.
@Akemi Akemi added core:color-management core:encoding core:option-param-conf config, parameters, properties, options, shortcuts and removed os:mac labels Oct 16, 2024
@sfan5
Copy link
Member

sfan5 commented Oct 16, 2024

Is your intent to take a screenshot that is itself HDR or just a (tonemapped) screenshot of HDR video? I'm assuming the former.

I might be wrong but storing an HDR image as RGB sounds fundamentally wrong to me.
Since VO screenshots always return RGB that means this could possibly only work with --screenshot-sw --screenshot-tag-colorspace --screenshot-high-bit-depth and a format that takes YUV >=10-bits.
Note that ffmpeg's JXL support doesn't take YUV so is unsuited for this.
AVIF might work.


If you want to the latter:
In case of --screenshot-sw=no the VO probably returns either mistagged RGB or RGB but with a HDR csp. seems weird and broken??
You could argue that it's the VO's responsibility to produce a tonemapped screenshot even if the native display is HDR (how will this be configured?).
In case of --screenshot-sw=yes we'd need to tonemap in software with sws or zimg. Problem is neither can.
zimg produces the least bad but still incorrect result, probably what you were seeing with "Adding --screenshot-sw=yes and repeating the test".

@low-batt
Copy link
Contributor Author

The intent is to obtain a HDR screenshot that matches what is seen on the screen, not a tone mapped screenshot. A JPEG XL screenshot that would activate EDR (with a viewer and a Mac that supports it). That is what is generated by --vo=gpu-next and what is generated with --vo=libmpv if I add --screenshot-sw=yes.

I believe at some point that is what mpv was generating without --screenshot-sw=yes as I don't remember needing to use that option when I was originally testing JPEG XL screenshot generation.

@haasn
Copy link
Member

haasn commented Oct 18, 2024

This is a vo_gpu limitation, no? It doesn't implement HDR screenshots. Shouldn't be too hard to support.

As a possible short term work around, we could fallback to sw screenshots if the VO does not support HDR screenshots.

@kasper93
Copy link
Contributor

I have branch with refactored screenshot code in mpv to resolve some common issues. Though was too lazy to finish it. Maybe I will get back to it. In the meantime if someone wants to fix vo_gpu and others, don't hesitate.

@sfan5
Copy link
Member

sfan5 commented Oct 18, 2024

so to be clear the screenshot under

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):

is perfectly fine? and RGB with a HDR colorspace is no problem?

@haasn
Copy link
Member

haasn commented Oct 18, 2024

is perfectly fine? and RGB with a HDR colorspace is no problem?

JXL converts everything to XYZ anyway, doesn't it? So that's just one less conversion step.

@low-batt
Copy link
Contributor Author

I played the movie with --vo=libmpv and --screenshot-sw=yes:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --script-opts-append=osc-visibility=always --pause --screenshot-format=jxl --screenshot-sw=yes --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Took a screenshot and then displayed the screenshot:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --pause --script-opts-append=osc-visibility=always --target-prim=bt.2020 --target-trc=pq --vo=libmpv mpv-shot0001.jxl

I then took a picture of my MacBook Pro (XDR screen). The screenshot is on top. To my eyes the screenshot looks like the video:
IMG_1070.HEIC.zip

@sfan5 sfan5 removed the core:option-param-conf config, parameters, properties, options, shortcuts label Oct 18, 2024
@sfan5 sfan5 changed the title JPEG XL screenshot of HDR video uses sRGB color space HDR screenshots are unimplemented Oct 18, 2024
@sfan5 sfan5 changed the title HDR screenshots are unimplemented HDR screenshots are unimplemented (gpu/gpu-next) Oct 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants