Alternative blending for Stride ImGui region

[gregsn]

Is your feature request related to a problem? Please describe.

Colors typically are too bright.

Describe the solution you'd like

My understanding is that this can't be fixed for the linear color space. Blending just works differently here.
That's why I would propose the user be able to choose between different techniques.
Sometimes one solution looks better than the other.

Describe alternatives you've considered

An additional in-between render target. But I think the alpha of that render target again will make a problem when applying it to the final output.
Also, I tried reasoning with some math, but to my understanding, it's not solvable.

Additional context
current:

new technique:

While the above example makes the new technique look good, there are counterexamples.

current:

new technique:

The new technique in the shader

if (TSRgb)
{
    streams.Color = float4(ColorUtility.ToLinear(streams.Color.rgb), ColorUtility.ToLinear(streams.Color.a));
}

My math and my reasoning as to why this is not solvable. I might be wrong

This is how I understand the graphics pipeline. Again, it's so *##$&ing complicated, I might be wrong. So that's just my opinion how it works ;)

drawing two triangles on top in linear color space:

destc1 = srcA.c * srcA.a + destc0.c * (1 - srcA.a)
destc2 = srcB.c * srcB.a + destc1.c * (1 - srcB.a)

finalcL = togamma(destc2)

destc0, destc1 & destc2 are different states of a pixel in the target surface. srcA and srcB is what the pixel shader outputs for those two calls.
finalcL is what I get to see. My understanding is that displaying a linear target surface to the gamma-based screen involves some final togamma() call somewhere at the very end of the pipeline.

this is how things have been with gamma colors all over the place:

drawing two triangles on top in gamma color space:

destGammac1 = srcGammaA.c * srcGammaA.a + destGammac0.c * (1 - srcGammaA.a)
destGammac2 = srcGammaB.c * srcGammaB.a + destGammac1.c * (1 - srcGammaB.a)

finalcG = destGammac2

it's a bit unfair. The names are longer. But it's simpler as the final togamma() call isn't performed by the pipeline.

task: make the final colors that hit your eye be equal
finalcL =? finalcG
ok. let's only focus on the second triangle been drawn.

We need to somehow make them equal.
togamma(srcB.c * srcB.a + destc1.c * (1 - srcB.a)) =? srcGammaB.c * srcGammaB.a + destGammac1.c * (1 - srcGammaB.a)

It already seems unlikely that this will work out.
Reasoning:
(a*b)^c = a^c*b^c
but surely
(a+b)^c != a^c+b^c

togamma(.. + ..) already gives me no hope to break this up in a way that I'll end up with something that looks exactly like on the right side of the equation.
Well, to be honest, I tried for fun. This is what I ended up with when trying to solve for srcB.a
srcB.a = [toLin(togamma(srcB.c) * srcGammaB.a + destGammac1.c * (1 - srcGammaB.a)) - destc1.c] / (srcB.c - destc1.c)

let's try again.
let's try to make them match somehow while keeping an eye on both sides without solving for one variable:

togamma(destc1.c + srcB.a * (srcB.c - destc1.c)) = destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c)
not a valid transformation:
destGammac1.c + togamma(srcB.a * (srcB.c - destc1.c)) = destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c)
valid:
destGammac1.c + togamma(srcB.a) * togamma(srcB.c - destc1.c) = destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c)
not a valid transformation:
destGammac1.c + togamma(srcB.a) * (togamma(srcB.c) - destGammac1.c) = destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c)
Ok, I have two invalid transformations in here, but funnily I see this guy here: togamma(srcB.a), which is basically what I propose to do: to tweak the alpha... At least in some cases, it looks better.

Design Options
So what do you think?
Can we have a boolean input pin or enum pin on the region where we choose between techniques?

Or should we playful and offer a float where you can lerp between the two solutions?

Thanks for your great work!
@kopffarben

[gregsn]

That's my test patch HowTo ImGui in Stride.txt

[gregsn]

For my own sanity, let's write it the other way around with the hope that this will lead to more insights on how this affects the shader code that is executed in the linear pipeline...

togamma(destc1.c + srcB.a * (srcB.c - destc1.c)) = destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c)
let's focus on ToLinear()
destc1.c + srcB.a * (srcB.c - destc1.c) = ToLinear(destGammac1.c + srcGammaB.a * (srcGammaB.c - destGammac1.c))
not a valid transformation:
destc1.c + srcB.a * (srcB.c - destc1.c) = destc1.c + ToLinear(srcGammaB.a * (srcGammaB.c - destGammac1.c))
valid if we assume ToLinear(srcGammaB.a) = srcB.a, which I am proposing I guess:
destc1.c + srcB.a * (srcB.c - destc1.c) = destc1.c + srcB.a * ToLinear((srcGammaB.c - destGammac1.c))
not a valid transformation:
destc1.c + srcB.a * (srcB.c - destc1.c) = destc1.c + srcB.a * (srcB.c - destc1.c)

two invalid transformations, but at least the left and right side look the same ;)

the one valid part was ToLinear(srcGammaB.a * ...) into srcB.a * ToLinear(...) at least when suggesting that ToLinear() is comparable to x^y.

[gregsn]

Another way to put this is:

If some of the transformations are invalid, and if the problem indeed is unsolvable, why is stride implementing the function ToLinear(float4) in this way:

    // Converts an srgb color to linear space
    float4 ToLinear(float4 sRGBa)
    {
        float3 sRGB = sRGBa.rgb;
        return float4(sRGB * (sRGB * (sRGB * 0.305306011 + 0.682171111) + 0.012522878), sRGBa.a);
    }

and not this way:

    // Converts an srgb color to linear space
    float4 ToLinear(float4 sRGBa)
    {
        return sRGB * (sRGB * (sRGB * 0.305306011 + 0.682171111) + 0.012522878);
    }

To my understanding, there is no right way to treat the alpha, but in our example, the original implementation leads to three invalid transformations, while the new technique only comes with two invalid transformations.
It's a weird way to do math. I'll give you that.

[azeno]

To your last question: I think it's the way how the hardware also does it. The alpha channel is not touched during the conversions. I couldn't find anything about it in the DirectX documentation, but in OpenGL they do answer this question both when reading textures and writing to framebuffers (search for "alpha" in both texts, it will lead to the corresponding question).

Update: The link they refer to in the second document is dead, I think this is a working one: http://alvyray.com/Memos/CG/Microsoft/17_nonln.pdf

[gregsn]

No. Alpha is correctly understood to be a weighting
factor that is best stored in a linear representation. The alpha
component should always be stored as a linear value.

I would subscribe to this way of looking at it.
However when looking closely at the right side of the equation (the gamma pipeline) and compare it to the linear pipeline, we get aware that back then in gamma space the alpha was basically treated in a non-linear way as every calculation is already happening in the gamma space.
When blending in gamma space the alpha therefore can be seen as non-linear.
To emulate the way it was - which is not always wanted, but in our case - we need to get the gamma alpha into linear form.

But ok. Probably there is no way of offering a function that always does the right thing. Depends on how you look at the problem at hand. If you want to emulate the wrong behavior of the gamma pipeline ToLinear(c.a) is your friend.

[gregsn]

Just for the record:
this is the result with SRgbToLinearPreciseA:

this is the result with GammaToLinearA:

namespace VL.ImGui.Stride.Effects
{
    internal shader ImGuiEffectShader<bool TSRgb> : ShaderBase, PositionStream2, ColorBase, Texturing
    {
        matrix proj;

        override stage void VSMain() 
        {
            streams.ShadingPosition = mul(proj, float4(streams.Position2, 0.0, 1.0f)) + float4(-1.0f, 1.0f, 0.0f, 0.0f);

            if (TSRgb)
            {
                streams.Color = GammaToLinearA(streams.Color);
            }
        }

        override stage void PSMain() 
        {
            streams.ColorTarget = streams.Color * Texture0.Sample(LinearSampler, streams.TexCoord);
        }

        // Converts an srgb color to linear space. Alpha is treated the same. 
        // https://github.com/vvvv/VL.StandardLibs/issues/669#issuecomment-1984881266
        // orginal function from Stride ColorUtility shader (ToLinear & SRgbToLinear)
        // SRgbToLinear refers to https://chilliant.blogspot.com/2012/08/srgb-approximations-for-hlsl.html
        float4 ToLinearA(float4 sRGB)
        {
            return sRGB * (sRGB * (sRGB * 0.305306011 + 0.682171111) + 0.012522878);
        }

        // Converts a color from sRGB to linear. Alpha is treated the same
        // https://github.com/vvvv/VL.StandardLibs/issues/669#issuecomment-1984881266
        // orginal function from Stride ColorUtility shader, which refers to this: 
        // https://github.com/vvvv/VL.Stride/pull/395#issuecomment-760253956
        float4 SRgbToLinearPreciseA(float4 srgb)
        {
            float4 higher = pow((srgb + 0.055) / 1.055, 2.4);
            float4 lower = srgb / 12.92;
            float4 cutoff = step(srgb, 0.04045);
            return lerp(higher, lower, cutoff);
        }

        // simple screen gamma conversion. Alpha is treated the same
        float4 GammaToLinearA(float4 RGBa, float Gamma = 2.2)
        {
            return pow(RGBa, Gamma);
        }
    };
}

[gregsn]

So since there seems to be no correct way I would propose the user to be able to select between 6 techniques.

• SrgbToLinearA
• SRgbToLinearPreciseA
• GammaToLinearA

Plus the three counterparts that leave alpha untouched

[gregsn]

yet another idea for a technique:
Do some tweaking on the colors and leave the alpha untouched with the hope that for other backgrounds than black we also get a better result.

    streams.Color = ColorUtility.SRgbToLinear(streams.Color);                
    float additionalAdjust = pow(streams.Color.a, 1.2); // = pow(streams.Color.a, 2.2) / streams.Color.a;
    streams.Color.rgb *= additionalAdjust;

---

All in all, all of these techniques only work well for dark backgrounds.
If there is a real need for a completely correct image, the only option I guess is to render the complete scenery into a texture, which then would be fed into the ImGui (Precomposed) [Stride] region.

The region would

• draw this texture fullscreen opaque as a backdrop for those areas where no ImGui elements are shown.
• additionally it would draw the ImGui mesh completely opaque on top. The effect would sample the precomposed image and do the blending manually. By that, it should be possible to do the blending in gamma space.

[gregsn]

had a look at the precompose idea.

I didn't think about the best way how the 2 regions could share code before knowing if this experiment would work out. So there is a lot of duplicated code here:

https://github.com/vvvv/VL.StandardLibs/tree/feature/VL.ImGui.Stride_PreComposed

There is a problem with the letters, which sample the original precomposed "scene".
So we'd need to render back into that texture...

So for now this is just an experiment that shows that the colors can be blended in a way like in gamma space if you have access to the destc.

[gregsn]

Studied this further:
https://github.com/vvvv/VL.StandardLibs/tree/feature/VL.ImGui.Stride_BlendingTests

I'd argue the best result is the one on the left.
Which is treating the ImGui elements as elements in the scene by making them blend on top of other elements like any transparent quad would do.

My intuition now is that the goal of having them behave as in gamma space was wrong.
I'd still recommend using this in the shader code.

            if (TSRgb)
            {
                streams.Color = ColorUtility.SRgbToLinear(streams.Color);                
                float additionalAdjust = pow(streams.Color.a, 1.2);
                streams.Color.rgb *= additionalAdjust;
            }

This makes sure that when rendered above a black background they look like in gamma space.

If there is no objection I'll do it.
@azeno @kopffarben

[kopffarben]

@gregsn
I've just had a look at your test patch. I agree with you. The left renderer delivers the best results.

do it