Having Issue with Subclassing in C++

[KrishBaidya]

Describe the bug

I am having issues with subclassing in C++, but it works correctly in C#.

The issue I am Getting is - Exception thrown at 0x00007FF9C7231DD4 (user32.dll) in WinUICPPTest.exe: 0xC0000005: Access violation reading location 0xFFFFFFFF9E76E4C0.

On this line return CallWindowProc(originalWndProc, hWnd, uMsg, wParam, lParam);

Here is the C++ version-

#include "pch.h"
#include "MainWindow.xaml.h"
#if __has_include("MainWindow.g.cpp")
#include "MainWindow.g.cpp"
#endif

#include <microsoft.ui.xaml.window.h>

using namespace winrt;
using namespace Microsoft::UI::Xaml;

namespace winrt::WinUICPPTest::implementation
{
    static WNDPROC originalWndProc;

    static LRESULT CustomWndProc(HWND hWnd, UINT uMsg, INT wParam, INT lParam)
    {
        switch (uMsg)
        {
        case WM_HOTKEY: // WM_HOTKEY
            if (IsWindowVisible(hWnd) == 0)
            {
                ShowWindow(hWnd, 5); // SW_SHOW
                SetForegroundWindow(hWnd);
            }
            else
            {
                ShowWindow(hWnd, 0); // SW_HIDE
            }
            break;
        }

        // Call the original window procedure for any unhandled messages
        return CallWindowProc(originalWndProc, hWnd, uMsg, wParam, lParam);
    }

    MainWindow::MainWindow()
    {
        // Xaml objects should not call InitializeComponent during construction.
        // See https://github.com/microsoft/cppwinrt/tree/master/nuget#initializecomponent

        auto windowNative{ this->m_inner.as<::IWindowNative>() };
        HWND hwnd{ 0 };
        windowNative->get_WindowHandle(&hwnd);

        // Register the hotkey
        RegisterHotKey(hwnd, 1, MOD_ALT, VK_F12);

        // Subclass the window procedure
        SubclassWndProc(hwnd);
    }

    int32_t MainWindow::MyProperty()
    {
        throw hresult_not_implemented();
    }

    void MainWindow::MyProperty(int32_t /* value */)
    {
        throw hresult_not_implemented();
    }

    void MainWindow::SubclassWndProc(HWND hwnd)
    {
        // Store the original window procedure
        originalWndProc = (WNDPROC)GetWindowLongPtr(hwnd, GWLP_WNDPROC);

        // Subclass the window procedure
        SetWindowLongPtr(hwnd, GWLP_WNDPROC, (LONG_PTR)CustomWndProc);
    }

    void MainWindow::RegisterF12HotKey(HWND hWnd)
    {
        // Register Alt + F12 as a hotkey
        RegisterHotKey(hWnd, 1, MOD_ALT, VK_F12);
    }
}

And Here is the C# version

namespace WinUITestApp
{
    public sealed partial class MainWindow : Window
    {
        private delegate IntPtr WndProcDelegate(IntPtr hWnd, uint msg, IntPtr wParam, IntPtr lParam);
        private static WndProcDelegate customWndProcDelegate;
        private static IntPtr originalWndProc = IntPtr.Zero;

        public MainWindow()
        {
            this.InitializeComponent();

            // Subclass the window procedure
            customWndProcDelegate = new WndProcDelegate(CustomWndProc);
            IntPtr hwnd = WinRT.Interop.WindowNative.GetWindowHandle(this);
            SubclassWndProc(hwnd);
            RegisterF12HotKey(hwnd);

            if (this.AppWindow != null)
            {
                // Get the presenter and modify its properties
                var presenter = this.AppWindow.Presenter as OverlappedPresenter;
                if (presenter != null)
                {
                    presenter.IsMaximizable = false;
                    presenter.IsMinimizable = false;
                    presenter.IsResizable = true;
                }
            }

        }

        // Import the necessary user32.dll functions
        [DllImport("user32.dll", SetLastError = true, EntryPoint = "SetWindowLongPtr")]
        private static extern IntPtr SetWindowLongPtr64(IntPtr hWnd, int nIndex, IntPtr dwNewLong);

        [DllImport("user32.dll", SetLastError = true, EntryPoint = "SetWindowLong")]
        private static extern IntPtr SetWindowLong32(IntPtr hWnd, int nIndex, IntPtr dwNewLong);

        [DllImport("user32.dll", SetLastError = true, EntryPoint = "GetWindowLongPtr")]
        private static extern IntPtr GetWindowLongPtr64(IntPtr hWnd, int nIndex);

        [DllImport("user32.dll", SetLastError = true, EntryPoint = "GetWindowLong")]
        private static extern IntPtr GetWindowLong32(IntPtr hWnd, int nIndex);

        private const int GWL_WNDPROC = -4;

        private static void SubclassWndProc(IntPtr hwnd)
        {
            // Store the original window procedure pointer
            originalWndProc = GetWindowLongPtr(hwnd, GWL_WNDPROC);

            // Set the new window procedure (custom one)
            SetWindowLongPtr(hwnd, GWL_WNDPROC, Marshal.GetFunctionPointerForDelegate(customWndProcDelegate));
        }

        private static IntPtr GetWindowLongPtr(IntPtr hWnd, int nIndex)
        {
            if (IntPtr.Size == 8) // 64-bit
            {
                return GetWindowLongPtr64(hWnd, nIndex);
            }
            else // 32-bit
            {
                return GetWindowLong32(hWnd, nIndex);
            }
        }

        private static IntPtr SetWindowLongPtr(IntPtr hWnd, int nIndex, IntPtr dwNewLong)
        {
            if (IntPtr.Size == 8) // 64-bit
            {
                return SetWindowLongPtr64(hWnd, nIndex, dwNewLong);
            }
            else // 32-bit
            {
                return SetWindowLong32(hWnd, nIndex, dwNewLong);
            }
        }

        private static IntPtr CustomWndProc(IntPtr hWnd, uint uMsg, IntPtr wParam, IntPtr lParam)
        {
            switch (uMsg)
            {
                case 0x0312: // WM_HOTKEY
                    if (IsWindowVisible(hWnd) == 0)
                    {
                        ShowWindow(hWnd, 5); // SW_SHOW
                        SetForegroundWindow(hWnd);
                    }
                    else
                    {
                        ShowWindow(hWnd, 0); // SW_HIDE
                    }
                    break;
            }

            // Call the original window procedure for any unhandled messages
            return CallWindowProc(originalWndProc, hWnd, uMsg, wParam, lParam);
        }

        [DllImport("user32.dll")]
        private static extern IntPtr CallWindowProc(IntPtr lpPrevWndFunc, IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

        [DllImport("user32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);

        [DllImport("user32.dll", SetLastError = true)]
        static extern bool SetForegroundWindow(IntPtr hWnd);

        [DllImport("user32.dll")]
        static extern int IsWindowVisible(IntPtr hWnd);

        [DllImport("user32.dll", SetLastError = true)]
        static extern bool RegisterHotKey(IntPtr hWnd, int id, uint fsModifiers, uint vk);

        private const int MOD_ALT = 0x0001;
        private const int VK_F12 = 0x7B;

        private void RegisterF12HotKey(IntPtr hWnd)
        {
            // Register Alt + F12 as a hotkey
            RegisterHotKey(hWnd, 1, MOD_ALT, VK_F12);
        }
    }
}

Steps to reproduce the bug

1. Create a new Win UI 3 Packaged C++ Project.
2. Copy and paste above code in MainWindow.xaml.cpp.
3. Add all Declarations needed in MainWindow.xaml.h
4. Build and Run the Application.

Expected behavior

The Window Subclassing should work fine without throwing the above exception. The Window should have invoked when HotKey is pressed.

Screenshots

No response

NuGet package version

None

Windows version

No response

Additional context

I am working on a WinUI3 app with C++, but this is the issue I am getting. I tried this on a C# WinUI 3 App and that's working correctly.

[DarranRowe]

Well, your subclass window procedure is badly defined. I wouldn't be surprised if this is causing truncation and sign extension resulting in the access violation of a kernel mode pointer.

The actual signature of a window procedure is:

LRESULT CALLBACK WndProc(HWND, UINT, WPARAM, LPARAM);

This is important for two reasons. First, the CALLBACK marks the function as __stdcall. For 32 bit applications, this sets the correct calling convention. It is ignored for 64 bit applications. The third and fourth parameters are typedefs for 64 bit types for 64 bit applications. WPARAM is a typedef for unsinged long long (uint64_t). LPARAM is a typedef for long long (int64_t). The type you use, INT is a typedef for int, a 32 bit unsigned type. This means that any types being passed into the window procedure will be truncated to 32 bits, and then sign extended again.

[KrishBaidya]

Yeah, I just found this. In my main project I was having different issue, that's why I created a sample C# with chatGPT, it was working fine and I copy pasted. Sorry, I have now close this.