Windows hooks

Or how to convince Windows to do things it doesn't want to

Gavin Smyth

This is an article which appeared in EXE magazine in May 2000. Note that the source code has been updated quite a lot since then so the description is somewhat out of date.


I'm sure a lot of you have made use of Nullsoft's Winamp, a very nice audio playback tool for Win32 (see Figure 1). One minor irritation I have with Winamp is that the controls for skipping tracks, etc. are available either on the Winamp window or via a menu from its system tray icon. In the former case, it means I have to waste a bit of screen space to keep that window visible (or have to find it behind some other window), though the window size can be reduced in "windowshade" mode as shown in Figure 2; in the latter, selecting the appropriate menu item is just too many mouse movements away (OK, I'm lazy). There is a plugin which docks the windowshade window in the title bar of the current foreground application, but this is still too visually distracting for me, so I wondered if I could find some other way to use some of that otherwise useless space between the window title and the maximise/minimise buttons on the right on a typical window's title bar: this way, I would not waste desktop space with a Winamp window or time trying to find the window.

Figure 1: Winamp main window
Winamp main window
Figure 2: Winamp in Windowshade mode
Winamp in Windowshade mode

It turns out that this is possible-otherwise this would turn out to be a fairly dull article...

Figure 3 shows my modified title bar. Rather than a complete Winamp interface, I just coded the few controls I tend to use most of the time-play, pause, and skip forwards and backwards a track. Generally, controls in the title bar are frowned upon - see the Interface Hall of Shame, for example. However, I justify it here because this utility is intended for knowledgeable users...

Figure 3: Modified title bar
Modified title bar

Painting windows

Most Windows programmers are aware that the title bar lies outside a window's client area, and therefore drawing it is controlled by the WM_NCPAINT message. This suggests that all I need to do to display some Winamp buttons is intercept this message, and blit in my buttons' bitmap after Windows has done its drawing-after Windows has done its job so that I do not have to draw the whole non-client area. (This could lead to some flicker, particularly when resizing a window in show contents mode, as the portion of the title bar covered by my bitmap ends up being drawn twice - once by Windows and once by my program - but it is considerably less effort than doing the job properly by manipulating Windows update regions: see the documentation for RectInRegion and CombineRegion for more information on that topic.) Figure 4 shows the routines for drawing the title bar.

Different windows can have different sets of features on the righthand side of the title bar: some may have only a close box; some have maximise and minimise as well (although windows styles allow to specify only one of the pair, I do not recall any applications that display one - they show both, but the non-specified one greyed out); and some have a help icon instead of the sizing buttons. WindowsFeaturesWidth is derived from experimentation to get a decent fit for most windows, and is used (via GetButtonPosition) by DrawButtons which works out where the bitmap should be drawn, and then blits it on to the title bar. As I will explain shortly, the window procedure of the foreground window is intercepted by WindowProc, which calls the old window function (stored in global SubclassedWindowProc) first, and then reacts to any messages I need to take note of, WM_NCPAINT being the only one of relevance at this point. You will notice that I blit all the bitmaps as one block - slightly more efficient than dealing with them separately. However, I have not bothered to handle any button animation (i.e., depressing them when the mouse is clicked) - I will leave that, as they say, as an exercise for the reader.

Figure 4: Title bar drawing code

static int WindowsFeaturesWidth( HWND hwnd, DWORD style )
{
int buttonWidth = GetSystemMetrics( SM_CXSIZE );
  int width = GetSystemMetrics( ( style & WS_THICKFRAME ) ? SM_CXSIZEFRAME : SM_CXFIXEDFRAME ) + 2;
  if( style & WS_SYSMENU )
    width += buttonWidth;
  if( style & ( WS_MAXIMIZEBOX | WS_MINIMIZEBOX ) )
    width += buttonWidth * 2 - 2; /* -2 since the buttons touch */
  else if( GetWindowLong( hwnd, GWL_EXSTYLE ) & WS_EX_CONTEXTHELP )
    width += buttonWidth;

  return width;
}

static void GetButtonPosition( HWND hwnd, DWORD style, LONG* buttonLeftOffset, LONG* buttonTopOffset )
{
  RECT r;
  GetWindowRect( hwnd, &r );
  /* r is now the window position in screen coordinates */

  *buttonLeftOffset = r.right - r.left - WindowsFeaturesWidth( hwnd, style ) - BUTTON_WIDTH * NUM_BUTTONS;
  *buttonTopOffset = GetSystemMetrics( ( style & WS_THICKFRAME ) ? SM_CYSIZEFRAME : SM_CYFIXEDFRAME ) + ( GetSystemMetrics( SM_CYSIZE ) - BUTTON_HEIGHT ) / 2;
}

static void DrawButtons( HWND hwnd, DWORD style )
{
  HDC hdc = GetWindowDC( hwnd );
  if( hdc )
  {
    LONG x, y;
    HDC hdcMem;
    HGDIOBJ oldBmp;

    GetButtonPosition( hwnd, style, &x, &y );

    hdcMem = CreateCompatibleDC( hdc );
    oldBmp = SelectObject( hdcMem, Image );
    BitBlt( hdc, x, y, BUTTON_WIDTH * NUM_BUTTONS, BUTTON_HEIGHT, hdcMem, 0, 0, SRCCOPY );
    SelectObject( hdcMem, oldBmp );
    DeleteDC( hdcMem );

    ReleaseDC( hwnd, hdc );
  } /* end if got DC */
}
LRESULT CALLBACK WindowProc( HWND hwnd, UINT msg, WPARAM wp, LPARAM lp )
{
  LRESULT res;
  DWORD style;

  res = CallWindowProc( SubclassedWindowProc, hwnd, msg, wp, lp );

  style = GetWindowLong( hwnd, GWL_STYLE );
  if( ( style & WS_CAPTION ) == WS_CAPTION )
    switch( msg )
    {
      /* Only NC paint shown for the time being: remainder in Figure 9 */
      case WM_NCPAINT:
        DrawButtons( hwnd, style );
        break;
    } /* end switch */

  return res;
}

Subclassing

That's all very well, but how do you intercept a window's message function? This is what window subclassing is all about - using SetWindowLong on a window's GWL_WNDPROC does the job. Figure 5 shows how the new procedure is attached and detached. There are a number of things worth noting about this code. First, the global variable SubclassedWindowProc stores a pointer to the subclassed window's original windows procedure, and I ensure that this is set to null when no window is subclassed. Before subclassing, I use it as an interlock, checking that this is indeed null, and do not subclass otherwise - this prevents multiple window managing threads within one process getting confused. There is still a race condition, such that no window might end up being subclassed, but that is a better state of affairs than two window procedures trying to share the same subclassing data. A more expensive way to avoid the problem would involve Windows semaphores, but as always in hook and subclassing routines, it is important to minimise the extra load injected by the new code.

I then check that a window actually has a caption before I subclass it - in my first version, I made the assumption that this has the side-effect of rendering it unnecessary to check within the drawing routine since I can be sure that at that point in the code, the window must have a caption. Unfortunately, applications such as the Windows clock prove this incorrect since the title bar can be added and removed from this program as it runs, so I needed the check in the windows procedure too. Next, if an Explorer Window crashes, it is a nuisance because the whole of Explorer is restarted, losing all the Explorer windows that might have been open. The check within #ifdef EXCLUDE_EXPLORER ... #endif avoids subclassing Explorer windows, just to make life easier when debugging. (Incidentally, a much nicer way of debugging would be to use Visual Studio's remote debugging facilities, but unfortunately I only have one PC for development.)

Finally, if the subclassing was successful, the bitmap is drawn, using the DrawButtons function described above.

Unsubclassing is much simpler - replace the window procedure, zero the interlock variable, once again asking the target window to redraw the non-client area.

Figure 5: Subclassing and unsubclassing

static void Subclass( HWND hwnd )
{
  DWORD style;

  if( SubclassedWindowProc )
    return;

  /* I can only take over a window with a caption, so if there isn't one... */
  style = GetWindowLong( hwnd, GWL_STYLE );
  if( ( style & WS_CAPTION ) != WS_CAPTION )
    return;

  /* For debugging, don't take over explorer windows since crashing one of those is a real pain! */
# ifdef EXCLUDE_EXPLORER
  {
    TCHAR buff[ 7 ];
    if( GetWindowText( hwnd, buff, sizeof( buff ) / sizeof( TCHAR ) ) &&
        !_tcsncmp( buff, _T("Explor"), 6 ) )
       return;
  }
# endif

  SubclassedWindow = hwnd;
  SubclassedWindowProc = (WNDPROC)SetWindowLong( hwnd, GWL_WNDPROC, (LONG)WindowProc );
  if( SubclassedWindowProc )
    DrawButtons( hwnd, style );
}

static void UnSubclass()
{
  SetWindowLong( hwnd, GWL_WNDPROC, (LONG)SubclassedWindowProc );
  SubclassedWindowProc = NULL;
  SendMessage( hwnd, WM_NCPAINT, 1, 0 );
}

Hooks

I have a subclass mechanism, but how do I trigger it on the foreground window? Windows offers a number of hooking mechanisms, each of which intercepts particular groups messages when processed by all windows. The relevant one here is the rather oddly named CBT hook - the message was originally intended for computer based training programs, but it is just what I need here. I can use it to intercept a window being activated - and this means activated in any way: by a mouse click, or by keyboard (for example, by ALT-TAB), or when an application is started, or when the current foreground one is closed or iconised, bringing another one to the foreground. Figure 6 shows the rather short hook routine. I invoke the previously installed hook (since I do not want to change existing window behaviour) and if the message indicates that a window is being activated, subclass the window. Hooks are expensive-the are invoked on every application's handler for the relevant message types, so they should be kept as small as possible. The CBT hook is installed with SetWindowsHookEx, but this is where it starts to get a bit tricky...

Figure 6: CBT hook procedure

LRESULT CALLBACK CbtProc( int code, WPARAM wp, LPARAM lp )
{
if( code == HCBT_ACTIVATE )
  {
    LPCBTACTIVATESTRUCT cb = (LPCBTACTIVATESTRUCT)lp;
    if( cb->hWndActive )
      SendMessage( cb->hWndActive, UnsubclassMessage, 0, 0 );

    if( wp && (HWND)wp != WinampWindow )
      Subclass( (HWND)wp );
  }

  return CallNextHookEx( PrevHook, code, wp, lp );
}

The CBT hook runs in the context of the application that handles the window activate message - in other words, potentially by any window-managing thread in the system. This immediately suggests that I need some system-wide shared memory since by default one process cannot access memory belonging to any other one. Shared static memory in itself is not very complex - the code fragment below shows how to achieve it:


#pragma data_seg( ".shared" )

static HHOOK  PrevHook = 0;
static UINT   UnsubclassMessage = 0;
static HWND   SubclassedWindow = 0;
static HWND   WinampWindow = 0;

#pragma data_seg()
#pragma comment( linker, "/SECTION:.shared,RWS" )

The data_seg pragma tells the compiler to insert the next variables in the named segment and the comment pragma tells the linker to label the segment as shared - there is no significance attached to the segment name except that it much match in the two pragmas. Two things are worth noting - I spent a few hours scratching my head before I appreciated their significance. First, the variables must be initialised - if I had just, for example, static HHOOK PrevHook it would not have been placed in the new segment although in general, you expect that omitting a static initialiser is equivalent to initialising to zero. Second, there must be no space in the string in the comment pragma - I originally had a space after the comma, but that resulted in the flags being quietly ignored and segment simply not being shared-instead distinct sets of the variables appeared in each application. (Actually, a third thing is worth noting: these pragmas are specific to Microsoft's compiler - other compilers will have facilities to achieve similar results, but I do not know what they are.)

One of the rules about Windows is that a system-wide hook procedure must be present in a DLL rather than an application (so that it can be loaded in any process). This is not much of a problem here since this code forms part of a Winamp plugin DLL. The hook needs to be installed once only and removed once only, using the fairly straightforward code in Figure 7. Also in Figure 7 is a call to register the unsubclass message: if a new message is being sent globally, its number should be determined via RegisterWindowMessage as a statically allocated value could collide with some other application. Although the unsubclass message is used only by this utility, remember that code to handle it is inserted into the windows procedure of every appropriate window.

Figure 7: Hook installation and uninstallation functions

int InitialiseGlobal( HINSTANCE hinst )
{
  UnsubclassMessage = RegisterWindowMessage( _T("BKWinampBarUnSubclassWindowMessage") );

  if( PrevHook )   /* If already hooked, something went wrong! */
    return -1;
  PrevHook = SetWindowsHookEx( WH_CBT, CbtProc, hinst, 0 );
  if( !PrevHook )
    return -1;

  return 0;
}

void ShutdownGlobal()
{
  if( SubclassedWindow )
    SendMessage( SubclassedWindow, UnsubclassMessage, 0, 0 );

  UnhookWindowsHookEx( PrevHook );
}

It is relatively easy to ensure that the installation and uninstallation code is indeed invoked only once in most cases but it might be worth protecting against multiple simultaneous instantiations of the loading program. The DllMain procedure, shown in Figure 8 demonstrates a cunning trick to achieve this and, as a pleasant side-effect, removes the need for an explicit unhook call since that happens automatically as the DLL is unloaded. There are a few bits of Windows hackery in this short routine:

For a DLL, the actual entry point or any DLL is _DllMainCRTStartup - this initialises the standard C library and C++ runtime, and then calls DllMain. However, if you do not use code from any libraries other than Windows standard ones (such as user32 or kernel32), there is no need to initialise the standard C library. In this application, as long as I am not in debug mode or using the string comparison in the Explorer check, I use only Windows libraries, so I can make my entry point _DllMainCRTStartup instead of DllMain. This shaves about 6KB off the size of the release DLL and speeds up load time - the latter is more important than the memory saving since the DLL may be loaded by a lot of applications simultaneously. Another speed up possibility occurs at link time: all DLLs have a base address, defaulting to 0x1000000 if none is explicitly given. If a DLL already occupies this address when one is being loaded, the new one is automatically relocated, but this takes a little time. You can speed up loading by choosing a different address: unfortunately, there is no easy way to pick a good address, but anything is better than the default since that is where most DLLs go!

The rest of the function splits into two sections: handling loading of the DLL and subsequent unloading. There is another load time saving in the load routine: DLLs are normally notified about thread attach and detach as well as process ones, but thread notifications are not required here so they are disabled via DisableThreadLibraryCalls.

After loading the button bitmap for the DLL's resources, I start the test to see if this is the first module to load the DLL. I do this by trying to create a mutex: only one process can actually create it, and subsequent calls will result in status code ERROR_ALREADY_EXISTS. In general, successful Win32 function returns do not affect the internal error code at all, so I explicitly initialise this to zero before trying to create the mutex. If the return code from CreateMutex indicates that this was the call that create the mutex, this must be the first process to load the DLL, so the rest of the initialisation can occur here, in addition starting with setting a flag to indicate that this was the hooking process. This flag is used in the corresponding DLL detach to determine if the hook should be uninstalled. (Similar manipulation can be used to ensure that an application runs as a singleton: a mutex is created at application startup and if the result indicates that it already existed, the application exits as there must already be a copy running.)

Figure 8: DllMain

#if defined( _DEBUG ) || defined( EXCLUDE_EXPLORER )
  BOOL WINAPI DllMain( HINSTANCE hinst, DWORD reason, LPVOID reserved )
#else
  BOOL WINAPI _DllMainCRTStartup( HINSTANCE hinst, DWORD reason, LPVOID reserved )
#endif
{
  static HANDLE Mutex;     /* Used to determine if first loading process */
  static BOOL   Hooker;

  if( reason == DLL_PROCESS_ATTACH )
  {
    DisableThreadLibraryCalls( hinst );

    Image = LoadImage( hinst, MAKEINTRESOURCE( IDB_BUTTONS ), IMAGE_BITMAP, 0, 0, LR_SHARED );
    if( !Image )
      return FALSE;

    SetLastError( 0 );
    Mutex = CreateMutex( NULL, FALSE, _T("BKWinampBarMutex") );
    if( Mutex )
    {
      if( GetLastError() != ERROR_ALREADY_EXISTS )
      {
        Hooker = TRUE;
        if( !InitialiseGlobal( hinst ) )
          return FALSE;
      }
    }
    else /* mutex not created for some reason */
      return FALSE;
  }
  else if( reason == DLL_PROCESS_DETACH && Hooker )
  {
    ShutdownGlobal();
    CloseHandle( Mutex );
  }

  return TRUE;
}

Winamp

All this discussion has got me no closer to controlling Winamp, so it is time to rectify that. The Winamp SDK, available in the form of a few header files from the Winamp web site, shows that Winamp can be operated by sending messages to the applications main window. All I need to do is translate a click on the bitmap I have placed on a title bar into a message code to Winamp. This is merely a case of working out the x-coordinate of the pointer on a mouse button press within the non-client caption area (OK: this traps a few extra areas, such as the small border above and below the bitmap, but that is hardly worth worrying about), subtracting the left hand side of the bitmap and dividing by the button width. The relevant code appears in Figure 9, the complete window procedure. Unlike the drawing code shown in Figure 4, mouse clicks return screen relative coordinates instead of windows relative ones, so there is a slightly different function, GetButtonAbsolutePosition, to return the position of the bitmap. Figure 9 also shows the straightforward handling of the disconnect message. Notice that there is no window close or destroy handler - this is just a bit more laziness on my part, since there is no real need to bother to unsubclass the window as it is about to vanish. As always, I have striven to keep the extra code in the window procedure very short since it is an extra layer within the already convoluted message handling code. There is also an argument for consuming the mouse button click totally within my code when it overlaps the bitmap instead of letting Windows field it too-this would prevent two clicks in rapid succession being interpreted as a double click which maximises the window. However, I did not want to remove existing Windows behaviour, particularly in the case where the window might be so narrow that there is no title bar beyond the bitmap.

As was hinted above, when subclassing is no longer required, a window unsubclasses itself on receipt of a detach message. This has to be managed in this way because subclassing and unsubclassing must occur in the thread that manages the window in question. Subclassing is easy, since that occurs when the CBT hook running in that thread notices an activate. Unsubclassing, however, could be triggered by another process - the system hook being uninstalled. That other process cannot perform the unsubclassing operation itself, so it sends a message to the window to be unsubclassed. Since this is a system wide message, instead of private to one application, it must be registered, using RegisterWindowMessage, as shown in Figure 7.

Figure 9: The complete window procedure.

static LONG GetButtonAbsolutePosition( HWND hwnd, DWORD style )
{
  RECT r;
  GetWindowRect( hwnd, &r );
  return r.right - WindowsFeaturesWidth( hwnd, style ) - BUTTON_WIDTH * NUM_BUTTONS;
}

LRESULT CALLBACK WindowProc( HWND hwnd, UINT msg, WPARAM wp, LPARAM lp )
{
  LRESULT res;
  DWORD style;

  if( msg == UnsubclassMessage )
  {
    UnSubclass( hwnd );
    return 0;
  }

  if( msg == WM_ACTIVATE && LOWORD( wp ) == WA_INACTIVE )
  {
    WNDPROC proc = SubclassedWindowProc; /* Because Unsubclass zeroes it */
    UnSubclass( hwnd );
    return CallWindowProc( proc, hwnd, msg, wp, lp );
  }

  res = CallWindowProc( SubclassedWindowProc, hwnd, msg, wp, lp );

  /* Only need to do something if the window has a caption */
  style = GetWindowLong( hwnd, GWL_STYLE );
  if( ( style & WS_CAPTION ) == WS_CAPTION )
    switch( msg )
    {
      case WM_NCLBUTTONDOWN:
        if( wp == HTCAPTION )
        {
          int x = GET_X_LPARAM( lp );
          int left = GetButtonAbsolutePosition( hwnd, style );
          if( x >= left && x < left + NUM_BUTTONS * BUTTON_WIDTH )
          {
            const static WPARAM buttons[ NUM_BUTTONS ] =
            { 
              WINAMP_BUTTON1,    /* Prev track */
              WINAMP_BUTTON2,    /* Play */
              WINAMP_BUTTON3,    /* Pause */
              WINAMP_BUTTON5,    /* Next track */
            };
            PostMessage( WinampWindow, WM_COMMAND, buttons[ ( x - left ) / BUTTON_WIDTH ], 0 );
          } /* end if within my button bar */
        } /* end if caption clicked */
        break;
      case WM_NCACTIVATE:
      case WM_SYSCOMMAND:
      case WM_SETTEXT:
      case WM_NCPAINT:
        DrawButtons( hwnd, style );
        break;
    } /* end switch */

  return res;
}

The CBT hook, in Figure 6, tries to unsubclass the currently subclassed window. However, I found that more often than not (and I have to confess that I do not underatand why), the hWndActive field had a zero value (at least, on NT - I did not check on any other OSes), so I had to find another way trigger unsubclassing. I thought about sending an unsubclass message to the window stored in SubclassedWindow, but would have required relatively expensive synchronisation operations to ensure thread safety. Instead, my solution was to trap the window's own deactivation via a WM_ACTIVATE handler.

The final piece of the picture is making this into a Winamp plugin, also as demonstrated in the Winamp SDK. All I have to do is provide a plugin definition structure containing a few strings and initialisation, close and configuration functions. The final one is not used here, since there is nothing to configure, but an empty function still has to be provided to prevent Winamp faulting. I could have placed the DLL global initialisation and tidy up code in the other two but, as I said earlier, I wanted to provide a more general mechanism. The plugin definition structure is initialised by my code and handed to Winamp via the function winampGetGeneralPurposePlugin. After executing this, Winamp fills in some extra fields in the structure (such as the Winamp window handle) and then calls my initialisation routine. The relevant code is shown in Figure 10.

Figure 10: Winamp integration

static int init();
static void config();
static void quit();

static winampGeneralPurposePlugin plugin =
{
  GPPHDR_VER,
  "Winamp title bar control 1.0 (gen_bktb.dll)",
  init,
  config,
  quit,
  NULL,
  NULL
};

static int init()
{
  WinampWindow = plugin.hwndParent;
  return 0;
}

static void config()
{
}

static void quit()
{
  if( SubclassedWindow )
    SendMessage( SubclassedWindow, DisconnectMessage, 0, 0 );
}

__declspec( dllexport ) winampGeneralPurposePlugin* winampGetGeneralPurposePlugin()
{
  return &plugin;
}

Summary

On the way to creating a user interface for Winamp suitable for a lazy music listener like me, I discovered more than I really wanted to know about window subclassing, system hooks and subtle Windows features. Unfortunately, I have not got to the end of the story yet - the buttons do not appear on NT command windows (though it works with Windows 95 ones) and the settings tab on my NT display properties will not display at all (again, Windows 95 is OK): more head scratching required. Notable points about the development are:

One final point is that integrating with Winamp is a doddle compared to the rest of this!

The code (updated and more fully commented than) is available, and has been tested on Windows 95, 98, NT 4.0 and 2000.

Notes on non-client area painting and activation

If you examine the code towards the bottom of the window procedure in Figure 9, you'll see that, despite what I said about WM_NCPAINT earlier, I really invoke the button drawing function on receipt of any of four windows messages - NC activation, system commands and setting the title text being unexpected extras. This is because it would appear that title bar drawing occurs in all of these, not just within the rather more obvious NC paint, as explained in Knowledge Base article Q99046. Ignoring these extras results in the bitmap not appearing on applications which recreate their title bar, such as Word, among many others. Such is life in the depths of Windows.

There is another interesting facet of NC activation: the wparam of this message is a flag indicating if the window is being activated or deactivated, so at the very least, you would expect me to draw the buttons only if the window is being activated. Unfortunately, there is a defect in MFC MDI code such that in some cases, a window being activated is sent a WM_NCACTIVATE(FALSE) immediately after a WM_NCACTIVATE(TRUE): in other words, it is deactivated as soon as it is activated! Although this is definitely a bug in MFC, similar seems to occur in some non-MFC applications such as Visual Studio and emacs. If I had used WM_NCACTIVATE(FALSE) instead of WM_ACTIVATE(WA_INACTIVE) to trap window deactivation, this behaviour would have resulted in the offending applications' windows being unsubclassed milliseconds after they were activated.


Gavin's home page | BeesKnees home page

Last modified on 4th January 2001