Remote Debugging using VMWare

6 minute read  • 

win32

Introduction

There are two ways to debug a device-driver in Windows. The first is to install the driver on your development machine and run a local kernel-mode debugger such as SoftIce. Although SoftIce is a very powerful tool, debugging a driver on your development machine is not a good idea because you will be guaranteed to destroy all your work if the driver crashes the system. You really need to install your driver on a separate machine and use remote kernel-debugging with a null-modem serial cable to protect your development box.

However if you are like me and either can’t afford a separate machine, or simply don’t have the space to keep one then keep reading because there is a nice solution for you. We will be using virtual-machine technology (i.e. VMWare or VirtualPC ), and using named pipes to emulate the serial-port connections.

Installing WinDbg

The preferred debugging method by most developers is to use Microsoft WinDbg - firstly because it is free - but more importantly because it is an incredibly powerful kernel-mode debugger which can be used to remotely debug a target machine via a null-modem serial cable. Follow the instructions below to install and configure WinDbg for remote debugging:

  1. Install WinDbg from the following location:

    http://www.microsoft.com/whdc/devtools/debugging/default.mspx

  2. Open the Symbol Search Path dialog from the file menu:

    File -> Symbol File Path... (Ctrl+S)

  3. Configure WinDbg to use Microsoft’s online Symbol Server and cache debug symbols locally:

    srv*c:\windows\symbols*http://msdl.microsoft.com/download/symbols

  4. Create a shortcut on your desktop to WinDbg, and specify the following command line:

    windbg -b -k com:pipe,port=\\.\pipe\com_1,resets=0

Once you have configured WinDbg, use the shortcut you created in step#4 to attach to a “remote” machine which has debugging enabled. In order to debug part of your driver, you will usually embed an “int 3” to make the debugger break. And if you installed the debug version of your driver, WinDbg will automatically find the corresponding source-file for that driver and step through it line-by-line.

Configure VMWare to be the remote host

VMWare is an amazing product which allows you to run a complete OS inside a single window on your PC. The product is far cheaper than buying a separate machine and is infinitely more flexible, because you can host multiple OS images (saved as files on your hard-disk). For example, you could have every version of Windows available to you for testing, all on one physical PC!

Once you have VMWare installed (note: VirtualPC from Microsoft will also work, but it isn’t quite as good), follow the steps below to enable the virtual machine to act as a kernel-debug target for WinDbg. The idea is that instead of using a null-modem cable between physical machines, we can set up a named pipe and attach it to the virtual machine’s COM port. WinDbg will then attach to the named pipe, and remote-debug the hosted OS inside VMWare.

The sequence of steps below was originally found at http://silverstr.ufies.org/lotr0 so I have reproduced them here.

  1. Open the Configuration Editor from the Settings menu

  2. Click the Add button to open the Add Hardware Wizard.

  3. Select Serial Port and click the Next button.

  4. Select Use Named Pipe , and change the name to: \\.\pipe\com_1

  5. Select This end is the server.

  6. Select The other end is an application.

  7. Click on Advanced , and select the " Yield on CPU poll" option.

  8. Click Finish and then OK to save your options. When you power up your VM image, the Devices->Serial0 menu will be available.

  9. When the virtual machine boots up, edit the boot.ini file inside the VM OS’s root directory. Add a new OS line to tell the “virtual” Windows OS (when it boots) to enable it’s kernel debugger. Two options need to be added to enable this feature:

    /debugport=com1 /baudrate=115200

    In practise you simply copy one of your OS configurations in boot.ini onto a new line, and add the additional two options onto the end:

    [boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - DEBUG" /fastdetect /debugport=com1 /baudrate=115200

    Note that all the options should be on one line, not wrapped-around like they are above. Save boot.ini (it will probably be read-only so disable that first) and reboot.

  10. At this point take a SnapShot of your VM Image so that you can roll back to this state at any time.

Using VMWare’s SnapShot feature is incredibly useful. Whenever your driver causes a blue-screen, you can simply Restore the VM Image and have another go! This saves an amazing amount of time - the repetitive task of rebooting, installing drivers and setting up the environment instantly disappears. If you get into this habit device driver development will be far less tiresome.

Debugging using WinDbg

To debug your driver simply reboot the VMWare OS. You will be presented with the OS loader menu which was modified by editing the boot.ini file. Select the DEBUG configuration and hit return!

Once the OS has started booting you can simply double-click your WinDbg shortcut and WinDbg will automatically attach itself to the VMWare session (using the named pipe “com_1”) and enter the kernel-debug mode. Note that this isn’t the same as running the debug version of Windows - we have simply enabled the guest OS’s kernel debugger that is present in all Windows NT OSs. Once you have “broken in” to the debuggee, the entire virtual machine will be stopped - not even the mouse will be active inside VMWare.

At this point you can just hit F5 in WinDbg to let the hosted OS continue booting.

Conclusion

I strongly advise anyone doing software development, and especially those doing driver work, to invest in either VMWare or VirtualPC. It’s a one-off payment and you won’t regret spending the money. The flexibility and ease of use it gives you for remote debugging just cannot be emphasised enough. Although a stand-alone dedicated test machine is better, the VM solution can solve many of your debugging scenarios.

One important thing to mention is the use of debug symbols. At the top of this tutorial we configured WinDbg to use Microsoft’s Debug Symbol Server. WinDbg will automatically connect to this server and download the appropriate debug symbols for your target operating system - so you won’t have to manually install symbols.

Note that we do not require the debug version of Windows. There are two versions of any Windows product - Debug and Retail (or Normal). The debug build contains a special version of the kernel which can be used to aid debugging. This modified kernel contains extra code and sanity checks to help you debug the kernel and your drivers. Not only is it quite a bit slower than the normal OS, but the modified kernel behaviour can be unhelpful at times. All we require are the debug symbols for the Release version of Windows - which is where the symbol server comes in useful.

The last thing to mention is the use of SMP machines (dual/multi-processor machines). All good programmers understand how important it is to develop (and test) on a multi-processor machine. This is especially important with device-driver writing. The problem with VMWare is that it only uses one of your CPUs even if you have two installed** - so at some point you will have to bite the bullet and buy a dual-processor machine. But it will be worth it just for the extra horse power!

**Update: The latest versions of VMWare now do support SMP machines, but there is still no substitute for a real hardware when you’re testing driver-code.

Well hopefully you have found this tutorial useful, if you have any feedback I’d love to hear it..