How To: Debug the WRK on Mac OS X Using VMware FusionWindows Research Kernel @ HPI
Debugging the WRK on Mac OS requires two virtual machines (VM):
the WRK VM, which will be used to debug the kernel, and the debugger
VM, which hosts the kernel debugger. Both VMs will use their serial
port to communicate with each other. In a previous
post, we have shown for VirtualBox how to connect both serial
ports over a named pipe. Now, we are going to describe how to
configure VMware Fusion 3.0 virtual machines to accomplish the same
Before we begin, here is a sketch of the architecture of our WRK debug scenario:
In this tutorial, we will not describe how to configure WinDbg or the WRK to communicate via the COM1 port since this is described elsewhere. We only concentrate on the configuration of the named pipe to connect both VMs. You will notice that the following screenshots were taken on a German system installation. However, the menu locations should be the same in your language and we will also describe what is shown in the image.
If you want to configure the serial port in VMware Fusion 3.0, you will only get this dialog:
Which means that the standard configuration for the serial port is used to connect your virtual machine with printers available on Mac OS. The only other option in the GUI you have is to connect the serial port with a file, which is not appropriate for our purpose.
The solution is to manually configure the VM configuration file.
Configuration files have the suffix .vmx attached. If you are using
a version of VMware Fusion prior to version 3, you will find the
configuration file right next to the virtual hard disk file. On Mac
OS, virtual machines typically reside in
Support/VMware Fusion/Virtual Machines/, but you may be using
a different location for your VMs. If you are using VMware Fusion 3,
VMs and their associated files are organized in packages with the
suffix .vmwarevm. To locate the VM configuration file, simply right
click the .vmwarevm package in Finder and select Show Packet
Contents, as shown in the following figure for the Boot Camp VM.
After locating the .vmx files for the WRK VM and the debugger VM,
open each file with a text editor and locate the lines containing
serial0. If your VM is configured to connect with
printers, you will find the following two lines in your
serial0.present = "TRUE" serial0.fileType = "thinprint"
Please note that the contents may differ or that the serial connection may not be present at all.
Since we are using a pipe, we need to define a server, which creates the pipe, and a client, which connects to the pipe. My proposal here is to use the WRK VM as a client and the debugger VM as the server. For example, on my Mac, I am using my Boot Camp partition as debugger VM. To configure the server end of the named pipe, replace all the lines starting with serial0 in your debugger VM configuration file with the following:
serial0.present = "TRUE" serial0.fileType = "pipe" serial0.fileName = "/private/tmp/com1" serial0.tryNoRxLoss = "FALSE" serial0.pipe.endPoint = "server"
This tells VMware that the serial port (
present) and represented by a
pipe. The named pipe must be located on the local file
system in a directory, for which VMware Fusion has write
permissions. My suggestion is to use
/tmp is only a
link. The endpoint property tells VMware that this virtual machine
is supposed to create the named pipe.
To configure the client end of the named pipe,
replace all the lines starting with
serial0 in your WRK
VM configuration file with the following:
serial0.present = "TRUE" serial0.fileType = "pipe" serial0.fileName = "/private/tmp/com1" serial0.tryNoRxLoss = "FALSE" serial0.pipe.endPoint = "client"
Again, we are using the named pipe
but this time as a client, which means that VMware expects the named
pipe to already being created. Apparently, this specification has
implications on the boot order of your virtual machines: since the
debugger VM is the server, it must be started prior to the WRK
Now safe the configuration files and close the editor. If you
check the virtual machine settings in VMware Fusion for either VM,
you will now find the following options dialog for the serial port
(which reads: Use user defined, unsupported settings):
If you boot the debugger VM first and then the WRK VM in
/debug mode with
should be able to connect the kernel debugger with the WRK.
Please note that this description is provided "as is" with absolutely no warranties. Since we are using undocumented properties, the configuration described herein may not be working in future releases of VMware Fusion. If that should be the case or you found out another way of how to connect two VMs with a named pipe in a documented fashion, please feel free to post your experiences here.
2 Responses to "How To: Debug the WRK on Mac OS X Using VMware Fusion"
I want to thank you for your wonderful post. Even though I am still struggling to get it to work, the fact that someone out there has really succeeded in getting this to work gives me the courage to keep going. Thank you for being so generous and writing it up. It's like a present from overseas in the new year!
I do not want to trouble you, but even following your instructions precisely, I do not get the WinDbg to connect to the WRK which is set to run in /debug mode over /com1.
Two things appear suspicious in my setup and I was wondering if you could confirm what they look like on yours:
1) The debugger VM (in my case running Windows 7) shows 2 COM ports under device manager even though only 1 serial port is listed under VMware settings and the vmx file specifies only serial0…
2) The WRK VM (in my case Windows 2003 SP1) shows only port under device manager but it is COM2 port (no COM1), and only 1 serial port under VMware settings); the vmx file specifies only serial0… Note that the kernel debugport was set to COM1.
Now, if I change the WRK kernel debugport to COM2, then the WRK VM (upon reboot) shows only the COM1 port under device manager but no is COM2 port, and only 1 serial port under VMware settings); the vmx file specifies only serial0. It is as though the port is deliberately trying to run away ☹️
Regardless, I am not able to get WinDbg to connect remotely to the WRK… the "Debugee not connected".
Any clues would be greatly appreciated.
This might be useful to others:
"When you enable kernel debugging on a serial port, Windows removes the specified port from the system device list. As a result, on computers with an ACPI BIOS, the port does not appear in any device lists, such as the one in Device Manager. On computers that do no have an ACPI BIOS, the port appears with an error message, such as "Not enough resources to use this port." These messages indicate that the port is under the control of the host debugging computer; they do not indicate a malfunction."