How To: Debug the WRK on Mac OS X Using VMware FusionWindows Research Kernel @ HPI
Debugging the WRK on Mac OS requires two virtual machines (VM):
the WRK VM, which will be used to debug the kernel, and the debugger
VM, which hosts the kernel debugger. Both VMs will use their serial
port to communicate with each other. In a previous
post, we have shown for VirtualBox how to connect both serial
ports over a named pipe. Now, we are going to describe how to
configure VMware Fusion 3.0 virtual machines to accomplish the same
Before we begin, here is a sketch of the architecture of our WRK debug scenario:
In this tutorial, we will not describe how to configure WinDbg or the WRK to communicate via the COM1 port since this is described elsewhere. We only concentrate on the configuration of the named pipe to connect both VMs. You will notice that the following screenshots were taken on a German system installation. However, the menu locations should be the same in your language and we will also describe what is shown in the image.
If you want to configure the serial port in VMware Fusion 3.0, you will only get this dialog:
Which means that the standard configuration for the serial port is used to connect your virtual machine with printers available on Mac OS. The only other option in the GUI you have is to connect the serial port with a file, which is not appropriate for our purpose.
The solution is to manually configure the VM configuration file.
Configuration files have the suffix .vmx attached. If you are using
a version of VMware Fusion prior to version 3, you will find the
configuration file right next to the virtual hard disk file. On Mac
OS, virtual machines typically reside in
Support/VMware Fusion/Virtual Machines/, but you may be using
a different location for your VMs. If you are using VMware Fusion 3,
VMs and their associated files are organized in packages with the
suffix .vmwarevm. To locate the VM configuration file, simply right
click the .vmwarevm package in Finder and select Show Packet
Contents, as shown in the following figure for the Boot Camp VM.
After locating the .vmx files for the WRK VM and the debugger VM,
open each file with a text editor and locate the lines containing
serial0. If your VM is configured to connect with
printers, you will find the following two lines in your
serial0.present = "TRUE" serial0.fileType = "thinprint"
Please note that the contents may differ or that the serial connection may not be present at all.
Since we are using a pipe, we need to define a server, which creates the pipe, and a client, which connects to the pipe. My proposal here is to use the WRK VM as a client and the debugger VM as the server. For example, on my Mac, I am using my Boot Camp partition as debugger VM. To configure the server end of the named pipe, replace all the lines starting with serial0 in your debugger VM configuration file with the following:
serial0.present = "TRUE" serial0.fileType = "pipe" serial0.fileName = "/private/tmp/com1" serial0.tryNoRxLoss = "FALSE" serial0.pipe.endPoint = "server"
This tells VMware that the serial port (
present) and represented by a
pipe. The named pipe must be located on the local file
system in a directory, for which VMware Fusion has write
permissions. My suggestion is to use
/tmp is only a
link. The endpoint property tells VMware that this virtual machine
is supposed to create the named pipe.
To configure the client end of the named pipe,
replace all the lines starting with
serial0 in your WRK
VM configuration file with the following:
serial0.present = "TRUE" serial0.fileType = "pipe" serial0.fileName = "/private/tmp/com1" serial0.tryNoRxLoss = "FALSE" serial0.pipe.endPoint = "client"
Again, we are using the named pipe
but this time as a client, which means that VMware expects the named
pipe to already being created. Apparently, this specification has
implications on the boot order of your virtual machines: since the
debugger VM is the server, it must be started prior to the WRK
Now safe the configuration files and close the editor. If you
check the virtual machine settings in VMware Fusion for either VM,
you will now find the following options dialog for the serial port
(which reads: Use user defined, unsupported settings):
If you boot the debugger VM first and then the WRK VM in
/debug mode with
should be able to connect the kernel debugger with the WRK.
Please note that this description is provided "as is" with absolutely no warranties. Since we are using undocumented properties, the configuration described herein may not be working in future releases of VMware Fusion. If that should be the case or you found out another way of how to connect two VMs with a named pipe in a documented fashion, please feel free to post your experiences here.