How To: Debug the WRK on Mac OS X Using VMware Fusion

By Alexander Schmidt

Debugging the WRK on Mac OS requires two virtual machines (VM): the WRK VM, which will be used to debug the kernel, and the debugger VM, which hosts the kernel debugger. Both VMs will use their serial port to communicate with each other. In a previous post, we have shown for VirtualBox how to connect both serial ports over a named pipe. Now, we are going to describe how to configure VMware Fusion 3.0 virtual machines to accomplish the same goal.

Before we begin, here is a sketch of the architecture of our WRK debug scenario:

In this tutorial, we will not describe how to configure WinDbg or the WRK to communicate via the COM1 port since this is described elsewhere. We only concentrate on the configuration of the named pipe to connect both VMs. You will notice that the following screenshots were taken on a German system installation. However, the menu locations should be the same in your language and we will also describe what is shown in the image.

If you want to configure the serial port in VMware Fusion 3.0, you will only get this dialog:

Which means that the standard configuration for the serial port is used to connect your virtual machine with printers available on Mac OS. The only other option in the GUI you have is to connect the serial port with a file, which is not appropriate for our purpose.

The solution is to manually configure the VM configuration file. Configuration files have the suffix .vmx attached. If you are using a version of VMware Fusion prior to version 3, you will find the configuration file right next to the virtual hard disk file. On Mac OS, virtual machines typically reside in ~/Library/Application Support/VMware Fusion/Virtual Machines/, but you may be using a different location for your VMs. If you are using VMware Fusion 3, VMs and their associated files are organized in packages with the suffix .vmwarevm. To locate the VM configuration file, simply right click the .vmwarevm package in Finder and select Show Packet Contents, as shown in the following figure for the Boot Camp VM.

After locating the .vmx files for the WRK VM and the debugger VM, open each file with a text editor and locate the lines containing serial0. If your VM is configured to connect with printers, you will find the following two lines in your configuration file:

serial0.present = "TRUE"
serial0.fileType = "thinprint"

Please note that the contents may differ or that the serial connection may not be present at all.

Since we are using a pipe, we need to define a server, which creates the pipe, and a client, which connects to the pipe. My proposal here is to use the WRK VM as a client and the debugger VM as the server. For example, on my Mac, I am using my Boot Camp partition as debugger VM. To configure the server end of the named pipe, replace all the lines starting with serial0 in your debugger VM configuration file with the following:

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "/private/tmp/com1"
serial0.tryNoRxLoss = "FALSE"
serial0.pipe.endPoint = "server"

This tells VMware that the serial port (serial0) is available (present) and represented by a pipe. The named pipe must be located on the local file system in a directory, for which VMware Fusion has write permissions. My suggestion is to use /tmp/com1 or /private/tmp/com1, since /tmp is only a link. The endpoint property tells VMware that this virtual machine is supposed to create the named pipe.

To configure the client end of the named pipe, replace all the lines starting with serial0 in your WRK VM configuration file with the following:

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "/private/tmp/com1"
serial0.tryNoRxLoss = "FALSE"
serial0.pipe.endPoint = "client"

Again, we are using the named pipe /private/tmp/com1 but this time as a client, which means that VMware expects the named pipe to already being created. Apparently, this specification has implications on the boot order of your virtual machines: since the debugger VM is the server, it must be started prior to the WRK VM.

Now safe the configuration files and close the editor. If you check the virtual machine settings in VMware Fusion for either VM, you will now find the following options dialog for the serial port (which reads: Use user defined, unsupported settings):

If you boot the debugger VM first and then the WRK VM in /debug mode with /debugport=com1, you should be able to connect the kernel debugger with the WRK.

DISCLAIMER

Please note that this description is provided "as is" with absolutely no warranties. Since we are using undocumented properties, the configuration described herein may not be working in future releases of VMware Fusion. If that should be the case or you found out another way of how to connect two VMs with a named pipe in a documented fashion, please feel free to post your experiences here.

Comments