Building Complete Call Graphs with Phoenix

Windows Research Kernel @ HPI

In my last post, I told you that we use Phoenix for building the WRK, which allows us to apply Phoenix' comprehensive set of analysis capabilities to the WRK. One particular analysis might be the construction of complete call graphs for functions interest.

Unfortunately, the build process of the WRK compiles each module separately and links it into a static library. As a final step, all static libraries will be linked together with pre-compiled libraries to the ntoskrnl.exe executable image. So building complete call graphs may be a problem, especially when a function calls or is called by a function within another module as Phoenix is only aware of functions within the compiled module.

But fortunately, Phoenix provides a solution to this problem!

Phoenix supports link-time code generation (LTCG), which is originally intended for whole program analysis. When using LTCG, Phoenix does not compile modules separately into COFF object files. It rather stores the intermediate language representation of that module in the object file or static library, respectively. Then, at link time, Phoenix is invoked again to compile all the modules at the same time, which allows global optimization or, in our case, enables us to provide a Phoenix Phase plug-in to construct a complete call graph of a function.

In order to enable LTCG with the WRK, you have to modify the provided makefiles a little bit. You must ensure that the compiler does not generate object code. This is done by the GL switch. As this is a target independent feature, add the /GL switch to the copts variable in file, which is in directory base/ntos/BUILD. Next, we must notify lib.exe that it has to link itermediate language files into static libraries. For the record, lib.exe determines that on its own, but declaring that fact explicitly may reduce link time. So add the /LTCG switch to the LIBFLAGS variable of the same makefile (base/ntos/BUILD/

Next, we need to inform the linker that it has to invoke Phoenix before linking the static libraries together. This is done again by the /LTCG switch. So edit the makefile (not!) in directory ntos/base/BUILD. I suggest here to modify the LINKFLAGS variable to contain the switch.

Finally, we need to inform Phoenix about what plug-in should be used. I recommend using the phx environment variable, which Phoenix will evaluate for command line options. As we want Phoenix to invoke the plug-in only at the link step and not for each module, we suggest to set the variable only for the linker. Modify therefore the kernelexe rule in build/ntos/BUILD/makefile. Below is a sample of our modification:

        @set phx=-plugin:path\to\plugin\plugin.dll <More Options>
        $(LINK) $(LINKFLAGS) $(ntoslinkopts) -out:$(fullkernel).exe -map:$(fullkernel).map -pdb:$(fullkernel).pdb -entry:$(entrypoint) \
                $(hotpatch) PREBUILT\$(targ)\ntoskrnl.res $(OBJ)\ntkrnlmp.obj $(OBJ)\*.lib $(ntoswrklib) hal.lib $(fullkernel).exp $(bootlibs)
        $(LINKEDIT) -section:.rsrc,!d $(fullkernel).exe

The first line of this rule sets the environment variable phx. The -plugin option allows to specify the DLL that contains your Phoenix plug-in. You may or may not specify other options as well.

I wrote a plug-in which generates a call graph in the dot format for each function being compiled. The call graph contains both callers and callees of the compiled function. To enhance readability, I limited the depth of the graph to 4 levels for callers and callees. Potentially, you may construct a complete call graph, but my experience shows, that this generates graphs that are too complex to handle for dot. But you may experiment about that on your own 😉 . Click on this link to see the call graph that was generated for the MiReplaceWorkingSetEntry function. This function will become important in one of my next posts. So keep watching …


3 Responses to "Building Complete Call Graphs with Phoenix"

  1. Martim on October 20th, 2008 00:36

    Nice post, useful information. When compiling the WRK with Phenix on LTCG mode, did you get lots of alignment-related warnings in the linking stage? For example:

    C:\Source\WRK-v1.2\base\ntos\mm\mmfault.c : warning C4742: '_MmAvailablePages' has different alignment in 'C:\Source\WRK-v1.2\base\ntos\mm\mmfault.c' and 'C:\Source\WRK-v1.2\base\ntos\mm\miglobal.c': 4 and 64

    If yes, did you do anything to solve it or did you just ignore it?


  2. Alexander Schmidt on October 28th, 2008 17:00

    Hi Martim,

    I just ignore the C4742 warning right now, as I don't care if the WRK does boot or not. So I just disabled the warning for the compiler.

    At the current release, I don't think Phoenix is capable of compiling the WRK as it is shipped without any modifications in such a way that it boots on a target machine. If you made it, just let me know what is necessary to fix it …

  3. Chelsea M Heffner on February 22nd, 2009 03:56

    I too would like to know the fix. Disabling the warning gets rid of the message but I'd prefer not to have any warnings in the first place. Less buggy code means less trouble down the road.