In part 2 of my explorations into Hypervisor-Managed Linear Address Translation (HLAT), I described where and when HLAT was enabled when Windows boots on an Intel Alder Lake CPU. In this installment, I dive deeper into the specifics of HLAT enablement on each “performance” logical processor.
Author’s Note: in Part 2, I mistakenly observed that all P-cores had HLAT enabled once P9 entered the NT normal kernel, when the 12th VM Launch breakpoint was hit. This turned out to be incorrect: the logical processors are enabled for HLAT in a deterministic, but unusual, pattern. I’ve since updated that prior blog to be more clear. And to see the pattern visually, here’s an update to the logical processor behavior as the target boots through 24 successive SourcePoint VM Launch breakpoints:
The legend associated with this matrix bears some explanation:
The rows display the status of each logical processor through successive VM Launch breaks, 1-24.
P0 – P7 are the efficient “E-cores” on this Core i7-1270PE Alder Lake CPU. The E-cores are single-threaded, with no hyperthreading, and there are eight of them.
P8 – P15 are the performance “P-cores”. These P-cores have hyperthreading enabled, so there are four cores and eight threads. We’ll call each thread a logical processor, for the sake of clarity. P8 is the “bootstrap” processor that runs alone single-threaded as UEFI boots, and then as Windows initializes first the rest of the p-cores and then the e-cores, code is run on these as well.
“P” in a cell means that the logical processor is in Protected Mode. These are in UEFI. If we use SourcePoint to tell us what program we are in, we find that these are running BootScriptExecutorDxe. An interesting note: SourcePoint will also tell you the build path when this AMI BIOS was built, which is:
C:\release\3009\Build\AlderLake\RELEASE_VS2015\X64\MdeMoldulePkg\Universal\Acpi\BootScriptExecutorDxe\DEBUG\
Editor’s Note: Please don’t confuse the “P” as in Protected, and the prefix “P” as in logical processor.
Note that ShvlpVtl1Entry and SkeStartProcessor are functions within the Secure Kernel, and KeStallExecutionProcessor, KiInitializeKernel and HalpLMStubForVM are in the Normal Kernel.
The cells colored green show the boot flow. For example, looking at the first VM Launch, the break occurs on P8, where it enters the Windows hypervisor, hvix64, in Guest mode. The second VM Launch is also encountered on P8, where we land in hvloader, again in Guest mode. The third VM Launch finds P8 in the Secure Kernel, in the function ShvlpVtl1Entry. The fourth VM Launch break occurs on P9, where it now enters hvix64 in Guest mode; at which point the bootstrap processor, P8, has done a VM Exit and made the transition to hvix64 in Host mode. And so on.
The key addition to the previous matrix is that when a particular logical processor is enabled for HLAT, there’s a cross-hatching pattern in the cell.
Given the above, it’s worthwhile staring at the matrix for a while; it provides a lot of insight into the Windows boot flow with Hyper-V enabled, and the pattern of HLAT initialization. I’ll highlight just a few points of interest:
At the 10th VM Launch breakpoint, all P-cores are in the Windows VMM, and all E-cores are still in UEFI.
When the 11th VM Launch breakpoint is hit, P9 has done a launch into the Secure Kernel ShvlpVtl1Entry function. At this moment in time, P8 is at SkeStartProcessor. Why is P9 then the active processor, you may ask, given that both of these processors are now in the Secure Kernel? Stare at it a little longer: you’ll see that P8 had already done a VM Launch into ShvlpVtl1Entry back at the 3rd VM Launch break, so in this case there would be a VM Resume into the Secure Kernel for P8; and we haven’t set a VM Resume breakpoint yet, just a VM Launch one. Cool, huh?
Note also that all of the E-cores have made a transition from UEFI into hvix64 in Host mode.
Then on the 12th VM Launch break, P9 does a VM Launch into the NT normal kernel, to ntkrnlmp!HalpLMStubForVM. At the same time, P8 is halted at KeStallExecutionProcessor. Note that the cross-hatching indicates that both P8 and P9 now have HLAT enabled. Both of these functions are in the NTOS, so presumably a VM Launch is required for both of these logical processors, as the NT kernel is new for both of them. So why was the break encountered at HalpLMStubForVM, as opposed to KeStallExecutionProcessor? A possible clue: HalpLMStubForVM only has two instructions:
mov rdi, rsp jmp ntkrnlmp!HalpLMStubVMTarget
Any thoughts?
An interesting pattern does emerge on P8, if you look closely: after the 10th VM Launch break, it alternates between SkeStartProcessor to KeStallExecutionProcessor, and each time the NT kernel is enabled for HLAT, but the Secure Kernel is not. HLAT turns on and off on P8. Why is this? Again, theories are welcome.
A short advertisement: if you’d like to know more about exploring the operation of the Windows hypervisor, visit our online store here.
