hypervisor: kvm: save and restore the guest shadow stack pointer - #4
Closed
tonicmuroq wants to merge 1 commit into
Closed
tonicmuroq wants to merge 1 commit into
tonicmuroq wants to merge 1 commit into
Conversation
On a host whose KVM exposes CET shadow stacks to the guest
(CPUID.(EAX=7,ECX=0):ECX[7]; e.g. Linux 7.0 on AMD Zen 3), a Windows
guest bugchecks 0x50 PAGE_FAULT_IN_NONPAGED_AREA at 0xfffffffffffffff8,
from nt!KePopulateContinuationContext, right after being restored from
some snapshots — about one in five on our node.
A vCPU paused while running a user thread with shadow stacks enabled
keeps its live SSP in the SSP register, not in MSR_IA32_PL3_SSP. KVM
exposes that register only as KVM_REG_GUEST_SSP through
KVM_{GET,SET}_ONE_REG, which Cloud Hypervisor never saved. The thread
therefore resumed with SSP=0, took a shadow-stack #PF on its next
CALL/RET, and the kernel faulted again reading 0 - 8 while building the
exception's continuation context. Snapshots taken while every vCPU was
in the kernel or running a thread without shadow stacks were unaffected,
which is why only some of them failed.
Save the register in VcpuKvmState when the guest CPUID exposes shadow
stacks and restore it after the MSRs, as QEMU does. The field is
optional, so snapshots taken before this change still restore, with the
old behaviour.
Signed-off-by: tonic <tonicbupt@gmail.com>
tonicmuroq
force-pushed
the
fix/x86-guest-ssp
branch
from
September 23, 2026 14:58
9769282 to
4ae2401
Compare
Collaborator
|
Closing in favour of the upstream PR cloud-hypervisor#8924, sent from the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a host whose KVM exposes CET shadow stacks to the guest (CPUID.(EAX=7,ECX=0):ECX[7]; seen with Linux 7.0 on an AMD EPYC 7443P / Zen 3), a Windows guest bugchecks
0x50PAGE_FAULT_IN_NONPAGED_AREA right after being restored from some snapshots. It happened to about one in five of our snapshots, and every restore of a bad snapshot failed the same way.!analyze -von the minidumps:FAILURE_BUCKET_ID: AV_(null)_nt!KePopulateContinuationContext, processsvchost.exe0xfffffffffffffff8(0 − 8), Arg20x44(user-mode, shadow-stack access)A vCPU paused while running a user thread with shadow stacks enabled keeps its live SSP in the SSP register, not in
MSR_IA32_PL3_SSP. KVM exposes that register only asKVM_REG_GUEST_SSP(0x2030000300000000) throughKVM_{GET,SET}_ONE_REG, and Cloud Hypervisor never saved it. The thread resumed with SSP=0, took a shadow-stack #PF on its next CALL/RET, and the kernel faulted again at 0 − 8 while building the exception's continuation context. Snapshots taken while every vCPU was in the kernel, or running a thread without shadow stacks, were unaffected — hence "some snapshots".What
VcpuKvmStategainsguest_ssp: Option<u64>(#[serde(default)], so older snapshots still deserialize).state()readsKVM_REG_GUEST_SSPwhen the guest CPUID exposes shadow stacks;set_state()writes it back after the MSRs, before the vCPU events — the same place QEMU restores it (i386/kvm: Add save/restore support for KVM_REG_GUEST_SSP).get_one_reg/set_one_regfor aarch64 and riscv64, so x86 issues the ioctls directly, like the existingKVM_*_DEVICE_ATTRones.Guests whose CPUID has no shadow stacks (every current Intel host we run, and any host whose KVM does not virtualize CET) take the old path unchanged.
Testing
cargo fmt --check, andcargo clippy -p hypervisor --features kvm --target x86_64-unknown-linux-gnu -- -D warnings.U_CET=1: save 9, where the vCPU hadPL3_SSPMSR0xc9513fef40andguest_sspregister0xc9513fef78. That one snapshot, restored: