[SRU][Focal][PATCH 0/1] kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[SRU][Focal][PATCH 0/1] kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts

Matthew Ruffell
BugLink: https://bugs.launchpad.net/bugs/1911848

[Impact]

On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail
to start once they have enabled the hyper-v role for nested virtualisation.

The Windows Server guests will get stuck in the late stages of boot, before the
graphical login screen appears, on Windows Server systems with the desktop
environment installed.

If you look at performance metrics for the guest, the CPU will appear to be
stuck at 100%, and it never changes from 100%. The Windows Server guest is
unresponsive.

The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very
specific settings needed for nested virtualisation. See testcase section.
If you use any other vcpu type, the problem does not reproduce.

Known workarounds are to install the 5.8 HWE kernel, in which case the server
will come up as expected.

[Fix]

The following commit fixes the issue, and landed in mainline 5.8-rc1:

commit 8081ad06b68a728e676d3b08e9ab70ce4039747b
Author: Sean Christopherson <[hidden email]>
Date:   Wed Apr 22 19:25:40 2020 -0700
Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set
Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b

It appears that pending requests to the hypervisor can be lost or delayed if
an immediate exit was requested in vcpu_enter_guest(). As the commit message
mentions, only the !injected case is affected, so we add a check at the
cancel_injection label to see if we got there as a result of an immediate exit,
and then re-issue a KVM_REQ_EVENT request if we are.

The Windows guest is waiting for an event to be processed, which never happens,
and so gets stuck.

Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my
testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the
issue, so SRU to Bionic will not be needed.

[Testcase]

A cascadelake based Xeon server is required. Anything else and the bug will not
reproduce.

I used a c5.metal server on AWS. It has the following processor:
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz

Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach
the desktop. Copy a Windows Server 2k19 image to the destination server, as well
as a recent ISO image of virtio drivers.

Install virt-manager.

Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb
ram. Click customise button to change settings before install.

Change the hard disk to be SATA, attach a new cd rom drive for the virtio
drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX.

Edit the virsh xml, and ensure you set the following features for CPU:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server-noTSX</model>
    <topology sockets='8' cores='1' threads='1'/>
    <feature policy='require' name='invpcid'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
  </cpu>

Those settings are an absolute must.

Boot the VM, and install Windows 2k19 with the desktop environment. Once it is
installed, open up computer management > device manager and install drivers from
the virtio ISO for missing hardware, likely the network and balloon devices.

From there, go to server manager, and install the hyper-v role.

Reboot the server. It will reboot a few times, and on the final time, it will
lock up before it reaches the log in screen.

In virt-manager, go to the performance tab. The CPU will be stuck at 100%.
The windows guest will be non responsive.

A patched kernel is available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test

If you install this kernel and boot the Windows 2k19 guest, it will come up
normally when the hyper-v role is enabled, and you will be able to log in.

[Where problems could occur]

This is a change to a core part of the kvm subsystem, so there is potential
for regression which could affect all users of KVM.

If a regression were to occur, there are no workarounds. Users would need to
downgrade their kernel while a fix is developed.

Sean Christopherson (1):
  KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit
    set

 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)

--
2.27.0


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[SRU][Focal][PATCH 1/1] KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set

Matthew Ruffell
From: Sean Christopherson <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1911848

Re-request KVM_REQ_EVENT if vcpu_enter_guest() bails after processing
pending requests and an immediate exit was requested.  This fixes a bug
where a pending event, e.g. VMX preemption timer, is delayed and/or lost
if the exit was deferred due to something other than a higher priority
_injected_ event, e.g. due to a pending nested VM-Enter.  This bug only
affects the !injected case as kvm_x86_ops.cancel_injection() sets
KVM_REQ_EVENT to redo the injection, but that's purely serendipitous
behavior with respect to the deferred event.

Note, emulated preemption timer isn't the only event that can be
affected, it simply happens to be the only event where not re-requesting
KVM_REQ_EVENT is blatantly visible to the guest.

Fixes: f4124500c2c13 ("KVM: nVMX: Fully emulate preemption timer")
Signed-off-by: Sean Christopherson <[hidden email]>
Message-Id: <[hidden email]>
Signed-off-by: Paolo Bonzini <[hidden email]>
(backported from commit 8081ad06b68a728e676d3b08e9ab70ce4039747b)
[mruffell: minor context adjustment]
Signed-off-by: Matthew Ruffell <[hidden email]>
---
 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e519e0af028a..dddbeb3ee2c6 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8345,6 +8345,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
  return r;
 
 cancel_injection:
+ if (req_immediate_exit)
+ kvm_make_request(KVM_REQ_EVENT, vcpu);
  kvm_x86_ops->cancel_injection(vcpu);
  if (unlikely(vcpu->arch.apic_attention))
  kvm_lapic_sync_from_vapic(vcpu);
--
2.27.0


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK/Cmnt: [SRU][Focal][PATCH 1/1] KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set

Stefan Bader-2
On 19.01.21 00:02, Matthew Ruffell wrote:

> From: Sean Christopherson <[hidden email]>
>
> BugLink: https://bugs.launchpad.net/bugs/1911848
>
> Re-request KVM_REQ_EVENT if vcpu_enter_guest() bails after processing
> pending requests and an immediate exit was requested.  This fixes a bug
> where a pending event, e.g. VMX preemption timer, is delayed and/or lost
> if the exit was deferred due to something other than a higher priority
> _injected_ event, e.g. due to a pending nested VM-Enter.  This bug only
> affects the !injected case as kvm_x86_ops.cancel_injection() sets
> KVM_REQ_EVENT to redo the injection, but that's purely serendipitous
> behavior with respect to the deferred event.
>
> Note, emulated preemption timer isn't the only event that can be
> affected, it simply happens to be the only event where not re-requesting
> KVM_REQ_EVENT is blatantly visible to the guest.
>
> Fixes: f4124500c2c13 ("KVM: nVMX: Fully emulate preemption timer")
> Signed-off-by: Sean Christopherson <[hidden email]>
> Message-Id: <[hidden email]>
> Signed-off-by: Paolo Bonzini <[hidden email]>
> (backported from commit 8081ad06b68a728e676d3b08e9ab70ce4039747b)
> [mruffell: minor context adjustment]
> Signed-off-by: Matthew Ruffell <[hidden email]>
Acked-by: Stefan Bader <[hidden email]>
> ---

The regression potential is probably covering all bases but also sounding rather
scary that way. I would say practically the change adds an exit case from
immediate re-schedule (iiuc). So regression potential would be stuck guests with
no cpu activity instead of 100% as before. Yeah, not sure that sounds better.
But at least a little more specific.

-Stefan

>  arch/x86/kvm/x86.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index e519e0af028a..dddbeb3ee2c6 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -8345,6 +8345,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>   return r;
>  
>  cancel_injection:
> + if (req_immediate_exit)
> + kvm_make_request(KVM_REQ_EVENT, vcpu);
>   kvm_x86_ops->cancel_injection(vcpu);
>   if (unlikely(vcpu->arch.apic_attention))
>   kvm_lapic_sync_from_vapic(vcpu);
>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

ACK: [SRU][Focal][PATCH 0/1] kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts

William Breathitt Gray
In reply to this post by Matthew Ruffell
On Tue, Jan 19, 2021 at 12:02:31PM +1300, Matthew Ruffell wrote:

> BugLink: https://bugs.launchpad.net/bugs/1911848
>
> [Impact]
>
> On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail
> to start once they have enabled the hyper-v role for nested virtualisation.
>
> The Windows Server guests will get stuck in the late stages of boot, before the
> graphical login screen appears, on Windows Server systems with the desktop
> environment installed.
>
> If you look at performance metrics for the guest, the CPU will appear to be
> stuck at 100%, and it never changes from 100%. The Windows Server guest is
> unresponsive.
>
> The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very
> specific settings needed for nested virtualisation. See testcase section.
> If you use any other vcpu type, the problem does not reproduce.
>
> Known workarounds are to install the 5.8 HWE kernel, in which case the server
> will come up as expected.
>
> [Fix]
>
> The following commit fixes the issue, and landed in mainline 5.8-rc1:
>
> commit 8081ad06b68a728e676d3b08e9ab70ce4039747b
> Author: Sean Christopherson <[hidden email]>
> Date:   Wed Apr 22 19:25:40 2020 -0700
> Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set
> Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b
>
> It appears that pending requests to the hypervisor can be lost or delayed if
> an immediate exit was requested in vcpu_enter_guest(). As the commit message
> mentions, only the !injected case is affected, so we add a check at the
> cancel_injection label to see if we got there as a result of an immediate exit,
> and then re-issue a KVM_REQ_EVENT request if we are.
>
> The Windows guest is waiting for an event to be processed, which never happens,
> and so gets stuck.
>
> Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my
> testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the
> issue, so SRU to Bionic will not be needed.
>
> [Testcase]
>
> A cascadelake based Xeon server is required. Anything else and the bug will not
> reproduce.
>
> I used a c5.metal server on AWS. It has the following processor:
> Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
>
> Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach
> the desktop. Copy a Windows Server 2k19 image to the destination server, as well
> as a recent ISO image of virtio drivers.
>
> Install virt-manager.
>
> Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb
> ram. Click customise button to change settings before install.
>
> Change the hard disk to be SATA, attach a new cd rom drive for the virtio
> drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX.
>
> Edit the virsh xml, and ensure you set the following features for CPU:
>
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Cascadelake-Server-noTSX</model>
>     <topology sockets='8' cores='1' threads='1'/>
>     <feature policy='require' name='invpcid'/>
>     <feature policy='require' name='pcid'/>
>     <feature policy='require' name='vmx'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='disable' name='mpx'/>
>     <feature policy='require' name='pku'/>
>     <feature policy='require' name='arch-capabilities'/>
>     <feature policy='require' name='rdctl-no'/>
>     <feature policy='require' name='ibrs-all'/>
>     <feature policy='require' name='skip-l1dfl-vmentry'/>
>     <feature policy='require' name='mds-no'/>
>   </cpu>
>
> Those settings are an absolute must.
>
> Boot the VM, and install Windows 2k19 with the desktop environment. Once it is
> installed, open up computer management > device manager and install drivers from
> the virtio ISO for missing hardware, likely the network and balloon devices.
>
> From there, go to server manager, and install the hyper-v role.
>
> Reboot the server. It will reboot a few times, and on the final time, it will
> lock up before it reaches the log in screen.
>
> In virt-manager, go to the performance tab. The CPU will be stuck at 100%.
> The windows guest will be non responsive.
>
> A patched kernel is available in the following ppa:
>
> https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test
>
> If you install this kernel and boot the Windows 2k19 guest, it will come up
> normally when the hyper-v role is enabled, and you will be able to log in.
>
> [Where problems could occur]
>
> This is a change to a core part of the kvm subsystem, so there is potential
> for regression which could affect all users of KVM.
>
> If a regression were to occur, there are no workarounds. Users would need to
> downgrade their kernel while a fix is developed.
>
> Sean Christopherson (1):
>   KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit
>     set
>
>  arch/x86/kvm/x86.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> --
> 2.27.0
>
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
Acked-by: William Breathitt Gray <[hidden email]>

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

APPLIED: [SRU][Focal][PATCH 0/1] kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts

Kelsey Skunberg
In reply to this post by Matthew Ruffell

Applied to Focal/master-next. thank you!

-Kelsey

On 2021-01-19 12:02:31 , Matthew Ruffell wrote:

> BugLink: https://bugs.launchpad.net/bugs/1911848
>
> [Impact]
>
> On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail
> to start once they have enabled the hyper-v role for nested virtualisation.
>
> The Windows Server guests will get stuck in the late stages of boot, before the
> graphical login screen appears, on Windows Server systems with the desktop
> environment installed.
>
> If you look at performance metrics for the guest, the CPU will appear to be
> stuck at 100%, and it never changes from 100%. The Windows Server guest is
> unresponsive.
>
> The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very
> specific settings needed for nested virtualisation. See testcase section.
> If you use any other vcpu type, the problem does not reproduce.
>
> Known workarounds are to install the 5.8 HWE kernel, in which case the server
> will come up as expected.
>
> [Fix]
>
> The following commit fixes the issue, and landed in mainline 5.8-rc1:
>
> commit 8081ad06b68a728e676d3b08e9ab70ce4039747b
> Author: Sean Christopherson <[hidden email]>
> Date:   Wed Apr 22 19:25:40 2020 -0700
> Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set
> Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b
>
> It appears that pending requests to the hypervisor can be lost or delayed if
> an immediate exit was requested in vcpu_enter_guest(). As the commit message
> mentions, only the !injected case is affected, so we add a check at the
> cancel_injection label to see if we got there as a result of an immediate exit,
> and then re-issue a KVM_REQ_EVENT request if we are.
>
> The Windows guest is waiting for an event to be processed, which never happens,
> and so gets stuck.
>
> Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my
> testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the
> issue, so SRU to Bionic will not be needed.
>
> [Testcase]
>
> A cascadelake based Xeon server is required. Anything else and the bug will not
> reproduce.
>
> I used a c5.metal server on AWS. It has the following processor:
> Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
>
> Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach
> the desktop. Copy a Windows Server 2k19 image to the destination server, as well
> as a recent ISO image of virtio drivers.
>
> Install virt-manager.
>
> Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb
> ram. Click customise button to change settings before install.
>
> Change the hard disk to be SATA, attach a new cd rom drive for the virtio
> drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX.
>
> Edit the virsh xml, and ensure you set the following features for CPU:
>
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Cascadelake-Server-noTSX</model>
>     <topology sockets='8' cores='1' threads='1'/>
>     <feature policy='require' name='invpcid'/>
>     <feature policy='require' name='pcid'/>
>     <feature policy='require' name='vmx'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='disable' name='mpx'/>
>     <feature policy='require' name='pku'/>
>     <feature policy='require' name='arch-capabilities'/>
>     <feature policy='require' name='rdctl-no'/>
>     <feature policy='require' name='ibrs-all'/>
>     <feature policy='require' name='skip-l1dfl-vmentry'/>
>     <feature policy='require' name='mds-no'/>
>   </cpu>
>
> Those settings are an absolute must.
>
> Boot the VM, and install Windows 2k19 with the desktop environment. Once it is
> installed, open up computer management > device manager and install drivers from
> the virtio ISO for missing hardware, likely the network and balloon devices.
>
> From there, go to server manager, and install the hyper-v role.
>
> Reboot the server. It will reboot a few times, and on the final time, it will
> lock up before it reaches the log in screen.
>
> In virt-manager, go to the performance tab. The CPU will be stuck at 100%.
> The windows guest will be non responsive.
>
> A patched kernel is available in the following ppa:
>
> https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test
>
> If you install this kernel and boot the Windows 2k19 guest, it will come up
> normally when the hyper-v role is enabled, and you will be able to log in.
>
> [Where problems could occur]
>
> This is a change to a core part of the kvm subsystem, so there is potential
> for regression which could affect all users of KVM.
>
> If a regression were to occur, there are no workarounds. Users would need to
> downgrade their kernel while a fix is developed.
>
> Sean Christopherson (1):
>   KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit
>     set
>
>  arch/x86/kvm/x86.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> --
> 2.27.0
>
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team