[SRU][Xenial][PATCH 0/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[SRU][Xenial][PATCH 0/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Joseph Salisbury-3
BugLink: http://bugs.launchpad.net/bugs/1775235

== SRU Justification ==
The bug reporter noticed that Xenial guests running on Nutanix AHV stopped
booting after they were upgraded to 4.4.0-127. Only guests with scsi mq
enabled suffered from this problem. AHV is one of the few hypervisor
products to offer multiqueue for virtio-scsi devices.

Upon further investigation, the saw that the kernel would hang during the
scanning of scsi targets. More specifically, immediately after coming
across a target without any luns present.

It was found the following commit introduced this regression:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh <[hidden email]>
Date: Thu Apr 19 21:40:00 2018 +0200

The patch spins on the target's 'reqs' counter waiting for the target to quiesce.

Further study revealed that virtio-scsi itself is broken in a way that it
doesn't increment the 'reqs' counter when submitting requests on MQ in
certain conditions. That caused the counter to go to -1 (on the completion
of the first request) and the CPU to hang indefinitely.

This regression is fixed by the requested SAUCE patch.


== Fix ==
UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

== Regression Potential ==
Low.  Limited to virtio and fixes a regression.

== Test Case ==
A test kernel was built with this patch and tested by the original bug reporter.
The bug reporter states the test kernel resolved the bug.

Felipe Franciosi (1):
  UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

 drivers/scsi/virtio_scsi.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[SRU][Xenial][PATCH 1/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Joseph Salisbury-3
From: Felipe Franciosi <[hidden email]>

BugLink: http://bugs.launchpad.net/bugs/1775235

virtio-scsi itself is broken in a way that it doesn't increment the
'reqs' counter when submitting requests on MQ in certain conditions.
That caused the counter to go to -1 (on the completion of the first
request) and the CPU to hang indefinitely.

Fixes: f1f609d8015e ("UBUNTU: SAUCE: (no-up) virtio-scsi: Fix race in
target free")

Signed-off-by: Felipe Franciosi <[hidden email]>
Signed-off-by: Joseph Salisbury <[hidden email]>
---
 drivers/scsi/virtio_scsi.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
index e2da312..b4a41d5 100644
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -588,11 +588,12 @@ static int virtscsi_queuecommand_single(struct Scsi_Host *sh,
 }
 
 static struct virtio_scsi_vq *virtscsi_pick_vq_mq(struct virtio_scsi *vscsi,
-  struct scsi_cmnd *sc)
+  struct virtio_scsi_target_state *tgt, struct scsi_cmnd *sc)
 {
  u32 tag = blk_mq_unique_tag(sc->request);
  u16 hwq = blk_mq_unique_tag_to_hwq(tag);
 
+ atomic_inc(&tgt->reqs);
  return &vscsi->req_vqs[hwq];
 }
 
@@ -642,7 +643,7 @@ static int virtscsi_queuecommand_multi(struct Scsi_Host *sh,
  struct virtio_scsi_vq *req_vq;
 
  if (shost_use_blk_mq(sh))
- req_vq = virtscsi_pick_vq_mq(vscsi, sc);
+ req_vq = virtscsi_pick_vq_mq(vscsi, tgt, sc);
  else
  req_vq = virtscsi_pick_vq(vscsi, tgt);
 
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK/Cmnt: [SRU][Xenial][PATCH 1/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Stefan Bader-2
On 08.06.2018 10:48, Joseph Salisbury wrote:

> From: Felipe Franciosi <[hidden email]>
>
> BugLink: http://bugs.launchpad.net/bugs/1775235
>
> virtio-scsi itself is broken in a way that it doesn't increment the
> 'reqs' counter when submitting requests on MQ in certain conditions.
> That caused the counter to go to -1 (on the completion of the first
> request) and the CPU to hang indefinitely.
>
> Fixes: f1f609d8015e ("UBUNTU: SAUCE: (no-up) virtio-scsi: Fix race in
> target free")
>
> Signed-off-by: Felipe Franciosi <[hidden email]>
> Signed-off-by: Joseph Salisbury <[hidden email]>
Acked-by: Stefan Bader <[hidden email]>
> ---

Ok, kind of ugly to have to fix that work-around hack which is already ugly on
its own. But it is what it is...

-Stefan

>  drivers/scsi/virtio_scsi.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
> index e2da312..b4a41d5 100644
> --- a/drivers/scsi/virtio_scsi.c
> +++ b/drivers/scsi/virtio_scsi.c
> @@ -588,11 +588,12 @@ static int virtscsi_queuecommand_single(struct Scsi_Host *sh,
>  }
>  
>  static struct virtio_scsi_vq *virtscsi_pick_vq_mq(struct virtio_scsi *vscsi,
> -  struct scsi_cmnd *sc)
> +  struct virtio_scsi_target_state *tgt, struct scsi_cmnd *sc)
>  {
>   u32 tag = blk_mq_unique_tag(sc->request);
>   u16 hwq = blk_mq_unique_tag_to_hwq(tag);
>  
> + atomic_inc(&tgt->reqs);
>   return &vscsi->req_vqs[hwq];
>  }
>  
> @@ -642,7 +643,7 @@ static int virtscsi_queuecommand_multi(struct Scsi_Host *sh,
>   struct virtio_scsi_vq *req_vq;
>  
>   if (shost_use_blk_mq(sh))
> - req_vq = virtscsi_pick_vq_mq(vscsi, sc);
> + req_vq = virtscsi_pick_vq_mq(vscsi, tgt, sc);
>   else
>   req_vq = virtscsi_pick_vq(vscsi, tgt);
>  
>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

ACK: [SRU][Xenial][PATCH 1/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Khaled Elmously
In reply to this post by Joseph Salisbury-3
On 2018-06-08 13:48:27 , Joseph Salisbury wrote:

> From: Felipe Franciosi <[hidden email]>
>
> BugLink: http://bugs.launchpad.net/bugs/1775235
>
> virtio-scsi itself is broken in a way that it doesn't increment the
> 'reqs' counter when submitting requests on MQ in certain conditions.
> That caused the counter to go to -1 (on the completion of the first
> request) and the CPU to hang indefinitely.
>
> Fixes: f1f609d8015e ("UBUNTU: SAUCE: (no-up) virtio-scsi: Fix race in
> target free")
>
> Signed-off-by: Felipe Franciosi <[hidden email]>
> Signed-off-by: Joseph Salisbury <[hidden email]>
> ---
>  drivers/scsi/virtio_scsi.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
> index e2da312..b4a41d5 100644
> --- a/drivers/scsi/virtio_scsi.c
> +++ b/drivers/scsi/virtio_scsi.c
> @@ -588,11 +588,12 @@ static int virtscsi_queuecommand_single(struct Scsi_Host *sh,
>  }
>  
>  static struct virtio_scsi_vq *virtscsi_pick_vq_mq(struct virtio_scsi *vscsi,
> -  struct scsi_cmnd *sc)
> +  struct virtio_scsi_target_state *tgt, struct scsi_cmnd *sc)
>  {
>   u32 tag = blk_mq_unique_tag(sc->request);
>   u16 hwq = blk_mq_unique_tag_to_hwq(tag);
>  
> + atomic_inc(&tgt->reqs);
>   return &vscsi->req_vqs[hwq];
>  }
>  
> @@ -642,7 +643,7 @@ static int virtscsi_queuecommand_multi(struct Scsi_Host *sh,
>   struct virtio_scsi_vq *req_vq;
>  
>   if (shost_use_blk_mq(sh))
> - req_vq = virtscsi_pick_vq_mq(vscsi, sc);
> + req_vq = virtscsi_pick_vq_mq(vscsi, tgt, sc);
>   else
>   req_vq = virtscsi_pick_vq(vscsi, tgt);
>

Acked-by: Khalid Elmously <[hidden email]>
 

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

APPLIED: [SRU][Xenial][PATCH 0/1] UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.

Khaled Elmously
In reply to this post by Joseph Salisbury-3
Applied to Xenial

On 2018-06-08 13:48:26 , Joseph Salisbury wrote:

> BugLink: http://bugs.launchpad.net/bugs/1775235
>
> == SRU Justification ==
> The bug reporter noticed that Xenial guests running on Nutanix AHV stopped
> booting after they were upgraded to 4.4.0-127. Only guests with scsi mq
> enabled suffered from this problem. AHV is one of the few hypervisor
> products to offer multiqueue for virtio-scsi devices.
>
> Upon further investigation, the saw that the kernel would hang during the
> scanning of scsi targets. More specifically, immediately after coming
> across a target without any luns present.
>
> It was found the following commit introduced this regression:
> commit f1f609d8015e1d34d39458924dcd9524fccd4307
> Author: Jay Vosburgh <[hidden email]>
> Date: Thu Apr 19 21:40:00 2018 +0200
>
> The patch spins on the target's 'reqs' counter waiting for the target to quiesce.
>
> Further study revealed that virtio-scsi itself is broken in a way that it
> doesn't increment the 'reqs' counter when submitting requests on MQ in
> certain conditions. That caused the counter to go to -1 (on the completion
> of the first request) and the CPU to hang indefinitely.
>
> This regression is fixed by the requested SAUCE patch.
>
>
> == Fix ==
> UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.
>
> == Regression Potential ==
> Low.  Limited to virtio and fixes a regression.
>
> == Test Case ==
> A test kernel was built with this patch and tested by the original bug reporter.
> The bug reporter states the test kernel resolved the bug.
>
> Felipe Franciosi (1):
>   UBUNTU: SAUCE: (no-up) virtio-scsi: Increment reqs counter.
>
>  drivers/scsi/virtio_scsi.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> --
> 2.7.4
>
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team