[azure/bionic][PATCH 0/2] [Hyper-V] IB/mlx5: Respect new UMR capabilities

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[azure/bionic][PATCH 0/2] [Hyper-V] IB/mlx5: Respect new UMR capabilities

Marcelo Henrique Cerri
BugLink: http://bugs.launchpad.net/bugs/1762554

Clean cherry picks from mainline.

Majd Dibbiny (2):
  IB/mlx5: Enable ECN capable bits for UD RoCE v2 QPs
  IB/mlx5: Respect new UMR capabilities

 drivers/infiniband/hw/mlx5/ah.c | 12 +++++++++++
 drivers/infiniband/hw/mlx5/mr.c | 35 ++++++++++++++++++++++++++++-----
 drivers/infiniband/hw/mlx5/qp.c | 21 +++++++++++++++++---
 include/linux/mlx5/mlx5_ifc.h   |  6 +++++-
 4 files changed, 65 insertions(+), 9 deletions(-)

--
2.17.1


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[azure/bionic][PATCH 1/2] IB/mlx5: Enable ECN capable bits for UD RoCE v2 QPs

Marcelo Henrique Cerri
From: Majd Dibbiny <[hidden email]>

BugLink: http://bugs.launchpad.net/bugs/1762554

When working with RC QPs, the FW sets the ECN capable bits for all
the RoCE v2 packets. On the other hand, for UD QPs, the driver needs
to set the the ECN capable bits in the Address Handler since the HW
generates each packet according to the Address Handler and not
the QP context.

If ECN is not enabled in NIC or switch, these bits are ignored.

Fixes: 2811ba51b049 ("IB/mlx5: Add RoCE fields to Address Vector")
Reviewed-by: Mark Bloch <[hidden email]>
Signed-off-by: Majd Dibbiny <[hidden email]>
Signed-off-by: Leon Romanovsky <[hidden email]>
Signed-off-by: Jason Gunthorpe <[hidden email]>
(cherry picked from commit ea8af0d2f2b5b16da4553205ddaf225e0a057e03)
Signed-off-by: Marcelo Henrique Cerri <[hidden email]>
---
 drivers/infiniband/hw/mlx5/ah.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/ah.c b/drivers/infiniband/hw/mlx5/ah.c
index fe269f680103..e6bde32a83f3 100644
--- a/drivers/infiniband/hw/mlx5/ah.c
+++ b/drivers/infiniband/hw/mlx5/ah.c
@@ -36,6 +36,9 @@ static struct ib_ah *create_ib_ah(struct mlx5_ib_dev *dev,
   struct mlx5_ib_ah *ah,
   struct rdma_ah_attr *ah_attr)
 {
+ enum ib_gid_type gid_type;
+ int err;
+
  if (rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH) {
  const struct ib_global_route *grh = rdma_ah_read_grh(ah_attr);
 
@@ -50,6 +53,12 @@ static struct ib_ah *create_ib_ah(struct mlx5_ib_dev *dev,
  ah->av.stat_rate_sl = (rdma_ah_get_static_rate(ah_attr) << 4);
 
  if (ah_attr->type == RDMA_AH_ATTR_TYPE_ROCE) {
+ err = mlx5_get_roce_gid_type(dev, ah_attr->port_num,
+     ah_attr->grh.sgid_index,
+     &gid_type);
+ if (err)
+ return ERR_PTR(err);
+
  memcpy(ah->av.rmac, ah_attr->roce.dmac,
        sizeof(ah_attr->roce.dmac));
  ah->av.udp_sport =
@@ -57,6 +66,9 @@ static struct ib_ah *create_ib_ah(struct mlx5_ib_dev *dev,
  rdma_ah_get_port_num(ah_attr),
  rdma_ah_read_grh(ah_attr)->sgid_index);
  ah->av.stat_rate_sl |= (rdma_ah_get_sl(ah_attr) & 0x7) << 1;
+ if (gid_type == IB_GID_TYPE_ROCE_UDP_ENCAP)
+#define MLX5_ECN_ENABLED BIT(1)
+ ah->av.tclass |= MLX5_ECN_ENABLED;
  } else {
  ah->av.rlid = cpu_to_be16(rdma_ah_get_dlid(ah_attr));
  ah->av.fl_mlid = rdma_ah_get_path_bits(ah_attr) & 0x7f;
--
2.17.1


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[azure/bionic][PATCH 2/2] IB/mlx5: Respect new UMR capabilities

Marcelo Henrique Cerri
In reply to this post by Marcelo Henrique Cerri
From: Majd Dibbiny <[hidden email]>

BugLink: http://bugs.launchpad.net/bugs/1762554

In some firmware configuration, UMR usage from Virtual Functions is restricted.
This information is published to the driver using new capability bits.

Avoid using UMRs in these cases and use the Firmware slow-path flow to create
mkeys and populate them with Virtual to Physical address translation.

Older drivers that do not have this patch, will end up using memory keys that
aren't populated with Virtual to Physical address translation that is done
part of the UMR work.

Reviewed-by: Mark Bloch <[hidden email]>
Signed-off-by: Majd Dibbiny <[hidden email]>
Signed-off-by: Leon Romanovsky <[hidden email]>
Tested-by: Laurence Oberman <[hidden email]>
Signed-off-by: Jason Gunthorpe <[hidden email]>
(cherry picked from commit c8d75a980fab886a9c716567e6b47cc414ad84ee)
Signed-off-by: Marcelo Henrique Cerri <[hidden email]>
---
 drivers/infiniband/hw/mlx5/mr.c | 35 ++++++++++++++++++++++++++++-----
 drivers/infiniband/hw/mlx5/qp.c | 21 +++++++++++++++++---
 include/linux/mlx5/mlx5_ifc.h   |  6 +++++-
 3 files changed, 53 insertions(+), 9 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index c51c602f06d6..170572250391 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -51,6 +51,21 @@ static int clean_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr);
 static int dereg_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr);
 static int mr_cache_max_order(struct mlx5_ib_dev *dev);
 static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr);
+static bool umr_can_modify_entity_size(struct mlx5_ib_dev *dev)
+{
+ return !MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled);
+}
+
+static bool umr_can_use_indirect_mkey(struct mlx5_ib_dev *dev)
+{
+ return !MLX5_CAP_GEN(dev->mdev, umr_indirect_mkey_disabled);
+}
+
+static bool use_umr(struct mlx5_ib_dev *dev, int order)
+{
+ return order <= mr_cache_max_order(dev) &&
+ umr_can_modify_entity_size(dev);
+}
 
 static int destroy_mkey(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr)
 {
@@ -943,7 +958,10 @@ static inline int populate_xlt(struct mlx5_ib_mr *mr, int idx, int npages,
 {
  struct mlx5_ib_dev *dev = mr->dev;
  struct ib_umem *umem = mr->umem;
+
  if (flags & MLX5_IB_UPD_XLT_INDIRECT) {
+ if (!umr_can_use_indirect_mkey(dev))
+ return -EPERM;
  mlx5_odp_populate_klm(xlt, idx, npages, mr, flags);
  return npages;
  }
@@ -990,6 +1008,10 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
  size_t pages_iter = 0;
  gfp_t gfp;
 
+ if ((flags & MLX5_IB_UPD_XLT_INDIRECT) &&
+    !umr_can_use_indirect_mkey(dev))
+ return -EPERM;
+
  /* UMR copies MTTs in units of MLX5_UMR_MTT_ALIGNMENT bytes,
  * so we need to align the offset and length accordingly
  */
@@ -1199,13 +1221,13 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
 {
  struct mlx5_ib_dev *dev = to_mdev(pd->device);
  struct mlx5_ib_mr *mr = NULL;
+ bool populate_mtts = false;
  struct ib_umem *umem;
  int page_shift;
  int npages;
  int ncont;
  int order;
  int err;
- bool use_umr = true;
 
  if (!IS_ENABLED(CONFIG_INFINIBAND_USER_MEM))
  return ERR_PTR(-EINVAL);
@@ -1230,26 +1252,29 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
  if (err < 0)
  return ERR_PTR(err);
 
- if (order <= mr_cache_max_order(dev)) {
+ if (use_umr(dev, order)) {
  mr = alloc_mr_from_cache(pd, umem, virt_addr, length, ncont,
  page_shift, order, access_flags);
  if (PTR_ERR(mr) == -EAGAIN) {
  mlx5_ib_dbg(dev, "cache empty for order %d\n", order);
  mr = NULL;
  }
+ populate_mtts = false;
  } else if (!MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset)) {
  if (access_flags & IB_ACCESS_ON_DEMAND) {
  err = -EINVAL;
  pr_err("Got MR registration for ODP MR > 512MB, not supported for Connect-IB\n");
  goto error;
  }
- use_umr = false;
+ populate_mtts = true;
  }
 
  if (!mr) {
+ if (!umr_can_modify_entity_size(dev))
+ populate_mtts = true;
  mutex_lock(&dev->slow_path_mutex);
  mr = reg_create(NULL, pd, virt_addr, length, umem, ncont,
- page_shift, access_flags, !use_umr);
+ page_shift, access_flags, populate_mtts);
  mutex_unlock(&dev->slow_path_mutex);
  }
 
@@ -1267,7 +1292,7 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
  update_odp_mr(mr);
 #endif
 
- if (use_umr) {
+ if (!populate_mtts) {
  int update_xlt_flags = MLX5_IB_UPD_XLT_ENABLE;
 
  if (access_flags & IB_ACCESS_ON_DEMAND)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 47b39c3e9812..9a3332a3aa54 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -3305,8 +3305,19 @@ static __be64 get_umr_update_pd_mask(void)
  return cpu_to_be64(result);
 }
 
-static void set_reg_umr_segment(struct mlx5_wqe_umr_ctrl_seg *umr,
- struct ib_send_wr *wr, int atomic)
+static int umr_check_mkey_mask(struct mlx5_ib_dev *dev, u64 mask)
+{
+ if ((mask & MLX5_MKEY_MASK_PAGE_SIZE &&
+     MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled)) ||
+    (mask & MLX5_MKEY_MASK_A &&
+     MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled)))
+ return -EPERM;
+ return 0;
+}
+
+static int set_reg_umr_segment(struct mlx5_ib_dev *dev,
+       struct mlx5_wqe_umr_ctrl_seg *umr,
+       struct ib_send_wr *wr, int atomic)
 {
  struct mlx5_umr_wr *umrwr = umr_wr(wr);
 
@@ -3338,6 +3349,8 @@ static void set_reg_umr_segment(struct mlx5_wqe_umr_ctrl_seg *umr,
 
  if (!wr->num_sge)
  umr->flags |= MLX5_UMR_INLINE;
+
+ return umr_check_mkey_mask(dev, be64_to_cpu(umr->mkey_mask));
 }
 
 static u8 get_umr_flags(int acc)
@@ -4160,7 +4173,9 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
  }
  qp->sq.wr_data[idx] = MLX5_IB_WR_UMR;
  ctrl->imm = cpu_to_be32(umr_wr(wr)->mkey);
- set_reg_umr_segment(seg, wr, !!(MLX5_CAP_GEN(mdev, atomic)));
+ err = set_reg_umr_segment(dev, seg, wr, !!(MLX5_CAP_GEN(mdev, atomic)));
+ if (unlikely(err))
+ goto out;
  seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
  size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
  if (unlikely((seg == qend)))
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 1391a82da98e..4bc1a0aea0b5 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -906,7 +906,11 @@ struct mlx5_ifc_cmd_hca_cap_bits {
  u8         reserved_at_202[0x1];
  u8         ipoib_enhanced_offloads[0x1];
  u8         ipoib_basic_offloads[0x1];
- u8         reserved_at_205[0x5];
+ u8         reserved_at_205[0x1];
+ u8         repeated_block_disabled[0x1];
+ u8         umr_modify_entity_size_disabled[0x1];
+ u8         umr_modify_atomic_disabled[0x1];
+ u8         umr_indirect_mkey_disabled[0x1];
  u8         umr_fence[0x2];
  u8         reserved_at_20c[0x3];
  u8         drain_sigerr[0x1];
--
2.17.1


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK/Cmnt: [azure/bionic][PATCH 0/2] [Hyper-V] IB/mlx5: Respect new UMR capabilities

Stefan Bader-2
In reply to this post by Marcelo Henrique Cerri
On 08.06.2018 06:32, Marcelo Henrique Cerri wrote:

> BugLink: http://bugs.launchpad.net/bugs/1762554
>
> Clean cherry picks from mainline.
>
> Majd Dibbiny (2):
>   IB/mlx5: Enable ECN capable bits for UD RoCE v2 QPs
>   IB/mlx5: Respect new UMR capabilities
>
>  drivers/infiniband/hw/mlx5/ah.c | 12 +++++++++++
>  drivers/infiniband/hw/mlx5/mr.c | 35 ++++++++++++++++++++++++++++-----
>  drivers/infiniband/hw/mlx5/qp.c | 21 +++++++++++++++++---
>  include/linux/mlx5/mlx5_ifc.h   |  6 +++++-
>  4 files changed, 65 insertions(+), 9 deletions(-)
>
bionic/azure? I believe I got some linux-azure under bionic but not the other
way round (</nitpick mode=pull_leg>)

Acked-by: Stefan Bader <[hidden email]>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [azure/bionic][PATCH 0/2] [Hyper-V] IB/mlx5: Respect new UMR capabilities

Paolo Pisati-5
In reply to this post by Marcelo Henrique Cerri
On Fri, Jun 8, 2018 at 3:32 PM, Marcelo Henrique Cerri
<[hidden email]> wrote:

> BugLink: http://bugs.launchpad.net/bugs/1762554
>
> Clean cherry picks from mainline.
>
> Majd Dibbiny (2):
>   IB/mlx5: Enable ECN capable bits for UD RoCE v2 QPs
>   IB/mlx5: Respect new UMR capabilities
>
>  drivers/infiniband/hw/mlx5/ah.c | 12 +++++++++++
>  drivers/infiniband/hw/mlx5/mr.c | 35 ++++++++++++++++++++++++++++-----
>  drivers/infiniband/hw/mlx5/qp.c | 21 +++++++++++++++++---
>  include/linux/mlx5/mlx5_ifc.h   |  6 +++++-
>  4 files changed, 65 insertions(+), 9 deletions(-)

Acked-by: Paolo Pisati <[hidden email]>

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

APPLIED: [azure/bionic][PATCH 0/2] [Hyper-V] IB/mlx5: Respect new UMR capabilities

Marcelo Henrique Cerri
In reply to this post by Marcelo Henrique Cerri