[SRU B] [PATCH 0/1] qlcnic: Firmware aborts/hangs in QLogic NIC

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[SRU B] [PATCH 0/1] qlcnic: Firmware aborts/hangs in QLogic NIC

Guilherme Piccoli
BugLink: http://bugs.launchpad.net/bugs/1815033

[Impact]

* In multi-queue configurations for qlcnic driver, there is a corner case
  in which TX queue zero is used at same time for regular data transmission
  by one CPU while another uses the same queue descriptor for MAC config.

* When such "race" indeed happens, it could lead to TX queue zero
  corruption, triggering as net result firmware aborts/hangs out of
  nowhere. The following kernel log messages were collected during the
  corruption event:

  qlcnic 0000:01:00.0: Pause control frames disabled on all ports
  qlcnic 0000:01:00.0: firmware hang detected
  qlcnic 0000:01:00.0: Dumping hw/fw registers
  PEG_HALT_STATUS1: 0x40001502, PEG_HALT_STATUS2: 0x3de7a0,
  PEG_NET_0_PC: 0x6d268, PEG_NET_1_PC: 0x6d2ac,
  PEG_NET_2_PC: 0x149, PEG_NET_3_PC: 0x6e105,
  PEG_NET_4_PC: 0x1e00b
  [...]
  qlcnic 0000:01:00.0: Detected state change from DEV_NEED_RESET, skipping
ack check

* The following device is known to suffer from the issue (lspci output),
  although a whole class of devices (named 82XX series from the vendor)
  are susceptible to this:
  01:00.0 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE
Controller [1077:8020]

* The fix is the following patch, present in mainline kernel as well as
  in supported stable branches:
  c333fa0c4f22 ("qlcnic: fix Tx descriptor corruption on 82xx devices").
  Link for patch in Linus tree: http://git.kernel.org/linus/c333fa0c4f22

[Test Case]

* Unfortunately this is not easy to reproduce; we have a user report of
  the issue with a pretty reliable reproducer - user is running a NFS
  workload on top of the above PCI adapter. His problem goes away with
  the patch proposed here to SRU. His problem happens in both kernels 4.4
  and 4.15, and the patch fixes it for both of them.
  (Notice this is a Bionic-only SRU, since Ubuntu 4.4 kernel got the
  patch from Greg's supported stable branch).

[Regression Potential]

* The patch scope is restricted to a single driver, and the code itself
  is self-contained - basically a restriction to specific tx_ring when
  setting filters. There is potential for regressions in this path for
  the driver which could cause different firmware issues for example,
  but the user testing exhibited great reliability - without the patch
  issue happens after ~6h of machine boot. With the patch the machine
  ran for more than 8 days without issues.

* Also the patch is present in mainline kernel as well as supported
  stable branches, and is already present in Ubuntu 4.4 kernel.

Shahed Shaikh (1):
  qlcnic: fix Tx descriptor corruption on 82xx devices

 drivers/net/ethernet/qlogic/qlcnic/qlcnic.h         |  8 +++++---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h      |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c      | 12 ++++++------
 5 files changed, 17 insertions(+), 12 deletions(-)

--
2.19.2


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[SRU B] [PATCH 1/1] qlcnic: fix Tx descriptor corruption on 82xx devices

Guilherme Piccoli
From: Shahed Shaikh <[hidden email]>

BugLink: http://bugs.launchpad.net/bugs/1815033

In regular NIC transmission flow, driver always configures MAC using
Tx queue zero descriptor as a part of MAC learning flow.
But with multi Tx queue supported NIC, regular transmission can occur on
any non-zero Tx queue and from that context it uses
Tx queue zero descriptor to configure MAC, at the same time TX queue
zero could be used by another CPU for regular transmission
which could lead to Tx queue zero descriptor corruption and cause FW
abort.

This patch fixes this in such a way that driver always configures
learned MAC address from the same Tx queue which is used for
regular transmission.

Fixes: 7e2cf4feba05 ("qlcnic: change driver hardware interface mechanism")
Signed-off-by: Shahed Shaikh <[hidden email]>
Signed-off-by: David S. Miller <[hidden email]>
(cherry picked from commit c333fa0c4f220f8f7ea5acd6b0ebf3bf13fd684d)
Signed-off-by: Guilherme G. Piccoli <[hidden email]>
---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic.h         |  8 +++++---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h      |  3 ++-
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c      | 12 ++++++------
 5 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
index 81312924df14..0c443ea98479 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
@@ -1800,7 +1800,8 @@ struct qlcnic_hardware_ops {
  int (*config_loopback) (struct qlcnic_adapter *, u8);
  int (*clear_loopback) (struct qlcnic_adapter *, u8);
  int (*config_promisc_mode) (struct qlcnic_adapter *, u32);
- void (*change_l2_filter) (struct qlcnic_adapter *, u64 *, u16);
+ void (*change_l2_filter)(struct qlcnic_adapter *adapter, u64 *addr,
+ u16 vlan, struct qlcnic_host_tx_ring *tx_ring);
  int (*get_board_info) (struct qlcnic_adapter *);
  void (*set_mac_filter_count) (struct qlcnic_adapter *);
  void (*free_mac_list) (struct qlcnic_adapter *);
@@ -2064,9 +2065,10 @@ static inline int qlcnic_nic_set_promisc(struct qlcnic_adapter *adapter,
 }
 
 static inline void qlcnic_change_filter(struct qlcnic_adapter *adapter,
- u64 *addr, u16 id)
+ u64 *addr, u16 vlan,
+ struct qlcnic_host_tx_ring *tx_ring)
 {
- adapter->ahw->hw_ops->change_l2_filter(adapter, addr, id);
+ adapter->ahw->hw_ops->change_l2_filter(adapter, addr, vlan, tx_ring);
 }
 
 static inline int qlcnic_get_board_info(struct qlcnic_adapter *adapter)
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
index 46b0372dd032..1fc84d8f891b 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
@@ -2134,7 +2134,8 @@ int qlcnic_83xx_sre_macaddr_change(struct qlcnic_adapter *adapter, u8 *addr,
 }
 
 void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
-  u16 vlan_id)
+  u16 vlan_id,
+  struct qlcnic_host_tx_ring *tx_ring)
 {
  u8 mac[ETH_ALEN];
  memcpy(&mac, addr, ETH_ALEN);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
index b75a81246856..73fe2f64491d 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
@@ -550,7 +550,8 @@ int qlcnic_83xx_wrt_reg_indirect(struct qlcnic_adapter *, ulong, u32);
 int qlcnic_83xx_nic_set_promisc(struct qlcnic_adapter *, u32);
 int qlcnic_83xx_config_hw_lro(struct qlcnic_adapter *, int);
 int qlcnic_83xx_config_rss(struct qlcnic_adapter *, int);
-void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *, u64 *, u16);
+void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
+  u16 vlan, struct qlcnic_host_tx_ring *ring);
 int qlcnic_83xx_get_pci_info(struct qlcnic_adapter *, struct qlcnic_pci_info *);
 int qlcnic_83xx_set_nic_info(struct qlcnic_adapter *, struct qlcnic_info *);
 void qlcnic_83xx_initialize_nic(struct qlcnic_adapter *, int);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
index 4bb33af8e2b3..56a3bd9e37dc 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
@@ -173,7 +173,8 @@ int qlcnic_82xx_napi_add(struct qlcnic_adapter *adapter,
  struct net_device *netdev);
 void qlcnic_82xx_get_beacon_state(struct qlcnic_adapter *);
 void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter,
-       u64 *uaddr, u16 vlan_id);
+       u64 *uaddr, u16 vlan_id,
+       struct qlcnic_host_tx_ring *tx_ring);
 int qlcnic_82xx_config_intr_coalesce(struct qlcnic_adapter *,
      struct ethtool_coalesce *);
 int qlcnic_82xx_set_rx_coalesce(struct qlcnic_adapter *);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
index 84dd83031a1b..9647578cbe6a 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
@@ -268,13 +268,12 @@ static void qlcnic_add_lb_filter(struct qlcnic_adapter *adapter,
 }
 
 void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
-       u16 vlan_id)
+       u16 vlan_id, struct qlcnic_host_tx_ring *tx_ring)
 {
  struct cmd_desc_type0 *hwdesc;
  struct qlcnic_nic_req *req;
  struct qlcnic_mac_req *mac_req;
  struct qlcnic_vlan_req *vlan_req;
- struct qlcnic_host_tx_ring *tx_ring = adapter->tx_ring;
  u32 producer;
  u64 word;
 
@@ -301,7 +300,8 @@ void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
 
 static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
        struct cmd_desc_type0 *first_desc,
-       struct sk_buff *skb)
+       struct sk_buff *skb,
+       struct qlcnic_host_tx_ring *tx_ring)
 {
  struct vlan_ethhdr *vh = (struct vlan_ethhdr *)(skb->data);
  struct ethhdr *phdr = (struct ethhdr *)(skb->data);
@@ -335,7 +335,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
     tmp_fil->vlan_id == vlan_id) {
  if (jiffies > (QLCNIC_READD_AGE * HZ + tmp_fil->ftime))
  qlcnic_change_filter(adapter, &src_addr,
-     vlan_id);
+     vlan_id, tx_ring);
  tmp_fil->ftime = jiffies;
  return;
  }
@@ -350,7 +350,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
  if (!fil)
  return;
 
- qlcnic_change_filter(adapter, &src_addr, vlan_id);
+ qlcnic_change_filter(adapter, &src_addr, vlan_id, tx_ring);
  fil->ftime = jiffies;
  fil->vlan_id = vlan_id;
  memcpy(fil->faddr, &src_addr, ETH_ALEN);
@@ -766,7 +766,7 @@ netdev_tx_t qlcnic_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
  }
 
  if (adapter->drv_mac_learn)
- qlcnic_send_filter(adapter, first_desc, skb);
+ qlcnic_send_filter(adapter, first_desc, skb, tx_ring);
 
  tx_ring->tx_stats.tx_bytes += skb->len;
  tx_ring->tx_stats.xmit_called++;
--
2.19.2


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK: [SRU B] [PATCH 1/1] qlcnic: fix Tx descriptor corruption on 82xx devices

Stefan Bader-2
On 07.02.19 13:44, Guilherme G. Piccoli wrote:

> From: Shahed Shaikh <[hidden email]>
>
> BugLink: http://bugs.launchpad.net/bugs/1815033
>
> In regular NIC transmission flow, driver always configures MAC using
> Tx queue zero descriptor as a part of MAC learning flow.
> But with multi Tx queue supported NIC, regular transmission can occur on
> any non-zero Tx queue and from that context it uses
> Tx queue zero descriptor to configure MAC, at the same time TX queue
> zero could be used by another CPU for regular transmission
> which could lead to Tx queue zero descriptor corruption and cause FW
> abort.
>
> This patch fixes this in such a way that driver always configures
> learned MAC address from the same Tx queue which is used for
> regular transmission.
>
> Fixes: 7e2cf4feba05 ("qlcnic: change driver hardware interface mechanism")
> Signed-off-by: Shahed Shaikh <[hidden email]>
> Signed-off-by: David S. Miller <[hidden email]>
> (cherry picked from commit c333fa0c4f220f8f7ea5acd6b0ebf3bf13fd684d)
> Signed-off-by: Guilherme G. Piccoli <[hidden email]>
Acked-by: Stefan Bader <[hidden email]>
> ---

In Xenial and Cosmic already.

>  drivers/net/ethernet/qlogic/qlcnic/qlcnic.h         |  8 +++++---
>  drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c |  3 ++-
>  drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h |  3 ++-
>  drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h      |  3 ++-
>  drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c      | 12 ++++++------
>  5 files changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
> index 81312924df14..0c443ea98479 100644
> --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
> +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
> @@ -1800,7 +1800,8 @@ struct qlcnic_hardware_ops {
>   int (*config_loopback) (struct qlcnic_adapter *, u8);
>   int (*clear_loopback) (struct qlcnic_adapter *, u8);
>   int (*config_promisc_mode) (struct qlcnic_adapter *, u32);
> - void (*change_l2_filter) (struct qlcnic_adapter *, u64 *, u16);
> + void (*change_l2_filter)(struct qlcnic_adapter *adapter, u64 *addr,
> + u16 vlan, struct qlcnic_host_tx_ring *tx_ring);
>   int (*get_board_info) (struct qlcnic_adapter *);
>   void (*set_mac_filter_count) (struct qlcnic_adapter *);
>   void (*free_mac_list) (struct qlcnic_adapter *);
> @@ -2064,9 +2065,10 @@ static inline int qlcnic_nic_set_promisc(struct qlcnic_adapter *adapter,
>  }
>  
>  static inline void qlcnic_change_filter(struct qlcnic_adapter *adapter,
> - u64 *addr, u16 id)
> + u64 *addr, u16 vlan,
> + struct qlcnic_host_tx_ring *tx_ring)
>  {
> - adapter->ahw->hw_ops->change_l2_filter(adapter, addr, id);
> + adapter->ahw->hw_ops->change_l2_filter(adapter, addr, vlan, tx_ring);
>  }
>  
>  static inline int qlcnic_get_board_info(struct qlcnic_adapter *adapter)
> diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
> index 46b0372dd032..1fc84d8f891b 100644
> --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
> +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
> @@ -2134,7 +2134,8 @@ int qlcnic_83xx_sre_macaddr_change(struct qlcnic_adapter *adapter, u8 *addr,
>  }
>  
>  void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
> -  u16 vlan_id)
> +  u16 vlan_id,
> +  struct qlcnic_host_tx_ring *tx_ring)
>  {
>   u8 mac[ETH_ALEN];
>   memcpy(&mac, addr, ETH_ALEN);
> diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
> index b75a81246856..73fe2f64491d 100644
> --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
> +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
> @@ -550,7 +550,8 @@ int qlcnic_83xx_wrt_reg_indirect(struct qlcnic_adapter *, ulong, u32);
>  int qlcnic_83xx_nic_set_promisc(struct qlcnic_adapter *, u32);
>  int qlcnic_83xx_config_hw_lro(struct qlcnic_adapter *, int);
>  int qlcnic_83xx_config_rss(struct qlcnic_adapter *, int);
> -void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *, u64 *, u16);
> +void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
> +  u16 vlan, struct qlcnic_host_tx_ring *ring);
>  int qlcnic_83xx_get_pci_info(struct qlcnic_adapter *, struct qlcnic_pci_info *);
>  int qlcnic_83xx_set_nic_info(struct qlcnic_adapter *, struct qlcnic_info *);
>  void qlcnic_83xx_initialize_nic(struct qlcnic_adapter *, int);
> diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
> index 4bb33af8e2b3..56a3bd9e37dc 100644
> --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
> +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
> @@ -173,7 +173,8 @@ int qlcnic_82xx_napi_add(struct qlcnic_adapter *adapter,
>   struct net_device *netdev);
>  void qlcnic_82xx_get_beacon_state(struct qlcnic_adapter *);
>  void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter,
> -       u64 *uaddr, u16 vlan_id);
> +       u64 *uaddr, u16 vlan_id,
> +       struct qlcnic_host_tx_ring *tx_ring);
>  int qlcnic_82xx_config_intr_coalesce(struct qlcnic_adapter *,
>       struct ethtool_coalesce *);
>  int qlcnic_82xx_set_rx_coalesce(struct qlcnic_adapter *);
> diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
> index 84dd83031a1b..9647578cbe6a 100644
> --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
> +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
> @@ -268,13 +268,12 @@ static void qlcnic_add_lb_filter(struct qlcnic_adapter *adapter,
>  }
>  
>  void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
> -       u16 vlan_id)
> +       u16 vlan_id, struct qlcnic_host_tx_ring *tx_ring)
>  {
>   struct cmd_desc_type0 *hwdesc;
>   struct qlcnic_nic_req *req;
>   struct qlcnic_mac_req *mac_req;
>   struct qlcnic_vlan_req *vlan_req;
> - struct qlcnic_host_tx_ring *tx_ring = adapter->tx_ring;
>   u32 producer;
>   u64 word;
>  
> @@ -301,7 +300,8 @@ void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
>  
>  static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
>         struct cmd_desc_type0 *first_desc,
> -       struct sk_buff *skb)
> +       struct sk_buff *skb,
> +       struct qlcnic_host_tx_ring *tx_ring)
>  {
>   struct vlan_ethhdr *vh = (struct vlan_ethhdr *)(skb->data);
>   struct ethhdr *phdr = (struct ethhdr *)(skb->data);
> @@ -335,7 +335,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
>      tmp_fil->vlan_id == vlan_id) {
>   if (jiffies > (QLCNIC_READD_AGE * HZ + tmp_fil->ftime))
>   qlcnic_change_filter(adapter, &src_addr,
> -     vlan_id);
> +     vlan_id, tx_ring);
>   tmp_fil->ftime = jiffies;
>   return;
>   }
> @@ -350,7 +350,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
>   if (!fil)
>   return;
>  
> - qlcnic_change_filter(adapter, &src_addr, vlan_id);
> + qlcnic_change_filter(adapter, &src_addr, vlan_id, tx_ring);
>   fil->ftime = jiffies;
>   fil->vlan_id = vlan_id;
>   memcpy(fil->faddr, &src_addr, ETH_ALEN);
> @@ -766,7 +766,7 @@ netdev_tx_t qlcnic_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
>   }
>  
>   if (adapter->drv_mac_learn)
> - qlcnic_send_filter(adapter, first_desc, skb);
> + qlcnic_send_filter(adapter, first_desc, skb, tx_ring);
>  
>   tx_ring->tx_stats.tx_bytes += skb->len;
>   tx_ring->tx_stats.xmit_called++;
>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

ACK: [SRU B] [PATCH 0/1] qlcnic: Firmware aborts/hangs in QLogic NIC

Marcelo Henrique Cerri
In reply to this post by Guilherme Piccoli
Acked-by: Marcelo Henrique Cerri <[hidden email]>

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (499 bytes) Download Attachment