[Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

Po-Hsu Lin (Sam)
BugLink: https://bugs.launchpad.net/bugs/1812352

== Justification ==
In the ubuntu_kernel_selftests, the skipped test will uses return code 4
(KSFT_SKIP).

However the code to handle this non-zero return code was not implemented
in the kselftest framework. And this will generate some false-positive
results as those skipped tests were treated as a failure.

For example the raw_skew test in timers on 4.15:
    1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons
    not ok 1..7 selftests: raw_skew [FAIL]

And the test_user_copy.sh test in user (failure could be found in KVM kernels):
    user: module test_user_copy is not found [SKIP]
    not ok 1..1 selftests: test_user_copy.sh [FAIL]

Xenial is affected by this issue as well. Considering the fact that this
issue in Bionic affects more tests / kernel variants, and it's way
easier to fix this in Bionic, I decided to go for Bionic first.

== Fix ==
771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
a3355440 (selftests: Fix lib.mk run_tests target shell script)
42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)

All these can be cherry-picked into Bionic.

The crucial patch is commit 3f4435b5, the other three patches are just
some minor fixes that pave the road for it to be cherry-picked.

== Test ==
Tested on a Bionic amd64 node, the result shows positive.

A test summary before the patch:
END GOOD ubuntu_kernel_selftests.setup
END GOOD ubuntu_kernel_selftests.breakpoints
END GOOD ubuntu_kernel_selftests.cpu-hotplug
END GOOD ubuntu_kernel_selftests.efivarfs
END GOOD ubuntu_kernel_selftests.memfd
END GOOD ubuntu_kernel_selftests.memory-hotplug
END GOOD ubuntu_kernel_selftests.mount
END GOOD ubuntu_kernel_selftests.net
END GOOD ubuntu_kernel_selftests.ptrace
END GOOD ubuntu_kernel_selftests.seccomp
END ERROR ubuntu_kernel_selftests.timers
END GOOD ubuntu_kernel_selftests.powerpc
END GOOD ubuntu_kernel_selftests.user
END GOOD ubuntu_kernel_selftests.ftrace

A test summary after the patch:
END GOOD ubuntu_kernel_selftests.setup
END GOOD ubuntu_kernel_selftests.breakpoints
END GOOD ubuntu_kernel_selftests.cpu-hotplug
END GOOD ubuntu_kernel_selftests.efivarfs
END GOOD ubuntu_kernel_selftests.memfd
END GOOD ubuntu_kernel_selftests.memory-hotplug
END GOOD ubuntu_kernel_selftests.mount
END GOOD ubuntu_kernel_selftests.net
END GOOD ubuntu_kernel_selftests.ptrace
END GOOD ubuntu_kernel_selftests.seccomp
END GOOD ubuntu_kernel_selftests.timers
END GOOD ubuntu_kernel_selftests.powerpc
END GOOD ubuntu_kernel_selftests.user
END GOOD ubuntu_kernel_selftests.ftrac

No tests were skipped unintentionally. And the raw_skew test in timers
is now marked as [SKIPPED] and no longer causing failure.

A complete test report could be found in the bug report attachment.

== Regression Potential ==
Low, this patch set just added support to handle the skip return code
and changes are limited to the kselftest framework itself, no actual
impact to normal users.

Mathieu Desnoyers (1):
  selftests: Fix lib.mk run_tests target shell script

Shuah Khan (1):
  selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers

Shuah Khan (Samsung OSG) (2):
  selftests: lib.mk: cleanup RUN_TESTS define and make it readable
  selftests: lib.mk: add SKIP handling to RUN_TESTS define

 tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Bionic][SRU][PATCH 1/4] selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers

Po-Hsu Lin (Sam)
From: Shuah Khan <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1812352

Set KSFT_TAP_LEVEL before running tests to prevent nested TAP header
printing from tests.

Signed-off-by: Shuah Khan <[hidden email]>
(cherry picked from commit 771cbc3bcbb59084766a501772853f2de7009534)
Signed-off-by: Po-Hsu Lin <[hidden email]>
---
 tools/testing/selftests/lib.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 6cd4695..51bc464 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -32,6 +32,7 @@ endif
 
 .ONESHELL:
 define RUN_TESTS
+ @export KSFT_TAP_LEVEL=`echo 1`;
  @test_num=`echo 0`;
  @echo "TAP version 13";
  @for TEST in $(1); do \
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Bionic][SRU][PATCH 2/4] selftests: Fix lib.mk run_tests target shell script

Po-Hsu Lin (Sam)
In reply to this post by Po-Hsu Lin (Sam)
From: Mathieu Desnoyers <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1812352

Within run_tests target, the whole script needs to be executed within
the same shell and not as separate subshells, so the initial test_num
variable set to 0 is still present when executing "test_num=`echo
$$test_num+1 | bc`;".

Demonstration of the issue (make run_tests):

TAP version 13
(standard_in) 1: syntax error
selftests: basic_test
========================================
ok 1.. selftests: basic_test [PASS]
(standard_in) 1: syntax error
selftests: basic_percpu_ops_test
========================================
ok 1.. selftests: basic_percpu_ops_test [PASS]
(standard_in) 1: syntax error
selftests: param_test
========================================
ok 1.. selftests: param_test [PASS]

With fix applied:

TAP version 13
selftests: basic_test
========================================
ok 1..1 selftests: basic_test [PASS]
selftests: basic_percpu_ops_test
========================================
ok 1..2 selftests: basic_percpu_ops_test [PASS]
selftests: param_test
========================================
ok 1..3 selftests: param_test [PASS]

Signed-off-by: Mathieu Desnoyers <[hidden email]>
Fixes: 1f87c7c15d7 ("selftests: lib.mk: change RUN_TESTS to print messages in TAP13 format")
CC: Shuah Khan <[hidden email]>
CC: [hidden email]
Signed-off-by: Shuah Khan (Samsung OSG) <[hidden email]>
(cherry picked from commit a33554401e4746cc33307910a1baad63ce3fd650)
Signed-off-by: Po-Hsu Lin <[hidden email]>
---
 tools/testing/selftests/lib.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 51bc464..a001ccc 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -32,10 +32,10 @@ endif
 
 .ONESHELL:
 define RUN_TESTS
- @export KSFT_TAP_LEVEL=`echo 1`;
- @test_num=`echo 0`;
- @echo "TAP version 13";
- @for TEST in $(1); do \
+ @export KSFT_TAP_LEVEL=`echo 1`; \
+ test_num=`echo 0`; \
+ echo "TAP version 13"; \
+ for TEST in $(1); do \
  BASENAME_TEST=`basename $$TEST`; \
  test_num=`echo $$test_num+1 | bc`; \
  echo "selftests: $$BASENAME_TEST"; \
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Bionic][SRU][PATCH 3/4] selftests: lib.mk: cleanup RUN_TESTS define and make it readable

Po-Hsu Lin (Sam)
In reply to this post by Po-Hsu Lin (Sam)
From: "Shuah Khan (Samsung OSG)" <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1812352

Refine RUN_TESTS define's output block for summary and non-summary code
to remove duplicate code and make it readable.

cd `dirname $$TEST` > /dev/null; and cd - > /dev/null; are moved
to common code block and indentation fixed.

Signed-off-by: Shuah Khan (Samsung OSG) <[hidden email]>
(cherry picked from commit 42b44c34136857ccdf90ebb1cbc38f2bf0aec7a1)
Signed-off-by: Po-Hsu Lin <[hidden email]>
---
 tools/testing/selftests/lib.mk | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index a001ccc..0b76e4a 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -44,11 +44,13 @@ define RUN_TESTS
  echo "selftests: Warning: file $$BASENAME_TEST is not executable, correct this.";\
  echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [FAIL]"; \
  else \
- if [ "X$(summary)" != "X" ]; then \
- cd `dirname $$TEST` > /dev/null; (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; cd - > /dev/null;\
+ cd `dirname $$TEST` > /dev/null; \
+ if [ "X$(summary)" != "X" ]; then \
+ (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
  else \
- cd `dirname $$TEST` > /dev/null; (./$$BASENAME_TEST && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; cd - > /dev/null;\
+ (./$$BASENAME_TEST && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
  fi; \
+ cd - > /dev/null; \
  fi; \
  done;
 endef
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Bionic][SRU][PATCH 4/4] selftests: lib.mk: add SKIP handling to RUN_TESTS define

Po-Hsu Lin (Sam)
In reply to this post by Po-Hsu Lin (Sam)
From: "Shuah Khan (Samsung OSG)" <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1812352

RUN_TESTS which is the common function that implements run_tests target,
treats all non-zero return codes from tests as failures. When tests are
skipped with non-zero return code, because of unmet dependencies and/or
unsupported configuration, it reports them as failed. This will lead to
too many false negatives even on the tests that couldn't be run.

RUN_TESTS is changed to test for SKIP=4 return from tests to enable the
framework for individual tests to return special SKIP code.

Tests will be changed as needed to report SKIP instead FAIL/PASS when
they get skipped.

Signed-off-by: Shuah Khan (Samsung OSG) <[hidden email]>
(cherry picked from commit 3f4435b5149372b3bbc5acab5c835d490490d6bc)
Signed-off-by: Po-Hsu Lin <[hidden email]>
---
 tools/testing/selftests/lib.mk | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 0b76e4a..02273bf 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -34,6 +34,7 @@ endif
 define RUN_TESTS
  @export KSFT_TAP_LEVEL=`echo 1`; \
  test_num=`echo 0`; \
+ skip=`echo 4`; \
  echo "TAP version 13"; \
  for TEST in $(1); do \
  BASENAME_TEST=`basename $$TEST`; \
@@ -46,9 +47,19 @@ define RUN_TESTS
  else \
  cd `dirname $$TEST` > /dev/null; \
  if [ "X$(summary)" != "X" ]; then \
- (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
+ (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \
+ echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || \
+ (if [ $$? -eq $$skip ]; then \
+ echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [SKIP]"; \
+ else echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
+ fi;) \
  else \
- (./$$BASENAME_TEST && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
+ (./$$BASENAME_TEST && \
+ echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || \
+ (if [ $$? -eq $$skip ]; then \
+ echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [SKIP]"; \
+ else echo "not ok 1..$$test_num selftests:  $$BASENAME_TEST [FAIL]"; \
+ fi;) \
  fi; \
  cd - > /dev/null; \
  fi; \
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK: [Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

Stefan Bader-2
In reply to this post by Po-Hsu Lin (Sam)
On 06.11.19 11:29, Po-Hsu Lin wrote:

> BugLink: https://bugs.launchpad.net/bugs/1812352
>
> == Justification ==
> In the ubuntu_kernel_selftests, the skipped test will uses return code 4
> (KSFT_SKIP).
>
> However the code to handle this non-zero return code was not implemented
> in the kselftest framework. And this will generate some false-positive
> results as those skipped tests were treated as a failure.
>
> For example the raw_skew test in timers on 4.15:
>     1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons
>     not ok 1..7 selftests: raw_skew [FAIL]
>
> And the test_user_copy.sh test in user (failure could be found in KVM kernels):
>     user: module test_user_copy is not found [SKIP]
>     not ok 1..1 selftests: test_user_copy.sh [FAIL]
>
> Xenial is affected by this issue as well. Considering the fact that this
> issue in Bionic affects more tests / kernel variants, and it's way
> easier to fix this in Bionic, I decided to go for Bionic first.
>
> == Fix ==
> 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
> a3355440 (selftests: Fix lib.mk run_tests target shell script)
> 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
> 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)
>
> All these can be cherry-picked into Bionic.
>
> The crucial patch is commit 3f4435b5, the other three patches are just
> some minor fixes that pave the road for it to be cherry-picked.
>
> == Test ==
> Tested on a Bionic amd64 node, the result shows positive.
>
> A test summary before the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END ERROR ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrace
>
> A test summary after the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END GOOD ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrac
>
> No tests were skipped unintentionally. And the raw_skew test in timers
> is now marked as [SKIPPED] and no longer causing failure.
>
> A complete test report could be found in the bug report attachment.
>
> == Regression Potential ==
> Low, this patch set just added support to handle the skip return code
> and changes are limited to the kselftest framework itself, no actual
> impact to normal users.
>
> Mathieu Desnoyers (1):
>   selftests: Fix lib.mk run_tests target shell script
>
> Shuah Khan (1):
>   selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers
>
> Shuah Khan (Samsung OSG) (2):
>   selftests: lib.mk: cleanup RUN_TESTS define and make it readable
>   selftests: lib.mk: add SKIP handling to RUN_TESTS define
>
>  tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------
>  1 file changed, 20 insertions(+), 6 deletions(-)
>
Acked-by: Stefan Bader <[hidden email]>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

ACK: [Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

Andrea Righi
In reply to this post by Po-Hsu Lin (Sam)
On Wed, Nov 06, 2019 at 06:29:02PM +0800, Po-Hsu Lin wrote:

> BugLink: https://bugs.launchpad.net/bugs/1812352
>
> == Justification ==
> In the ubuntu_kernel_selftests, the skipped test will uses return code 4
> (KSFT_SKIP).
>
> However the code to handle this non-zero return code was not implemented
> in the kselftest framework. And this will generate some false-positive
> results as those skipped tests were treated as a failure.
>
> For example the raw_skew test in timers on 4.15:
>     1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons
>     not ok 1..7 selftests: raw_skew [FAIL]
>
> And the test_user_copy.sh test in user (failure could be found in KVM kernels):
>     user: module test_user_copy is not found [SKIP]
>     not ok 1..1 selftests: test_user_copy.sh [FAIL]
>
> Xenial is affected by this issue as well. Considering the fact that this
> issue in Bionic affects more tests / kernel variants, and it's way
> easier to fix this in Bionic, I decided to go for Bionic first.
>
> == Fix ==
> 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
> a3355440 (selftests: Fix lib.mk run_tests target shell script)
> 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
> 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)
>
> All these can be cherry-picked into Bionic.
>
> The crucial patch is commit 3f4435b5, the other three patches are just
> some minor fixes that pave the road for it to be cherry-picked.
>
> == Test ==
> Tested on a Bionic amd64 node, the result shows positive.
>
> A test summary before the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END ERROR ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrace
>
> A test summary after the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END GOOD ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrac
>
> No tests were skipped unintentionally. And the raw_skew test in timers
> is now marked as [SKIPPED] and no longer causing failure.
>
> A complete test report could be found in the bug report attachment.
>
> == Regression Potential ==
> Low, this patch set just added support to handle the skip return code
> and changes are limited to the kselftest framework itself, no actual
> impact to normal users.
>
> Mathieu Desnoyers (1):
>   selftests: Fix lib.mk run_tests target shell script
>
> Shuah Khan (1):
>   selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers
>
> Shuah Khan (Samsung OSG) (2):
>   selftests: lib.mk: cleanup RUN_TESTS define and make it readable
>   selftests: lib.mk: add SKIP handling to RUN_TESTS define
>
>  tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------
>  1 file changed, 20 insertions(+), 6 deletions(-)

Makes sense to me and low-impact, considering that it's only affecting
the kselftest framework, therefore:

Acked-by: Andrea Righi <[hidden email]>

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

APPLIED: [Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

Khaled Elmously
In reply to this post by Po-Hsu Lin (Sam)
On 2019-11-06 18:29:02 , Po-Hsu Lin wrote:

> BugLink: https://bugs.launchpad.net/bugs/1812352
>
> == Justification ==
> In the ubuntu_kernel_selftests, the skipped test will uses return code 4
> (KSFT_SKIP).
>
> However the code to handle this non-zero return code was not implemented
> in the kselftest framework. And this will generate some false-positive
> results as those skipped tests were treated as a failure.
>
> For example the raw_skew test in timers on 4.15:
>     1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons
>     not ok 1..7 selftests: raw_skew [FAIL]
>
> And the test_user_copy.sh test in user (failure could be found in KVM kernels):
>     user: module test_user_copy is not found [SKIP]
>     not ok 1..1 selftests: test_user_copy.sh [FAIL]
>
> Xenial is affected by this issue as well. Considering the fact that this
> issue in Bionic affects more tests / kernel variants, and it's way
> easier to fix this in Bionic, I decided to go for Bionic first.
>
> == Fix ==
> 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
> a3355440 (selftests: Fix lib.mk run_tests target shell script)
> 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
> 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)
>
> All these can be cherry-picked into Bionic.
>
> The crucial patch is commit 3f4435b5, the other three patches are just
> some minor fixes that pave the road for it to be cherry-picked.
>
> == Test ==
> Tested on a Bionic amd64 node, the result shows positive.
>
> A test summary before the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END ERROR ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrace
>
> A test summary after the patch:
> END GOOD ubuntu_kernel_selftests.setup
> END GOOD ubuntu_kernel_selftests.breakpoints
> END GOOD ubuntu_kernel_selftests.cpu-hotplug
> END GOOD ubuntu_kernel_selftests.efivarfs
> END GOOD ubuntu_kernel_selftests.memfd
> END GOOD ubuntu_kernel_selftests.memory-hotplug
> END GOOD ubuntu_kernel_selftests.mount
> END GOOD ubuntu_kernel_selftests.net
> END GOOD ubuntu_kernel_selftests.ptrace
> END GOOD ubuntu_kernel_selftests.seccomp
> END GOOD ubuntu_kernel_selftests.timers
> END GOOD ubuntu_kernel_selftests.powerpc
> END GOOD ubuntu_kernel_selftests.user
> END GOOD ubuntu_kernel_selftests.ftrac
>
> No tests were skipped unintentionally. And the raw_skew test in timers
> is now marked as [SKIPPED] and no longer causing failure.
>
> A complete test report could be found in the bug report attachment.
>
> == Regression Potential ==
> Low, this patch set just added support to handle the skip return code
> and changes are limited to the kselftest framework itself, no actual
> impact to normal users.
>
> Mathieu Desnoyers (1):
>   selftests: Fix lib.mk run_tests target shell script
>
> Shuah Khan (1):
>   selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers
>
> Shuah Khan (Samsung OSG) (2):
>   selftests: lib.mk: cleanup RUN_TESTS define and make it readable
>   selftests: lib.mk: add SKIP handling to RUN_TESTS define
>
>  tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------
>  1 file changed, 20 insertions(+), 6 deletions(-)
>
> --
> 2.7.4
>
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team