autopkgtest verision deployed on production infrastructure updated

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

autopkgtest verision deployed on production infrastructure updated

Iain Lane-6
Hi all,

I've just pushed a bunch of commits to our deployed branch of
autopkgtest¹ that bring us up to date with upstream, where the
developers have done a lot of work. Many thanks to them. The most
interesting thing to subscribers to this list will be the changes -
mainly additions - to the list of known test restrictions. I've pasted
the diff of README.package-tests.rst below.

I don't currently expect any breakage to result from this, but if you
notice anything weird please reply here or come to #ubuntu-release and
we'll take a look.

Cheers,

--
Iain Lane                                  [ [hidden email] ]
Debian Developer                                   [ [hidden email] ]
Ubuntu Developer                                   [ [hidden email] ]

[1] https://code.launchpad.net/~ubuntu-release/autopkgtest/+git/development/

@@ -237,14 +245,84 @@ needs-reboot
     The test wants to reboot the machine using
     ``/tmp/autopkgtest-reboot`` (see below).
 
-needs-recommends
+needs-recommends (deprecated)
     Enable installation of recommended packages in apt for the test
     dependencies. This does not affect build dependencies.
 
+    Don't use this for new tests, we want to remove it altogether.
+    See https://lists.debian.org/debian-ci/2018/06/msg00016.html
+    and bug 896698 for the rationale.
+
+flaky
+    The test is expected to fail intermittently, and is not suitable for
+    gating continuous integration. This indicates a bug in either the
+    package under test, a dependency or the test itself, but such bugs
+    can be difficult to fix, and it is often difficult to know when the
+    bug has been fixed without running the test for a while. If a
+    ``flaky`` test succeeds, it will be treated like any other
+    successful test, but if it fails it will be treated as though it
+    had been skipped.
+
+skippable
+    The test might need to be skipped for reasons that cannot be
+    described by an existing restriction such as isolation-machine or
+    breaks-testbed, but must instead be detected at runtime. If the
+    test exits with status 77 (a convention borrowed from Automake), it
+    will be treated as though it had been skipped. If it exits with any
+    other status, its success or failure will be derived from the exit
+    status and stderr as usual. Test authors must be careful to ensure
+    that ``skippable`` tests never exit with status 77 for reasons that
+    should be treated as a failure.
+
+skip-not-installable
+    This test might have test dependencies that can't be fulfilled on all
+    architectures. Therefore, when apt-get installs the dependencies, it will
+    fail. Don't treat this as a test failure, but instead treat it as if the
+    test was skipped.
+
+hint-testsuite-triggers
+    This test exists purely as a hint to suggest when rerunning the
+    tests is likely to be useful.  Specifically, it exists to
+    influence the way dpkg-source generates the Testsuite-Triggers
+    .dsc header from test metadata: the Depends for this test are
+    to be added to Testsuite-Triggers.  (Just as they are for any other
+    test.)
+
+    The test with the hint-testsuite-triggers restriction should not
+    actually be run.
+
+    The packages listed as Depends for this test are usually indirect
+    dependencies, updates to which are considered to pose a risk of
+    regressions in other tests defined in this package.
+
+    There is currently no way to specify this hint on a per-test
+    basis; but in any case the debian.org machinery is not able to
+    think about triggering individual tests.
+
+superficial
+    The test does not provide significant test coverage, so if it
+    passes, that does not necessarily mean that the package under test
+    is actually functional. If a ``superficial`` test fails, it will be
+    treated like any other failing test, but if it succeeds, this is
+    only a weak indication of success. Continuous integration systems
+    should treat a package where all non-superficial tests are skipped as
+    equivalent to a package where all tests are skipped.
+
+    For example, a C library might have a superficial test that simply
+    compiles, links and executes a "hello world" program against the
+    library under test but does not attempt to make use of the library's
+    functionality, while a Python or Perl library might have a
+    superficial test that runs ``import foo`` or ``require Foo;`` but
+    does not attempt to use the library beyond that.
+
 Defined features
 ----------------
 
-There are no currently defined Features.
+test-name
+    Set an explicit test name for the log heading and the ``summary`` file
+    for a ``Test-Command:`` inline test. When not given, these are
+    enumerated like ``command1``.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: autopkgtest verision deployed on production infrastructure updated

Andreas Hasenack-5
On Mon, Dec 10, 2018 at 8:11 AM Iain Lane <[hidden email]> wrote:

>
> +skippable
> +    The test might need to be skipped for reasons that cannot be
> +    described by an existing restriction such as isolation-machine or
> +    breaks-testbed, but must instead be detected at runtime. If the
> +    test exits with status 77 (a convention borrowed from Automake), it
> +    will be treated as though it had been skipped. If it exits with any
> +    other status, its success or failure will be derived from the exit
> +    status and stderr as usual. Test authors must be careful to ensure
> +    that ``skippable`` tests never exit with status 77 for reasons that
> +    should be treated as a failure.
>

Is it ok to use skippable for arches where the test is known to not
work? AFAIK there is no arch restriction support in d/t/control

I would then mark the test as skippable, detect the arch at runtime,
and  if it's one we know it won't work, exit 77. I understand care
must be taken to check the test and make sure it doesn't exit 77 for
other reasons in that case.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: autopkgtest verision deployed on production infrastructure updated

Iain Lane-6
On Tue, Dec 11, 2018 at 12:16:26PM -0200, Andreas Hasenack wrote:

> On Mon, Dec 10, 2018 at 8:11 AM Iain Lane <[hidden email]> wrote:
> >
> > +skippable
> > +    The test might need to be skipped for reasons that cannot be
> > +    described by an existing restriction such as isolation-machine or
> > +    breaks-testbed, but must instead be detected at runtime. If the
> > +    test exits with status 77 (a convention borrowed from Automake), it
> > +    will be treated as though it had been skipped. If it exits with any
> > +    other status, its success or failure will be derived from the exit
> > +    status and stderr as usual. Test authors must be careful to ensure
> > +    that ``skippable`` tests never exit with status 77 for reasons that
> > +    should be treated as a failure.
> >
>
> Is it ok to use skippable for arches where the test is known to not
> work? AFAIK there is no arch restriction support in d/t/control
>
> I would then mark the test as skippable, detect the arch at runtime,
> and  if it's one we know it won't work, exit 77. I understand care
> must be taken to check the test and make sure it doesn't exit 77 for
> other reasons in that case.
That sounds fine, but consider first if you can encode *what* about the
system makes the test fail, if that can be done in a better way than the
architecture itself. You might be able to use skip-not-installable if
this is expressable via dependencies.

Cheers,

--
Iain Lane                                  [ [hidden email] ]
Debian Developer                                   [ [hidden email] ]
Ubuntu Developer                                   [ [hidden email] ]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment