[SRU disco/cosmic/bionic] linux-buildinfo getabis enhancements

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[SRU disco/cosmic/bionic] linux-buildinfo getabis enhancements

Andy Whitcroft-3
In reviewing the linux-buildinfo support it was pointed out that getabis
would have problems at the transition from one form to the other.  We have
also long had a problem with errors during downloads being hidden from the
user (as we expect some configurations to not have all of the packages).
This leads us to commit partial ABI information and us to have build
failures which could be avoided.

This patch kit does four things.  Firstly it adds descrimination
of download failures, separating transport failures (which it makes
instantly fatal) from server generated failures (404 etc).  Secondly it
uses this discrimination to allow us to identify the presence of valid
combinations of packages and avoid partial ABI downloads.  Thirdly it
uses this ability to identify valid combinations to iterate over the
combinations valid in any series, allowing us to use the same getabis
script for all series unmodified.  Finally it adds support for parsing
complete version numbers to simply execution.

Proposing for application on top of the linux-buildinfo fixes on disco,
cosmic, and bionic.

Below is an example transcript of this running in cosmic to make it
easier to see how the output is modified:

    $ ./debian/scripts/misc/getabis
    4.18.0-12.13
    Fetching generic(armhf)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=n
      set: linux-image=y linux-modules=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic-lpae(armhf)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=n
      set: linux-image=y linux-modules=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic(amd64)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=y linux-modules=y linux-modules-extra=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching lowlatency(amd64)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=y linux-modules=y linux-modules-extra=n
      set: linux-image-unsigned=y linux-modules=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic(i386)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching lowlatency(i386)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=n
      set: linux-image=y linux-modules=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic(arm64)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching snapdragon(arm64)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=n
      set: linux-image=y linux-modules=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic(ppc64el)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=y linux-modules=y linux-modules-extra=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    Fetching generic(s390x)...
      set: linux-buildinfo=n
      set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
      set: linux-image-unsigned=n linux-modules=-
      set: linux-image=y linux-modules=y linux-modules-extra=y
      extracting...
      images...
        GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
      done
    rm 'debian.master/abi/4.18.0-11.12/abiname'
    [...]
    $

-apw

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[disco/master-next 1/2] UBUNTU: [Packaging] getabis -- handle all known package combinations

Andy Whitcroft-3
Traditionally we have tried to download all and any packages we can
find.  If we have any packages we just assume that what we got is a
consistent set and use it.  This leads to incomplete sets being
committed on network failure.

Firstly detect and differentiate transport errors and valid missing
packages.  Secondly switch to analysing known good package set
combinations; this relies on the presumption that the publisher
only publishes all or none of a binary package set.  This lets us
throw errors when we are unable to find an internally consistent
set of packages.

BugLink: http://bugs.launchpad.net/bugs/1806380
Signed-off-by: Andy Whitcroft <[hidden email]>
---
 debian/scripts/misc/getabis | 155 +++++++++++++++++++++---------------
 1 file changed, 91 insertions(+), 64 deletions(-)

diff --git a/debian/scripts/misc/getabis b/debian/scripts/misc/getabis
index 42690b0311e2..edba4b078867 100755
--- a/debian/scripts/misc/getabis
+++ b/debian/scripts/misc/getabis
@@ -28,7 +28,7 @@ fwinfo=$abidir/fwinfo
 test -d $tmpdir || mkdir $tmpdir
 
 package_prefixes() {
- __package_prefixes="$@"
+ : # no longer used ...
 }
 
 getall() {
@@ -39,89 +39,118 @@ getall() {
 
  for sub in $@; do
  if [ -f $abidir/$arch/$sub ]; then
- echo "Exists: $sub"
+ echo "Existing $sub($arch)..."
  continue
  fi
- echo -n "Fetching $sub($arch)..."
- prefixes=""
+ echo "Fetching $sub($arch)..."
+ getall_set "linux-buildinfo" "$arch" "$sub" || \
+ getall_set "linux-image-unsigned linux-modules linux-modules-extra" "$arch" "$sub" || \
+ getall_set "linux-image-unsigned linux-modules" "$arch" "$sub" || \
+ getall_set "linux-image linux-modules linux-modules-extra" "$arch" "$sub" || \
+ getall_set "linux-image linux-modules" "$arch" "$sub" || \
+ getall_set "linux-image linux-image-extra" "$arch" "$sub" || \
+ getall_set "linux-image" "$arch" "$sub" || \
+ { echo "FAILED"; exit 1; }
+ done
+}
+getall_set()
+{
+ prefixes="$1"
+ arch="$2"
+ sub="$3"
+ (
+ echo -n "  set:"
  filenames=""
  cd $tmpdir
- for prefix in $__package_prefixes
+ found=1
+ for prefix in $prefixes
  do
+ echo -n " $prefix="
+ if [ "$found" = 0 ]; then
+ echo -n "-"
+ continue
+ fi
  filename=${prefix}-${verabi}-${sub}_${verfull}_${arch}.deb
  for r in "${repo_list[@]}"
  do
  if ! [ -f $filename ]; then
  $WGET $r/$filename
+ rc="$?"
+ # If this was not successful or a valid error
+ # return from the server all bets are off, bail.
+ [ "$rc" != 0 -a "$rc" != 8 ] && return 1
  fi
  if [ -f $filename ]; then
- prefixes="$prefixes $prefix"
+ echo -n "y"
  filenames="$filenames $filename"
  break
  fi
  done
+ if [ ! -f "$filename" ]; then
+ echo -n "n"
+ found=0
+ fi
  done
- if [ "$filenames" != "" ]; then
- echo -n "extracting$prefixes..."
- for filename in $filenames
- do
- dpkg-deb --extract $filename tmp
+ echo ""
+ if [ "$found" = 0 ]; then
+ return 1
+ fi
+ echo "  extracting..."
+ for filename in $filenames
+ do
+ dpkg-deb --extract $filename tmp
+ done
+ # FORM 1: linux-image et al extracted here.
+ if [ -d tmp/boot ]; then
+ echo "  images..."
+ find tmp -name "*.ko" | while read f; do
+ modinfo $f | grep ^firmware >> $fwinfo
  done
- # FORM 1: linux-image et al extracted here.
- if [ -d tmp/boot ]; then
- echo -n "images..."
- find tmp -name "*.ko" | while read f; do
- modinfo $f | grep ^firmware >> $fwinfo
- done
- if [ -f tmp/boot/abi-* ]; then
- mv tmp/boot/abi-* $abidir/$arch/$sub
- else
- echo -n "NO ABI FILE..."
- fi
- if [ -f tmp/boot/retpoline-* ]; then
- mv tmp/boot/retpoline-* $abidir/$arch/$sub.retpoline
- else
- echo -n "NO RETPOLINE FILE..."
- fi
- (cd tmp; find lib/modules/$verabi-$sub/kernel -name '*.ko') | \
- sed -e 's/.*\/\([^\/]*\)\.ko/\1/' | sort > \
- $abidir/$arch/$sub.modules
- (
- cd tmp;
- # Prevent exposing some errors when called by python scripts. SIGPIPE seems to get
- # exposed when using the `find ...` form of the command.
- ko=$(find lib/modules/$verabi-$sub/kernel \
- -name '*.ko' | head -1)
- readelf -p .comment "$ko" | gawk '
- ($1 == "[") {
- printf("%s", $3);
- for (n=4; n<=NF; n++) {
- printf(" %s", $n);
- }
- print ""
- }' | sort -u >$abidir/$arch/$sub.compiler
- version=`cat $abidir/$arch/$sub.compiler`
- echo -n "$version..."
- )
- # FORM 2: moduleinfo packages
- # cranky-fix -- modinfo supported
+ if [ -f tmp/boot/abi-* ]; then
+ mv tmp/boot/abi-* $abidir/$arch/$sub
+ else
+ echo "    NO ABI FILE"
+ fi
+ if [ -f tmp/boot/retpoline-* ]; then
+ mv tmp/boot/retpoline-* $abidir/$arch/$sub.retpoline
  else
- echo -n "buildinfo..."
- base="tmp/usr/lib/linux/${verabi}-${sub}"
- mv "$base/abi" "$abidir/$arch/$sub"
- for comp in 'modules' 'retpoline' 'compiler'
- do
- mv "$base/$comp" "$abidir/$arch/$sub.$comp"
- done
- cat "$base/fwinfo" >>"$fwinfo"
+ echo "    NO RETPOLINE FILE"
  fi
- rm -rf tmp $filenames
- echo "done."
+ (cd tmp; find lib/modules/$verabi-$sub/kernel -name '*.ko') | \
+ sed -e 's/.*\/\([^\/]*\)\.ko/\1/' | sort > \
+ $abidir/$arch/$sub.modules
+ (
+ cd tmp;
+ # Prevent exposing some errors when called by python scripts. SIGPIPE seems to get
+ # exposed when using the `find ...` form of the command.
+ ko=$(find lib/modules/$verabi-$sub/kernel \
+ -name '*.ko' | head -1)
+ readelf -p .comment "$ko" | gawk '
+ ($1 == "[") {
+ printf("%s", $3);
+ for (n=4; n<=NF; n++) {
+ printf(" %s", $n);
+ }
+ print ""
+ }' | sort -u >$abidir/$arch/$sub.compiler
+ version=`cat $abidir/$arch/$sub.compiler`
+ echo "    $version"
+ )
+ # FORM 2: moduleinfo packages
+ # cranky-fix -- modinfo supported
  else
- echo "FAILED."
+ echo "  buildinfo..."
+ base="tmp/usr/lib/linux/${verabi}-${sub}"
+ mv "$base/abi" "$abidir/$arch/$sub"
+ for comp in 'modules' 'retpoline' 'compiler'
+ do
+ mv "$base/$comp" "$abidir/$arch/$sub.$comp"
+ done
+ cat "$base/fwinfo" >>"$fwinfo"
  fi
- cd $origdir
- done
+ rm -rf tmp $filenames
+ echo "  done"
+ )
 }
 
 # MAIN
@@ -133,8 +162,6 @@ echo $abi > $abidir/abiname
 # NOTE: The flavours are hardcoded, because they may have changed from the
 # current build.
 
-__package_prefixes="linux-image"
-
 . $DEBIAN/etc/getabis
 
 # Extract compiler source package version from e.g.:
--
2.19.1


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[disco/master-next 2/2] UBUNTU: [Packaging] getabis -- support parsing a simple version

Andy Whitcroft-3
In reply to this post by Andy Whitcroft-3
BugLink: http://bugs.launchpad.net/bugs/1806380
Signed-off-by: Andy Whitcroft <[hidden email]>
---
 debian/scripts/misc/getabis | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/debian/scripts/misc/getabis b/debian/scripts/misc/getabis
index edba4b078867..24e19d94c1ff 100755
--- a/debian/scripts/misc/getabis
+++ b/debian/scripts/misc/getabis
@@ -2,7 +2,11 @@
 
 export LC_ALL=C.UTF-8
 
+if [ "$#" = "1" ]; then
+ set - $(echo "$1" | sed -e 's/-/ /')
+fi
 if [ "$#" != "2" ]; then
+ echo "Usage: $0 <version>" 1>&2
  echo "Usage: $0 <release> <revision>" 1>&2
  exit 1
 fi
--
2.19.1


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK / APPLIED[D]: [SRU disco/cosmic/bionic] linux-buildinfo getabis enhancements

Seth Forshee
In reply to this post by Andy Whitcroft-3
On Tue, Dec 04, 2018 at 09:32:00PM +0000, Andy Whitcroft wrote:

> In reviewing the linux-buildinfo support it was pointed out that getabis
> would have problems at the transition from one form to the other.  We have
> also long had a problem with errors during downloads being hidden from the
> user (as we expect some configurations to not have all of the packages).
> This leads us to commit partial ABI information and us to have build
> failures which could be avoided.
>
> This patch kit does four things.  Firstly it adds descrimination
> of download failures, separating transport failures (which it makes
> instantly fatal) from server generated failures (404 etc).  Secondly it
> uses this discrimination to allow us to identify the presence of valid
> combinations of packages and avoid partial ABI downloads.  Thirdly it
> uses this ability to identify valid combinations to iterate over the
> combinations valid in any series, allowing us to use the same getabis
> script for all series unmodified.  Finally it adds support for parsing
> complete version numbers to simply execution.
>
> Proposing for application on top of the linux-buildinfo fixes on disco,
> cosmic, and bionic.

Looks good to me, seems to be working for disco/master-next at least.

Acked-by: Seth Forshee <[hidden email]>

Applied to disco/master-next, thanks!

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

ACK: [SRU disco/cosmic/bionic] linux-buildinfo getabis enhancements

Kamal Mostafa-2
In reply to this post by Andy Whitcroft-3

Looks good.

Acked-by: Kamal Mostafa <[hidden email]>

 -Kamal

On Tue, Dec 04, 2018 at 09:32:00PM +0000, Andy Whitcroft wrote:

> In reviewing the linux-buildinfo support it was pointed out that getabis
> would have problems at the transition from one form to the other.  We have
> also long had a problem with errors during downloads being hidden from the
> user (as we expect some configurations to not have all of the packages).
> This leads us to commit partial ABI information and us to have build
> failures which could be avoided.
>
> This patch kit does four things.  Firstly it adds descrimination
> of download failures, separating transport failures (which it makes
> instantly fatal) from server generated failures (404 etc).  Secondly it
> uses this discrimination to allow us to identify the presence of valid
> combinations of packages and avoid partial ABI downloads.  Thirdly it
> uses this ability to identify valid combinations to iterate over the
> combinations valid in any series, allowing us to use the same getabis
> script for all series unmodified.  Finally it adds support for parsing
> complete version numbers to simply execution.
>
> Proposing for application on top of the linux-buildinfo fixes on disco,
> cosmic, and bionic.
>
> Below is an example transcript of this running in cosmic to make it
> easier to see how the output is modified:
>
>     $ ./debian/scripts/misc/getabis
>     4.18.0-12.13
>     Fetching generic(armhf)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=n
>       set: linux-image=y linux-modules=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic-lpae(armhf)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=n
>       set: linux-image=y linux-modules=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic(amd64)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=y linux-modules=y linux-modules-extra=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching lowlatency(amd64)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=y linux-modules=y linux-modules-extra=n
>       set: linux-image-unsigned=y linux-modules=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic(i386)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching lowlatency(i386)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=n
>       set: linux-image=y linux-modules=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic(arm64)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching snapdragon(arm64)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=n
>       set: linux-image=y linux-modules=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic(ppc64el)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=y linux-modules=y linux-modules-extra=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     Fetching generic(s390x)...
>       set: linux-buildinfo=n
>       set: linux-image-unsigned=n linux-modules=- linux-modules-extra=-
>       set: linux-image-unsigned=n linux-modules=-
>       set: linux-image=y linux-modules=y linux-modules-extra=y
>       extracting...
>       images...
>         GCC: (Ubuntu 8.2.0-7ubuntu1) 8.2.0
>       done
>     rm 'debian.master/abi/4.18.0-11.12/abiname'
>     [...]
>     $
>
> -apw
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team