data.tar.xz support added to Launchpad

[Colin Watson]

It should now be possible to use xz compression for .deb packages
uploaded to Ubuntu oneiric (using 'dh_builddeb -- -Zxz').  In order to
do so, you must include this field in the relevant binary packages:

  Pre-Depends: dpkg (>= 1.15.6)

(This implies that such packages can only be uploaded to maverick and
later.)

Please do not use this for packages that are Priority: required or
Priority: important, as you will break the installer if you do (and
relaxing that restriction would have implications for people's ability
to debootstrap Ubuntu on various minimal systems).

Please check that it's actually worth compressing a package using xz
before doing so!  You can easily take the data.tar.gz, recompress it
using xz, and compare the size.

If you maintain a package that already uses -Zlzma, consider converting
it to -Zxz; compression ratios should be similar, and it's a
better-designed format.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Barry Warsaw]

On Aug 26, 2011, at 10:32 PM, Colin Watson wrote:

>It should now be possible to use xz compression for .deb packages
>uploaded to Ubuntu oneiric (using 'dh_builddeb -- -Zxz').  In order to
>do so, you must include this field in the relevant binary packages:
>
>  Pre-Depends: dpkg (>= 1.15.6)

I just ran into this when I merged testing's version of m2crypto, which had
switched to xz compression.  The merge was tricky and I missed this tidbit in
the Debian packaging, but I did do a local sbuild for precise before uploaded,
and it succeeded without the Pre-Depends.

However, when the buildd got to that point, it failed:

https://launchpadlibrarian.net/83677349/buildlog_ubuntu-precise-amd64.m2crypto_0.21.1-2ubuntu1_BUILDING.txt.gz

I need to try it again on pbuilder to see what happens there.  Of course, it's
easy enough to add the Pre-Depends, but now I'm gunshy and don't trust my
sbuild environment, so I'm building it in a PPA first.  Unfortunately, that's
scheduled for *2 days* from now (!).  Yeah, I'll try to get it re-scored, but
that's beside the point.

Any idea why it succeeded in sbuild, and do you have any other suggestions for
testing such package builds locally before uploading?

-Barry

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Micah Gersten-8]

On 10/25/2011 03:15 PM, Barry Warsaw wrote:

On Aug 26, 2011, at 10:32 PM, Colin Watson wrote:

It should now be possible to use xz compression for .deb packages
uploaded to Ubuntu oneiric (using 'dh_builddeb -- -Zxz').  In order to
do so, you must include this field in the relevant binary packages:

 Pre-Depends: dpkg (>= 1.15.6)

I just ran into this when I merged testing's version of m2crypto, which had
switched to xz compression.  The merge was tricky and I missed this tidbit in
the Debian packaging, but I did do a local sbuild for precise before uploaded,
and it succeeded without the Pre-Depends.

However, when the buildd got to that point, it failed:

https://launchpadlibrarian.net/83677349/buildlog_ubuntu-precise-amd64.m2crypto_0.21.1-2ubuntu1_BUILDING.txt.gz

I need to try it again on pbuilder to see what happens there.  Of course, it's
easy enough to add the Pre-Depends, but now I'm gunshy and don't trust my
sbuild environment, so I'm building it in a PPA first.  Unfortunately, that's
scheduled for *2 days* from now (!).  Yeah, I'll try to get it re-scored, but
that's beside the point.

Any idea why it succeeded in sbuild, and do you have any other suggestions for
testing such package builds locally before uploading?

-Barry

It's a check on upload and not during the build would be my guess.  Maybe we should add a lintian warning/error if it uses .xz debs w/out the pre-depends.

Micah

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Steve Langasek-6]

On Tue, Oct 25, 2011 at 04:15:06PM -0400, Barry Warsaw wrote:
> However, when the buildd got to that point, it failed:

> https://launchpadlibrarian.net/83677349/buildlog_ubuntu-precise-amd64.m2crypto_0.21.1-2ubuntu1_BUILDING.txt.gz

I think you probably mean this error:

chroot-autobuild/build/buildd/python-m2crypto_0.21.1-2ubuntu1_amd64.deb:
dpkg-deb: file `chroot-autobuild/build/buildd/python-m2crypto_0.21.1-2ubuntu1_amd64.deb' contains ununderstood data member data.tar.xz     , giving up

(had to search on 'xz' to find the error in the build logs)

This is not part of the package build per se, but is part of extra
inspection the buildds do of the package after it's built in order to put
useful information into the build logs.  This part runs using the dpkg-deb
from outside the build chroot, and outside the build chroot we're not
running precise - we're running lucid.  So you're going to get this error in
the build log of any package that uses .xz, whether or not you add the
Pre-Depends, and should ignore it.

As Micah says, the error we need to care about is not a build-time error at
all; it's the archive check that blocks the *upload* of a package that
doesn't pass muster.  So there's nothing wrong with your sbuild setup that
you aren't seeing this locally.

> Any idea why it succeeded in sbuild, and do you have any other suggestions
> for testing such package builds locally before uploading?

Fine-toothed-combing of the merge delta, and, when all else fails, that's
what the archive check is there for.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Barry Warsaw-2]

Thanks Micah and Steve for the great explanations.

On Oct 25, 2011, at 03:18 PM, Micah Gersten wrote:

>It's a check on upload and not during the build would be my guess.
>Maybe we should add a lintian warning/error if it uses .xz debs w/out
>the pre-depends.

I think that could help, but it's also not a huge deal to add the Pre-Depends
and bump the version number.  It's pretty obvious from the error message what
the problem was (even if the ultimate location of the problem wasn't).  If it
was added to lintian, I think it should be an error because warnings can
sometimes get overlooked, and this is a pretty severe failure.

But this makes me think of another question: why didn't the Debian package
have that same Pre-Depends.  Extrapolating from Steve's explanation, my guess
is that they don't have the same distro-version restriction setup that we do
(i.e. outside the chroot).

So what would happen if I just sync'd m2crypto instead of merging it?  Would
it still fail without the Pre-Depends? (my guess: yes).  This would probably
force me to add the Pre-Depends and upload a 1ubuntu1 version, since we'd be
carrying a delta.  Unless i could convince the Debian maintainer to add that
line for us.  It wouldn't have any negative consequences for Debian, would it?

Cheers,
-Barry

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Stefano Rivera-13]

Hi Barry (2011.10.25_22:45:23_+0200)

> But this makes me think of another question: why didn't the Debian package
> have that same Pre-Depends.  Extrapolating from Steve's explanation, my guess
> is that they don't have the same distro-version restriction setup that we do
> (i.e. outside the chroot).
>
> So what would happen if I just sync'd m2crypto instead of merging it?  Would
> it still fail without the Pre-Depends? (my guess: yes).  This would probably
> force me to add the Pre-Depends and upload a 1ubuntu1 version, since we'd be
> carrying a delta.  Unless i could convince the Debian maintainer to add that
> line for us.  It wouldn't have any negative consequences for Debian, would it?

Yes to all of the above.

There were a bunch of missing Pre-Depends in the initial syncs this
cycle.  They were all caught pretty quickly.

SR

--
Stefano Rivera
  http://tumbleweed.org.za/
  H: [phone] C: [phone]  UCT: x3127

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Steve Langasek-6]

On Tue, Oct 25, 2011 at 04:45:23PM -0400, Barry Warsaw wrote:

> But this makes me think of another question: why didn't the Debian package
> have that same Pre-Depends.  Extrapolating from Steve's explanation, my
> guess is that they don't have the same distro-version restriction setup
> that we do (i.e.  outside the chroot).

The pre-dependency is satisfied by the version of dpkg in the previous
Debian release, so having it or not has no impact on Debian upgrades.  It
impacts Ubuntu because lucid shipped with an earlier version of dpkg, so the
Pre-Depends is needed to give smooth LTS->LTS upgrades.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Dustin Kirkland-5]

On Tue, Oct 25, 2011 at 3:18 PM, Micah Gersten <[hidden email]> wrote:
> It's a check on upload and not during the build would be my guess.  Maybe we
> should add a lintian warning/error if it uses .xz debs w/out the

Speaking of Lintian warnings, would it make sense to have lintian
duplicate the original .gz, compress with xz, check the difference and
if it's greater than $threshold percentage, print a warning (or info)
that using xz compression would save $some %?

--
:-Dustin

Dustin Kirkland
Ubuntu Core Developer

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Colin Watson]

On Tue, Oct 25, 2011 at 04:13:17PM -0500, Dustin Kirkland wrote:
> On Tue, Oct 25, 2011 at 3:18 PM, Micah Gersten <[hidden email]> wrote:
> > It's a check on upload and not during the build would be my guess.  Maybe we
> > should add a lintian warning/error if it uses .xz debs w/out the
>
> Speaking of Lintian warnings, would it make sense to have lintian
> duplicate the original .gz, compress with xz, check the difference and
> if it's greater than $threshold percentage, print a warning (or info)
> that using xz compression would save $some %?

We shouldn't actively promote xz like that until the need for the
Pre-Depends has gone away (i.e. after 12.04 LTS).

Furthermore, Lintian checks need to be reasonably quick or else people
will Ctrl-C it.  The above would be very slow on many packages.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Ritchie]

On 10/26/2011 12:45 AM, Colin Watson wrote:

> On Tue, Oct 25, 2011 at 04:13:17PM -0500, Dustin Kirkland wrote:
>> On Tue, Oct 25, 2011 at 3:18 PM, Micah Gersten <[hidden email]> wrote:
>>> It's a check on upload and not during the build would be my guess.  Maybe we
>>> should add a lintian warning/error if it uses .xz debs w/out the
>>
>> Speaking of Lintian warnings, would it make sense to have lintian
>> duplicate the original .gz, compress with xz, check the difference and
>> if it's greater than $threshold percentage, print a warning (or info)
>> that using xz compression would save $some %?
>
> We shouldn't actively promote xz like that until the need for the
> Pre-Depends has gone away (i.e. after 12.04 LTS).
>
> Furthermore, Lintian checks need to be reasonably quick or else people
> will Ctrl-C it.  The above would be very slow on many packages.
>

If you're not going to lintian it, a periodic archive scan/report
somewhere would be appreciated.  In fact we may as well do one soon.

Thanks,
Scott Ritchie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel