Re: Ubuntu Focal update of broken Calibre package

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

James Cuzella
Hello,

I recently wanted to test out Calibre with a newer Kindle Fire tablet. It turns out that I ran into many of the issues mentioned with the broken 4.99.4+dfsg+really4.12.0-1build1 package version.  I was able to get this old version of Calibre working to identify the tablet on Focal with an updated version of libmtp.  This is in my PPA here: https://launchpad.net/~trinitronx/+archive/ubuntu/calibre

So, knowing some packaging tricks for Ubuntu & Debian, I decided to try and create an updated Calibre package in a PPA for Focal / 20.04.1.  I ran into the issue of first needing to backport all the build dependencies.  I started out by using the same version of Calibre ( 5.6.0+dfsg ) from the source package in Ubuntu groovy, and then after reading this thread trying to backport the versions from Debian bullseye & sid.  So far I've been able to backport the following:

  • pyqt-builder  1.4.0+dfsg-1 and 1.6.0+dfsg-1
  • sip4  4.19.24+dfsg-1
  • sip5  5.3.0+dfsg-2  and 5.5.0+dfsg-1
  • pyqt5-sip 12.8.1-1
Note: I also backported debhelper 13.2 in that PPA, but ended up not using it because strict build-depends on "debhelper (=12)" is on most of the other packages in Focal, and Soyuz / pbuilder will fail on that build dependency.  Instead, I just switched back to "debhelper (=12)" in the source package's debian/control files.
 
Most packages I mentioned will need a rebuild and a patch to use a different
location for PyQt5 *.sip files.

This explains exactly the issues I ran into for the main Calibre package! 😊
So far I have tried two routes to resolve the build dependency:

  • sip4 4.19.24+dfsg-1 + pyqt-builder 1.4.0+dfsg-1
    • Patched Calibre's setup.py to use Python3
    • Patched Calibre's setup/build.py to use sip-module = "sip" 
    • Ran into issues with python3-sipbuilder
      • It could not find the package path, as you described:
        SIPing 3 files...
        /usr/bin/python3.8 -c import os; os.chdir('/media/terabyte/src/pub/calibre-debian/calibre/build/pyqt/pictureflow'); from sipbuild.tools.build import main; main(); --verbose --no-make --qmake /usr/bin/qmake
        Querying qmake about your Qt installation...
        /usr/bin/qmake -query
        These bindings will be built: pictureflow.
        Generating the pictureflow bindings...
        -c: Unable to find file "QtWidgets/QtWidgetsmod.sip"

      • There was a problem with how python3-sipbuilder's "get_package_dir" function was working.
        The "sip_parts" result of the split only had 1 array item: ["sip"]
        The code discards the last array item, assuming it's a filename extension.
        Therefore, it creates an empty array that gets passed to: os.path.join()
  • sip5 5.3.0+dfsg-2  + pyqt-builder 1.4.0+dfsg-1 + pyqt5-dev 5.14.1+dfsg-3build1
    • This option required just patching Calibre's setup.py to use Python3
    • Tried downgrading Build-Depends to pyqt5-dev (>= 5.14.1+dfsg-3build1)
    • I became blocked at first on backporting the build dependencies for the pyqt5-sip / PyQt5.sip and many other pyqt5 packages.  Not sure what the issue was (I've never seen or used PyQt5 or SIP before this... seemed like a large rabbit hole)
  •  sip5 5.5.0+dfsg-1 + pyqt-builder / python3-pyqtbuild 1.6.0+dfsg-1
     + pyqt5-sip / python3-pyqt5.sip
     12.8.1-1 + pyqt5 
    5.15.2+dfsg-1 (all backported)
    • Again, just added the patch Calibre's setup.py to use Python3
    • This time it's missing a package: python3-pyqt5.qtwebengine

      dpkg-checkbuilddeps: error: Unmet build dependencies: python3-pyqt5.qtwebengine (>= 5.12.1-4+b1)
       

You can see this in Debian: after I uploaded new PyQt5 (built with SIP 5) to
unstable, immediately some packages started to fail to build from source:

- https://bugs.debian.org/971173 (krita)
- https://bugs.debian.org/971217 (python-poppler-qt5)
- https://bugs.debian.org/971172 (veusz)

And packages that were not rebuilt got runtime errors:

- https://bugs.debian.org/971538 (qgis)
- https://bugs.debian.org/970921 (calibre)

So what I said is true: this requires simultaneous changes in many packages,
even if it's just a rebuild for some of them.


I think if I or someone else here can figure out how to get python3-pyqt5.qtwebengine to build, then Calibre may also be able to build with this set of dependencies.  Note, I'm not an "official" Debian or Ubuntu developer and have not gone through the onboarding / mentorship process to become one.  So for now, these efforts are for a Focal targeted PPA only.  However, with some help this could be the first step towards getting an official set into focal-backports.

I would appreciate any help with figuring out how to get that last python3-pyqt5.qtwebengine package to be generated from the pyqt5 source package.  I'm assuming this is where it comes from, given that all the other similarly named ones were generated from that one.  Also, the Debian package site shows this as the source package: https://packages.debian.org/stretch/python3-pyqt5.qtwebengine

Thanks,
- James Cuzella

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Dmitry Shachnev-2
Hi James!

On Fri, Dec 11, 2020 at 08:43:16PM -0700, James Cuzella wrote:
> I would appreciate any help with figuring out how to get that
> last python3-pyqt5.qtwebengine package to be generated from the pyqt5
> source package.  I'm assuming this is where it comes from, given that all
> the other similarly named ones were generated from that one.  Also, the
> Debian package site shows this as the source package:
> https://packages.debian.org/stretch/python3-pyqt5.qtwebengine

It used to be built from pyqt5 source, but since then it got a new source
package, pyqt5webengine:

https://launchpad.net/ubuntu/+source/pyqt5webengine

https://packages.debian.org/sid/source/pyqt5webengine

--
Dmitry Shachnev

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

James Cuzella
Thanks Dmitry for pointing me in the right direction!

I tried to backport the set of Qt5 and qtwebengine packages and found out what large set of build dependencies it was!

So what I said is true: this requires simultaneous changes in many packages,
even if it's just a rebuild for some of them.

Yes!  Qt5 is indeed a very large project too!  I'm sure there is a Lord of the Rings meme that applies here like: "One does not simply backport Qt5" 😄

Simply using the backportpackage tool on qtwebengine-opensource-src started a build dependency hunt that resulted in finding a massive set of Qt5 packages that needed to be backported (without relaxing version constraints).
A lot of these eventually resolved into being blocked on circular dependency chains.

I was able to spend a few hours on this to visually represent what I found (See Gist linked below):


I'm not sure how the Debian or Ubuntu teams build and backport Qt5 packages when so many build dependencies resolve circularly.
I suppose they must bootstrap this dependency chain somehow, but it seems like more work than I have time to figure out at the moment.

Any ideas for how to simplify this?

Thanks,
- James Cuzella 

On Sun, Dec 13, 2020 at 10:16 AM Dmitry Shachnev <[hidden email]> wrote:
Hi James!

On Fri, Dec 11, 2020 at 08:43:16PM -0700, James Cuzella wrote:
> I would appreciate any help with figuring out how to get that
> last python3-pyqt5.qtwebengine package to be generated from the pyqt5
> source package.  I'm assuming this is where it comes from, given that all
> the other similarly named ones were generated from that one.  Also, the
> Debian package site shows this as the source package:
> https://packages.debian.org/stretch/python3-pyqt5.qtwebengine

It used to be built from pyqt5 source, but since then it got a new source
package, pyqt5webengine:

https://launchpad.net/ubuntu/+source/pyqt5webengine

https://packages.debian.org/sid/source/pyqt5webengine

--
Dmitry Shachnev

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Dmitry Shachnev-2
Hi James!

On Tue, Dec 22, 2020 at 06:11:15PM -0700, James Cuzella wrote:
> Yes!  Qt5 is indeed a very large project too!  I'm sure there is a Lord of
> the Rings meme that applies here like: "One does not simply backport Qt5" 😄

This is true :)

> Simply using the backportpackage tool on qtwebengine-opensource-src started
> a build dependency hunt that resulted in finding a massive set of Qt5
> packages that needed to be backported (without relaxing version
> constraints).
> A lot of these eventually resolved into being blocked on circular
> dependency chains.
>
> I was able to spend a few hours on this to visually represent what I found
> (See Gist linked below):
>
> https://gist.github.com/b005fa0ef6e600c6a6e0dfd22dd3e604
>
> I'm not sure how the Debian or Ubuntu teams build and backport Qt5 packages
> when so many build dependencies resolve circularly.
> I suppose they must bootstrap this dependency chain somehow, but it seems
> like more work than I have time to figure out at the moment.
Yes, in Ubuntu we bootstrap the packages manually for every new Qt release.

If you build packages locally, you can follow the instruction in qtbase's
debian/README.source (I have just pushed a minor update for it):

https://salsa.debian.org/qt-kde-team/qt/qtbase/-/blob/master/debian/README.source#L16

If you are pushing packages to a PPA, you can use the following procedure:

- Push qtbase and qtdeclarative where the -doc and -doc-html packages, and
also all Build-Depends-Indep are removed.

- Push qttools with the same change, and additionally remove libqt5webkit5-dev
build-dependency and make qwebview_archs empty in debian/rules.

- Wait until qttools builds on amd64, then push the normal versions of all
packages.

In Debian this bootstrapping happens automatically because Debian does
separate builds for "amd64" (arch-dependent) and "all" (arch-indep)
architectures.

Also keep in mind that if you update Qt, you will also need to do no-change
rebuilds of packages that depend on qtbase-abi-5-x-y virtual package or on
qtdeclarative-abi-5-x-y (usually these are packages that use private ABI).
Otherwise they can stop working.

--
Dmitry Shachnev

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment