Ubuntu Focal update of broken Calibre package

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Ubuntu Focal update of broken Calibre package

Norbert Preining-2
Dear all,

(please Cc)

I am the Debian maintainer of Calibre, and unfortunately it seems that
for Focal Ubuntu has pulled a preliminary version of Calibre, which is
**seriously** broken and unusable, not even starting in most cases.

We were forced by the Python3 transition to temporarily ship pre-release
versions of Calibre. In particular, Ubuntu Focal ships
        4.99.4+dfsg+really4.12.0-1build1
which is version 4.12 with experimental Python3 patches on top of it.
This worked for a short time being until Calibre 5 was released with
proper Python3 support.

Due to this unfortunate squeeze in release timing, Ubuntu Focal users
now have a seriously broken Calibre, and upstream is swamped with bug
reports.

I would strongly suggest and support, and help preparing, an update to
Focal based on the current version in Debian/testing, 5.2.0+dfsg-1,
which has been out since quite some time and field-tested with Python3
in various environments, due to upstream having switched to Py3, too.

Is the above (update to 5.2.0) possible in Ubuntu Focal, and if yes,
what kind if steps are necessary?

Note that I am not an Ubuntu developers, but Debian developer and
maintainer of Calibre.

Thanks and all the best

Norbert

--
PREINING Norbert                              https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Lukasz Zemczak
Hello Norbert,

With my SRU team hat on, after reading what you said about the status
of current calibre, I would say it is possible to update the package
version in focal to 5.2.0. But we would need to know a bit more, and
there is some (important) paperwork to do.

First thing to remember is that for a new version to be releasable to
a stable series it also has to be present in all the newer series as
well - so groovy upwards. We are long past debian import freeze so
groovy is still on 4.99.12+dfsg+really4.23.0-1 - and syncing 5.2.0
now, so late in the cycle, requires an approved Feature Freeze
Exception [1], since we're also past feature freeze. It's a very very
unfortunate time, since Final Freeze for groovy is tomorrow, and we're
a bit reluctant with accepting risky pieces. The 'good' news is,
calibre is only seeded in our Ubuntu Studio flavor, so the impact
should be manageable. I would like the Ubuntu Studio flavor
representatives to chip in and (maybe) help out with the paperwork
there.

Once we have it in groovy+ we can look into backporting it to focal.
This requires filling in some SRU paperwork as per the policy [2].
What makes it a bit problematic from the SRU perspective is that the
debdiff between 4.99.4+dfsg+really4.12.0-1build1 (in focal) and 5.2.0
is 7600433 lines long. Looking at the diff itself, most of it are
translation changes, but still... it's quite a change nevertheless.
The things that need to be thought of before performing the backport:
 * How badly is calibre broken on focal right now? Is it really
unusable in its current state? Examples of how broken things are so
that we can understand the situation better
 * How does the automated test coverage on calibre look like? Do all
new features come with unit testing? What about autopkgtests (I don't
think I see any?)?
 * What would be the acceptance criteria for the new version? What
testing should be performed to make sure the new version works as
expected and doesn't regress any existing users (assuming calibre in
focal right now is at least usable to some extent)

If needed, we can help a bit with some of the SRU bits.

Cheers,

[1] https://wiki.ubuntu.com/FreezeExceptionProcess#FeatureFreeze_for_new_upstream_versions
[2] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

On Wed, 14 Oct 2020 at 10:20, Norbert Preining <[hidden email]> wrote:

>
> Dear all,
>
> (please Cc)
>
> I am the Debian maintainer of Calibre, and unfortunately it seems that
> for Focal Ubuntu has pulled a preliminary version of Calibre, which is
> **seriously** broken and unusable, not even starting in most cases.
>
> We were forced by the Python3 transition to temporarily ship pre-release
> versions of Calibre. In particular, Ubuntu Focal ships
>         4.99.4+dfsg+really4.12.0-1build1
> which is version 4.12 with experimental Python3 patches on top of it.
> This worked for a short time being until Calibre 5 was released with
> proper Python3 support.
>
> Due to this unfortunate squeeze in release timing, Ubuntu Focal users
> now have a seriously broken Calibre, and upstream is swamped with bug
> reports.
>
> I would strongly suggest and support, and help preparing, an update to
> Focal based on the current version in Debian/testing, 5.2.0+dfsg-1,
> which has been out since quite some time and field-tested with Python3
> in various environments, due to upstream having switched to Py3, too.
>
> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if yes,
> what kind if steps are necessary?
>
> Note that I am not an Ubuntu developers, but Debian developer and
> maintainer of Calibre.
>
> Thanks and all the best
>
> Norbert
>
> --
> PREINING Norbert                              https://www.preining.info
> Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



--
Ɓukasz 'sil2100' Zemczak
 Foundations Team
 [hidden email]
 www.canonical.com

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Dmitry Shachnev-2
In reply to this post by Norbert Preining-2
Hi Norbert, Lukasz and all!

On Wed, Oct 14, 2020 at 09:06:35AM +0900, Norbert Preining wrote:

> Dear all,
>
> (please Cc)
>
> I am the Debian maintainer of Calibre, and unfortunately it seems that
> for Focal Ubuntu has pulled a preliminary version of Calibre, which is
> **seriously** broken and unusable, not even starting in most cases.
>
> We were forced by the Python3 transition to temporarily ship pre-release
> versions of Calibre. In particular, Ubuntu Focal ships
> 4.99.4+dfsg+really4.12.0-1build1
> which is version 4.12 with experimental Python3 patches on top of it.
> This worked for a short time being until Calibre 5 was released with
> proper Python3 support.
>
> Due to this unfortunate squeeze in release timing, Ubuntu Focal users
> now have a seriously broken Calibre, and upstream is swamped with bug
> reports.
>
> I would strongly suggest and support, and help preparing, an update to
> Focal based on the current version in Debian/testing, 5.2.0+dfsg-1,
> which has been out since quite some time and field-tested with Python3
> in various environments, due to upstream having switched to Py3, too.
Unfortunately, Calibre 5.x requires SIP 5 and PyQt5 that is built against
SIP 5. Moving PyQt5 to the new SIP is a major transition, that happened
in Debian recently (two weeks ago) and did not happen in Ubuntu yet because
of freeze.

This requires changes in many packages simultaneously: at least pyqt5,
pyqt5charts, pyqt5webengine, qscintilla2, calibre, python-poppler-qt5,
veusz, krita and qgis.

I am planning to land this change early in Groovy+1 cycle.

> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if yes,
> what kind if steps are necessary?

So for Focal and Groovy we need a version of Calibre that still uses SIP 4.
Last such version in Debian was 4.99.12+dfsg+really4.23.0-1, Groovy already
has that. If you know some specific fixes, maybe they can be applied on top
of what Focal or Groovy has.

--
Dmitry Shachnev

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Eli Schwartz
On 10/14/20 7:29 AM, Dmitry Shachnev wrote:

> Hi Norbert, Lukasz and all!
>
> Unfortunately, Calibre 5.x requires SIP 5 and PyQt5 that is built
> against SIP 5. Moving PyQt5 to the new SIP is a major transition,
> that happened in Debian recently (two weeks ago) and did not happen
> in Ubuntu yet because of freeze.
>
> This requires changes in many packages simultaneously: at least
> pyqt5, pyqt5charts, pyqt5webengine, qscintilla2, calibre,
> python-poppler-qt5, veusz, krita and qgis.
>
> I am planning to land this change early in Groovy+1 cycle.
Just for the record, this is untrue.

Arch Linux has built python3 PyQt5 using Sip 5 via /usr/bin/sip-build,
since Dec 13 18:01:34 2019 while continuing to build python2 PyQt5 using
Sip 4 via python2 configure.py

This worked well enough that even though on Nov 24 20:33:38 2019 I began
shipping multiple repository packages for calibre -- one "calibre"
package built with python2 and one "calibre-python3" package built with
python3 -- both using Sip 4, they worked fine. The calibre-python3
package was, as expected, buggy due to being beta quality, but it never
failed due to pyqt5/sip itself.

There are still various packages in our distro archives building with
Sip 4 but successfully using the pyqt5 bindings built using Sip 5.

Old versions of some of those packages (at least krita, qgis) did need
patches to change the location of the sip dir.
qscintilla2 did need to be rebuilt with no changes, then a week later
the 2.11.4 update moved to sip-build.

It's plainly possible to mix them at least a little. From memory, we did
not even need to rebuild (most of) the packages.

YMMV, but it should definitely be feasible to update pyqt5/sip5, test
everything that build-depends on them, and leave many of them alone if
they're not ready to move.

>> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if
>> yes, what kind if steps are necessary?
>
> So for Focal and Groovy we need a version of Calibre that still uses
> SIP 4. Last such version in Debian was 4.99.12+dfsg+really4.23.0-1,
> Groovy already has that. If you know some specific fixes, maybe they
> can be applied on top of what Focal or Groovy has.

Even if you could solve the immediate, critical crashes, calibre +
python3 is a combination that is only beta quality until 5.0.0, as many
issues were found during the final stages of the beta.

I strongly advise Ubuntu Focal to NOT ship beta-quality code that
upstream never released and refuses to sign off on, for the next 5 years
as an LTS. It is a recipe for heartbreak and bad relationships with
upstream.
It was never intended to be present in a stable, frozen distro release
anywhere.

In recent times, the Debian package for calibre has gained a lot of
polish, and gotten rid of legacy baggage from the "bad old days" where
upstream simply told all users that Debian was the main reason users
should never use distro packages. I'm kind of hoping that Ubuntu doesn't
become a new reason for upstream to begrudge distros.

...

Possible options going forward:

very much not ideal:

- reinstate a stable legacy python2 build for Ubuntu (needs various
  python2 deps re-added)
- dropping calibre from the archives
- doing nothing: keep a broken package that crashes on startup and
  drives upstream nuts
- cherry-picking some specific fix: keep a partially broken beta package
  that crashes at odd, unpredictable times, and drives upstream
  occasionally nuts

ideal:

- getting a Sip 5 collection in place which still supports some packages
  stuck on Sip 4, and package calibre 5.2.0

not great, but workable:

- Revert code in calibre to make it build with Sip 4 again, and package
  calibre 5.2.0:

https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025


...

I suspect that last option would be the easiest resolution. It should work.

--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Dmitry Shachnev-2
Hi Eli!

On Wed, Oct 14, 2020 at 11:52:44AM -0400, Eli Schwartz wrote:

> > This requires changes in many packages simultaneously: at least
> > pyqt5, pyqt5charts, pyqt5webengine, qscintilla2, calibre,
> > python-poppler-qt5, veusz, krita and qgis.
> >
> > I am planning to land this change early in Groovy+1 cycle.
>
> Just for the record, this is untrue.
>
> Arch Linux has built python3 PyQt5 using Sip 5 via /usr/bin/sip-build,
> since Dec 13 18:01:34 2019 while continuing to build python2 PyQt5 using
> Sip 4 via python2 configure.py
>
> This worked well enough that even though on Nov 24 20:33:38 2019 I began
> shipping multiple repository packages for calibre -- one "calibre"
> package built with python2 and one "calibre-python3" package built with
> python3 -- both using Sip 4, they worked fine. The calibre-python3
> package was, as expected, buggy due to being beta quality, but it never
> failed due to pyqt5/sip itself.
>
> There are still various packages in our distro archives building with
> Sip 4 but successfully using the pyqt5 bindings built using Sip 5.
>
> Old versions of some of those packages (at least krita, qgis) did need
> patches to change the location of the sip dir.
> qscintilla2 did need to be rebuilt with no changes, then a week later
> the 2.11.4 update moved to sip-build.
>
> It's plainly possible to mix them at least a little. From memory, we did
> not even need to rebuild (most of) the packages.

Most packages I mentioned will need a rebuild and a patch to use a different
location for PyQt5 *.sip files.

You can see this in Debian: after I uploaded new PyQt5 (built with SIP 5) to
unstable, immediately some packages started to fail to build from source:

- https://bugs.debian.org/971173 (krita)
- https://bugs.debian.org/971217 (python-poppler-qt5)
- https://bugs.debian.org/971172 (veusz)

And packages that were not rebuilt got runtime errors:

- https://bugs.debian.org/971538 (qgis)
- https://bugs.debian.org/970921 (calibre)

So what I said is true: this requires simultaneous changes in many packages,
even if it's just a rebuild for some of them.

> YMMV, but it should definitely be feasible to update pyqt5/sip5, test
> everything that build-depends on them, and leave many of them alone if
> they're not ready to move.

Tomorrow is final freeze, so it is really bad timing for this.

> not great, but workable:
>
> - Revert code in calibre to make it build with Sip 4 again, and package
>   calibre 5.2.0:
>
> https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025
>
> I suspect that last option would be the easiest resolution. It should work.

That would be the easiest option, yes.

I don't volunteer to work on this (no time, sorry), but I can sponsor an
upload if someone gets a feature freeze exception for this and prepares/tests
the upload.

--
Dmitry Shachnev

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Norbert Preining-2
Hi Lukasz, Dmitry, Eli, all

thanks for your messages and interest and support. Sorry for the late
replies, time zones (I am Japan based).

On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
> With my SRU team hat on, after reading what you said about the status
> of current calibre, I would say it is possible to update the package
> version in focal to 5.2.0. But we would need to know a bit more, and
> there is some (important) paperwork to do.

Sounds like a good idea.

On Wed, 14 Oct 2020, Dmitry Shachnev wrote:
> Tomorrow is final freeze, so it is really bad timing for this.

Indeed.

> > - Revert code in calibre to make it build with Sip 4 again, and package
> >   calibre 5.2.0:
> >
> > https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025
> >
> > I suspect that last option would be the easiest resolution. It should work.
>
> That would be the easiest option, yes.
>
> I don't volunteer to work on this (no time, sorry), but I can sponsor an
> upload if someone gets a feature freeze exception for this and prepares/tests
> the upload.

I think I can prepare a package based on the Debian calibre 5.2.0
version in sid/testing, that reverts the Sip5 changes.

I will install a focal vm and see if building works out of the box.

If that is done, I will contact you again for further steps.

Is this ok a plan for you or do you want/need anything else in this
area right now?

All the best

Norbert

--
PREINING Norbert                              https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Norbert Preining-2
In reply to this post by Lukasz Zemczak
Hi Lukasz,

so, here are a few more answers to your questions:

On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
>  * How badly is calibre broken on focal right now? Is it really
> unusable in its current state? Examples of how broken things are so
> that we can understand the situation better

Yes. I just installed a vm, updated to the latest version of the
packages of focal, installed calibre, started calibre, and got

norbert@ubuntu2004-vm:~$ calibre
Traceback (most recent call last):
  File "/usr/bin/calibre", line 20, in <module>
    sys.exit(calibre())
  File "/usr/lib/calibre/calibre/gui_launch.py", line 73, in calibre
    main(args)
  File "/usr/lib/calibre/calibre/gui2/main.py", line 543, in main
    listener = create_listener()
  File "/usr/lib/calibre/calibre/gui2/main.py", line 514, in create_listener
    return Listener(address=gui_socket_address())
  File "/usr/lib/calibre/calibre/utils/ipc/server.py", line 110, in __init__
    self._listener._unlink.cancel()
AttributeError: 'NoneType' object has no attribute 'cancel'
norbert@ubuntu2004-vm:~$ vim

So well, it is completely useless.

This can also be seen by the list of upstream bugs reported
https://bugs.launchpad.net/bugs/1899700
https://bugs.launchpad.net/bugs/1899674
https://bugs.launchpad.net/bugs/1899355
https://bugs.launchpad.net/bugs/1899035
https://bugs.launchpad.net/bugs/1899029
https://bugs.launchpad.net/bugs/1898940
https://bugs.launchpad.net/bugs/1898904


>  * How does the automated test coverage on calibre look like? Do all
> new features come with unit testing? What about autopkgtests (I don't
> think I see any?)?

There are not autopkgtests, but there is an extensive test suite built
into calibre.

>  * What would be the acceptance criteria for the new version? What
> testing should be performed to make sure the new version works as
> expected and doesn't regress any existing users (assuming calibre in
> focal right now is at least usable to some extent)

Since it does not even start, I guess there is no regression for
focal users, only for those upgrading from a previous release.


Together with YOKOTA Hiroshi (in Cc), who has done most of the work on
recent packaging, I have prepared a version for focal (SIP4, debhelper
12), built it on my focal machine, and successfully run it. Source and
amd64 packages are available here:

        deb http://www.preining.info/debian focal main
        deb-src http://www.preining.info/debian focal main

(signed with my gpg key https://www.preining.info/rsa.asc)

What are the next steps you are expecting from me?

- prepare a package for groovy and separately for focal?
- what are the version numbers you want to see?
- how should we proceed?

Best

Norbert

--
PREINING Norbert                              https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu Focal update of broken Calibre package

Sai Vinoba
In reply to this post by Norbert Preining-2
On Thu, 15 Oct 2020 08:35:36 +0900, Norbert Preining <[hidden email]> wrote:

> Hi Lukasz, Dmitry, Eli, all
>
> thanks for your messages and interest and support. Sorry for the late
> replies, time zones (I am Japan based).
>
> On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
> > With my SRU team hat on, after reading what you said about the status
> > of current calibre, I would say it is possible to update the package
> > version in focal to 5.2.0. But we would need to know a bit more, and
> > there is some (important) paperwork to do.
>
> Sounds like a good idea.
>
> On Wed, 14 Oct 2020, Dmitry Shachnev wrote:
> > Tomorrow is final freeze, so it is really bad timing for this.
>
> Indeed.
>
> > > - Revert code in calibre to make it build with Sip 4 again, and package
> > >   calibre 5.2.0:
> > >
> > > https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025
> > >
> > > I suspect that last option would be the easiest resolution. It should work.
> >
> > That would be the easiest option, yes.
> >
> > I don't volunteer to work on this (no time, sorry), but I can sponsor an
> > upload if someone gets a feature freeze exception for this and prepares/tests
> > the upload.
>
> I think I can prepare a package based on the Debian calibre 5.2.0
> version in sid/testing, that reverts the Sip5 changes.
>
> I will install a focal vm and see if building works out of the box.
>
> If that is done, I will contact you again for further steps.
>
> Is this ok a plan for you or do you want/need anything else in this
> area right now?
>
> All the best
>
> Norbert
>

Hi all,
I had sent an email to ubuntu-devel-discuss (instead of here) and not sure if any of you read. Reposting here just to be sure.

I checked Calibre in Groovy (version 4.99.12+dfsg+really4.23.0-1) and it works fine. As per bug report #1898904 (https://bugs.launchpad.net/ubuntu/+source/calibre/+bug/1898904/comments/7), Calibre >=4.15 has the required patch. So, instead of waiting for 5.2, can we backport current Groovy version of Calibre to Focal?


Regards,

Sai

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel