[jak@debian.org: Frontend locking in APT clients]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jak@debian.org: Frontend locking in APT clients]

Julian Andres Klode
Forwarding this to ubuntu-devel.

----- Forwarded message from Julian Andres Klode <[hidden email]> -----

Date: Mon, 18 Jun 2018 20:19:17 +0200
From: Julian Andres Klode <[hidden email]>
To: [hidden email], [hidden email]
Subject: Frontend locking in APT clients
Message-ID: <[hidden email]>
Accept-Language: de-DE, de, en-GB, en-US, en
User-Agent: NeoMutt/20180512

Hi folks,

With frontend locking in dpkg git, I think it's time I clear up
some potential confusion as to how this is supposed to work in the
APT world.

The idea is that the current _system->Lock() / apt_pkg.SystemLock
/ apt_pkg.pkgsystem_lock() will start to manage _both_ lock-frontend
and lock, and new methods LockInner() and UnlockInner() will be
provided to release "lock". Code running dpkg will need to call
those around dpkg runs, rather than the generic Lock ones.

python-apt is currently broken in that you need to release the lock
prior to calling commit() on an apt.Cache. This will change soon
- no unlocking will be needed. A python-apt (>> 1.7~alpha1~) client
should behave as following:

Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
  with apt_pkg.SystemLock():
    main()
- forget about locks if you don't invoke dpkg manually - do not
release that, ever. If you do run dpkg manually do it like this:

  with apt_pkg.NoInnerLock():
    subprocess.check_call(["dpkg", "--configure", "-a"])

or instead of context managers, use the functions
pkgsystem_{un,}lock{,_inner}.

This will ensure that your apt client will be safe against
any other apt client, and any other client implementing frontend
locking. It will not be safe against other frontends that acquire
the dpkg lock themselves, those will need fixing too. It will however,
be safe against concurrent runs of dpkg by users or frontends not
implementing locking.

Thanks,
Julian
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

----- End forwarded message -----

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: [jak@debian.org: Frontend locking in APT clients]

Julian Andres Klode
Regarding stable releases. I plan to backport the dpkg, apt, and
python-apt changes to bionic and xenial. It will be worth it, and
together with a fix in u-u and aptdaemon and packagekit, we will
be looking at I guess 99% of issues with lock races fixed.

While I'm at it, I want to note that apt in its current merge request state,
automatically sets DPKG_FRONTEND_LOCKED when invoking dpkg
if the system lock is acquired (after the fork, directly
before the execvp). If you run dpkg yourself, like for calling
dpkg --configure -a you will probably have to set the environment
variable yourself.


On Mon, Jun 18, 2018 at 08:20:04PM +0200, Julian Andres Klode wrote:

> Forwarding this to ubuntu-devel.
>
> ----- Forwarded message from Julian Andres Klode <[hidden email]> -----
>
> Date: Mon, 18 Jun 2018 20:19:17 +0200
> From: Julian Andres Klode <[hidden email]>
> To: [hidden email], [hidden email]
> Subject: Frontend locking in APT clients
> Message-ID: <[hidden email]>
> Accept-Language: de-DE, de, en-GB, en-US, en
> User-Agent: NeoMutt/20180512
>
> Hi folks,
>
> With frontend locking in dpkg git, I think it's time I clear up
> some potential confusion as to how this is supposed to work in the
> APT world.
>
> The idea is that the current _system->Lock() / apt_pkg.SystemLock
> / apt_pkg.pkgsystem_lock() will start to manage _both_ lock-frontend
> and lock, and new methods LockInner() and UnlockInner() will be
> provided to release "lock". Code running dpkg will need to call
> those around dpkg runs, rather than the generic Lock ones.
>
> python-apt is currently broken in that you need to release the lock
> prior to calling commit() on an apt.Cache. This will change soon
> - no unlocking will be needed. A python-apt (>> 1.7~alpha1~) client
> should behave as following:
>
> Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
>   with apt_pkg.SystemLock():
>     main()
> - forget about locks if you don't invoke dpkg manually - do not
> release that, ever. If you do run dpkg manually do it like this:
>
>   with apt_pkg.NoInnerLock():
>     subprocess.check_call(["dpkg", "--configure", "-a"])
>
> or instead of context managers, use the functions
> pkgsystem_{un,}lock{,_inner}.
>
> This will ensure that your apt client will be safe against
> any other apt client, and any other client implementing frontend
> locking. It will not be safe against other frontends that acquire
> the dpkg lock themselves, those will need fixing too. It will however,
> be safe against concurrent runs of dpkg by users or frontends not
> implementing locking.
>
> Thanks,
> Julian
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer                              i speak de, en
>
> ----- End forwarded message -----
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer                              i speak de, en

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: [jak@debian.org: Frontend locking in APT clients]

Seth Arnold
On Mon, Jun 18, 2018 at 08:33:17PM +0200, Julian Andres Klode wrote:
> automatically sets DPKG_FRONTEND_LOCKED when invoking dpkg
> if the system lock is acquired (after the fork, directly
> before the execvp). If you run dpkg yourself, like for calling
> dpkg --configure -a you will probably have to set the environment
> variable yourself.

Hi Julian, I think I'm mostly excited about the lock changes you've
described (to the extent I recall previous discussions with you, this
has the potential to reduce some seriously confusing situations).

But dpkg --configure -a is already deep into the "unhappy user" territory
and needing to remember an additional environment variable to make it
work correctly is going to add an additional level of frustration.

Is there no way to make dpkg --configure -a work correctly without
manually setting environment variables?

Thanks

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [jak@debian.org: Frontend locking in APT clients]

Philipp Kern-6
In reply to this post by Julian Andres Klode
On 2018-06-18 14:20, Julian Andres Klode wrote:
> Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
>   with apt_pkg.SystemLock():
>     main()

Will this also allow to wait on the lock?

Kind regards
Philipp Kern

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: [jak@debian.org: Frontend locking in APT clients]

Julian Andres Klode
On Mon, Jun 18, 2018 at 08:58:50PM -0400, Philipp Kern wrote:
> On 2018-06-18 14:20, Julian Andres Klode wrote:
> > Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
> >   with apt_pkg.SystemLock():
> >     main()
>
> Will this also allow to wait on the lock?

No. mvo pushed https://salsa.debian.org/apt-team/apt/merge_requests/6
to implement waiting (in the src:apt frontends), but that's just
busy waiting.

For our own frontends like that we could implement non-busy waiting
using LOCK_EX without LOCK_NB and setting a SIGALRM to cancel the
lock attempt after a timeout[1]. I don't really think that's reasonable
for a library to do, though.

[1] Compare https://code.launchpad.net/~juliank/apport/lp1746874/+merge/337558
    for the idea
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: [jak@debian.org: Frontend locking in APT clients]

Julian Andres Klode
In reply to this post by Seth Arnold
On Mon, Jun 18, 2018 at 01:58:24PM -0700, Seth Arnold wrote:

> On Mon, Jun 18, 2018 at 08:33:17PM +0200, Julian Andres Klode wrote:
> > automatically sets DPKG_FRONTEND_LOCKED when invoking dpkg
> > if the system lock is acquired (after the fork, directly
> > before the execvp). If you run dpkg yourself, like for calling
> > dpkg --configure -a you will probably have to set the environment
> > variable yourself.
>
> Hi Julian, I think I'm mostly excited about the lock changes you've
> described (to the extent I recall previous discussions with you, this
> has the potential to reduce some seriously confusing situations).
>
> But dpkg --configure -a is already deep into the "unhappy user" territory
> and needing to remember an additional environment variable to make it
> work correctly is going to add an additional level of frustration.
>
> Is there no way to make dpkg --configure -a work correctly without
> manually setting environment variables?
It works just fine. "You" in this case meant "an apt frontend holding
the lock". As long as you don't hold the lock (I mean, you probably
could, even in a shell), it would be wrong.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment