Backports and a Proposal for Changing the Process

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Backports and a Proposal for Changing the Process

Thomas Ward-2

Hello.

Back in November 2018 [1] and subsequent requests since [2] [3], I made several
attempts to reach out to the Backports team on public lists to volunteer my
time and energy to work on the Backports queue and to clear them out, and to
get more information on what the team requires for someone to help with the
backports process.

With no public responses to my messages, the third message [3] I sent included
direct messages to the admins.  I had received only one response off-list, and
it was suggested through there that:

 1. The backports process in general is broken.

 2. Putting the burden on a tiny number of admins to do checking and uploading
    does not scale to the volume of requests

 3. Given points #1 and #2 my request to assist was not likely to be actioned
    on.

It was also stated that because the current process is broken, it no longer is
capable of being considered a valid approach to backports.

In ongoing research into all backport requests currently in the queue including
for now-EOL releases (bugs in the Backports targets projects that have yet to
be closed or acted on at all), I have come to the following conclusions:

 1. That the current process is indeed broken for Backports, concurring with
 the private opinions provided to me.  (With some smaller exceptions for
 certain packages, such as cockpit as an example, where developers directly
 handle their own requests.)

 2. That the current process for backporting things is not feasible for
 scaling, also concurring with the private opinions provided to me.

 3. That the vast overwhelming majority of backports requests are made by
 mostly 'random users' or users who are not 100% familiar with requirements
 for backporting or package versions to be included (including build and
 testing requirements).

 4. That the vast majority of requests do not have ample justifications tied
 to them other than "I want this version available"

In separate discussions with interested parties and DMB members off-list and
in IRC discussions privately, it was suggested that if the current process is
broken, we need to redesign it.

I am proposing that we change the process and how it's handled in a substantial
way, redefining the requirements for Backports and the process to be more
efficient.


This proposal makes some key changes to the processes, in that:

 1. It moves uploading of backports from the small backporters team to the
 larger developer body, and removes entirely the "Please backport xxx" class
 of requests.

 2. It will retain the Backporters team's ability to review backports, and
 ultimately approve backports so they can be moved from the Upload queue into
 the specific repositories, in a manner similar to how the SRU team processes
 the SRU queue, but further restricts the requirements to require dedicated
 Ubuntu Developer (Team or Individual) backing.

 3. This retains the Backporters' privilege of review before releasing things
 into the -backports repository, in a manner similar to how SRUs would behave.
 (This might require some retooling behind the scenes for the upload queue
 though...)

While this proposal changes how things're delegated and handled, this also
handles the problem of "too many requests" - if a request can only move forward
with an individual or team backing it and they're an established developer,
that solves many of the problems that the current system has shown.  This also
means that the number of things that needs actual approval from the Backports
team is limited, as well as the fact that the 'random backport request' won't
go through if it's otherwise not getting support from an established developer.

Please also note that while this proposal touches base on how I would envision
the process to operate, I welcome *all opinions and refinements to this
proposal and process*, as I intend this to only be the 'beginning' of a more
refined proposal or process declaration for changing the Backports process.


The following is my proposal in detail:


Backporting Requirements
------------------------

The uploading of backports is now to be performed by Ubuntu developers. The ~motu team
can upload any backport, and we will request the DMB to extend PPU rights to apply to
all backports pockets too.

 1) Developers file requests in the regular Ubuntu project, try their best to
 ensure that the backport is good (build/install/run test, post in the bug
 confirming this has been done), and upload it to the queue. The "Continued
 Functionality of Reverse-Dependencies" requirement from [0] is to be relaxed.
 Each and every reverse dependency need not be tested; the uploader should use
 their judgement, which the backports team will need to concur with. This requirement
 has been responsbile for making it practically impossible to backport anything complex.
 
 1.5) People who need sponsoring can perform step 1) and subscribe
 ~ubuntu-sponsors *once there is a debdiff/package on the bug to upload*.
 
 2) The ~ubuntu-backporters team reviews uploads from the queue and accepts/rejects,
 in much the same way as the SRU team does currently.
 
 3) Any changes to the package which are required for backporting must be minor
 in nature or otherwise required, and will be reviewed specifically by the
 Backporters Team prior to backport acceptance.

 4) A backport request must specify the SOURCE release to backport a package
 from and the TARGET release being backpoted to.

 5) Both SOURCE and TARGET releases must be a currently-supported release of
 Ubuntu.  Requests to backport to or from EOL releases will be rejected.

------

An additional point of discussion is that if we can remove 'random end user
requesting' from the equation, that would also lighten up the backports
queues. When an end user wants to make a request, they should field the
request via [hidden email] or a similar mailing list for
community input on the request; if a developer or sponsor wants to assist
for that request, then with the revisions to the process above, the backport
process can move forward without major impact to the queue and without
adding additional major stressors to the Backporters team, as those with
upload rights would already be able to do the 'upload' steps.

------

Opinions, alterations, recommendations, changes, etc. are all welcome, as I
stated I intend this to more or less spark further discussion rather than be
accepted currently verbatim.

I hope that this will ultimately lead to improving the Backports process, as it
currently is defunct for the most part and does not (and as it currently is,
will not) be able to continue to process new Backport requests without radical
changes to the process.

(Note that parts of this proposal have had contributions from Iain Lane,
Robie Basak, and Seth Arnold, at my request, prior to my submitting this,
in order to better phrase and construct this proposal in an acceptable manner.)

------
Thomas Ward
Ubuntu Server Team Member
LP: ~teward

[0]: https://wiki.ubuntu.com/UbuntuBackports#Continued_Functionality_of_Reverse-Dependencies

[1]: https://lists.ubuntu.com/archives/ubuntu-devel/2018-November/040549.html

[2]: https://lists.ubuntu.com/archives/ubuntu-devel/2018-December/040563.html

[3]: https://lists.ubuntu.com/archives/ubuntu-devel/2018-December/040569.html


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Backports and a Proposal for Changing the Process

Robie Basak-4
Hi Thomas,

Thank you for driving this, and sorry for the slow reply.

I'm in general agreement with your proposal. In particular, given that
the existing backporters team members haven't been able to find the time
to review and/or adjust the process recently, I'm in favour of asking
them to hand the reins to others who can, and I consider you able.

I also think there's little to lose in relaxing the backports rules to
encourage more to land right now, even if that increases regression
risk, given that the alternative is that backporting activity grinds to
a halt and the pocket remains unused.

Finally, there's the obvious question of snaps. I think that snaps are
much easier to manage for very similar use cases here. For example I was
able to snap certbot relatively easily, but am still working (with many
others) on getting the certbot SRU landed, which from a technical work
perspective is essentially the same task as a backport. I would still
encourage those interested in backports to look into snaps first.

However, I see no reason that this should block a bunch of volunteers
who prefer instead to continue with the traditional backports process
from doing so. We rely upon and value those who volunteer their time,
and I don't think it's up to anyone else to dictate where they choose to
spend their efforts. An exception might be if there were some technical
collision that forces us to choose, but I see no such collision here. As
long as we don't draw others into the deb backports process who would
genuinely prefer the snap process.

Minor comments on some aspects of your proposal below.

On Mon, Jan 14, 2019 at 04:39:32PM -0500, Thomas Ward wrote:
> Backporting Requirements
> ------------------------
>
> The uploading of backports is now to be performed by Ubuntu developers.
> The ~motu team
> can upload any backport, and we will request the DMB to extend PPU
> rights to apply to
> all backports pockets too.

I suggest that it might be easier and cleaner to simply say "anyone who
can upload a given package to the development release can also upload
its backport", at least to start with. Assuming we can configure
Launchpad to allow this.

>  1) Developers file requests in the regular Ubuntu project, try their
> best to
>  ensure that the backport is good (build/install/run test, post in the bug
>  confirming this has been done), and upload it to the queue. The "Continued
>  Functionality of Reverse-Dependencies" requirement from [0] is to be
> relaxed.
>  Each and every reverse dependency need not be tested; the uploader
> should use
>  their judgement, which the backports team will need to concur with.
> This requirement
>  has been responsbile for making it practically impossible to backport
> anything complex.
Perhaps ask for it to be demonstrated working in some (driver-defined)
PPA and tested from there? Then there'd be a nice progressive procedure
to get a backport landed - so interested parties could be using a PPA
before "graduating" the package to the official backports pocket.

We might still need to use the backports project though, to avoid
collisions with SRU bug tasks.

>  
>  1.5) People who need sponsoring can perform step 1) and subscribe
>  ~ubuntu-sponsors *once there is a debdiff/package on the bug to upload*.
>  
>  2) The ~ubuntu-backporters team reviews uploads from the queue and
> accepts/rejects,
>  in much the same way as the SRU team does currently.

I suggest that we discuss and then define specifically what aspects of
the upload are the responsibility of the uploader and the backporter
team isn't expected to review. If we agree to cut down on what the
backporter team has to check, then hopefully that will help with
velocity.

>  3) Any changes to the package which are required for backporting must
> be minor
>  in nature or otherwise required, and will be reviewed specifically by the
>  Backporters Team prior to backport acceptance.

I suggest we define how to communicate these aspects with the
backporters team, so they don't have to rediscover the changes by a
potentially more time consuming review.

> ------
>
> An additional point of discussion is that if we can remove 'random end user
> requesting' from the equation, that would also lighten up the backports
> queues. When an end user wants to make a request, they should field the
> request via [hidden email] or a similar mailing list for
> community input on the request; if a developer or sponsor wants to assist
> for that request, then with the revisions to the process above, the
> backport
> process can move forward without major impact to the queue and without
> adding additional major stressors to the Backporters team, as those with
> upload rights would already be able to do the 'upload' steps.
Perhaps reuse one of the existing lists, like devel-discuss? One thing
to be careful of here though is setting expectations - we need to make
sure that users don't think that backports will magically appear if they
are requested.

Robie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Backports and a Proposal for Changing the Process

Thomas Ward-2

My apologies for a slower reply, been messing with my mail gateways and had to deal with that first.

On 2/7/19 11:01 AM, Robie Basak wrote:
Minor comments on some aspects of your proposal below.

On Mon, Jan 14, 2019 at 04:39:32PM -0500, Thomas Ward wrote:
Backporting Requirements
------------------------

The uploading of backports is now to be performed by Ubuntu developers.
The ~motu team
can upload any backport, and we will request the DMB to extend PPU
rights to apply to
all backports pockets too.
I suggest that it might be easier and cleaner to simply say "anyone who
can upload a given package to the development release can also upload
its backport", at least to start with. Assuming we can configure
Launchpad to allow this.
I am not in disagreement with this.
 1) Developers file requests in the regular Ubuntu project, try their
best to
 ensure that the backport is good (build/install/run test, post in the bug
 confirming this has been done), and upload it to the queue. The "Continued
 Functionality of Reverse-Dependencies" requirement from [0] is to be
relaxed.
 Each and every reverse dependency need not be tested; the uploader
should use
 their judgement, which the backports team will need to concur with.
This requirement
 has been responsbile for making it practically impossible to backport
anything complex.
Perhaps ask for it to be demonstrated working in some (driver-defined)
PPA and tested from there? Then there'd be a nice progressive procedure
to get a backport landed - so interested parties could be using a PPA
before "graduating" the package to the official backports pocket.

We might still need to use the backports project though, to avoid
collisions with SRU bug tasks.


Driver-defined PPA is going to be a little trickier, I think, here, Robie.

Assuming you mean that it'd need uploaded to a 'proposed' PPA like the Security Team has for landing patches, then that'd be locked for uploads to just the Backporters team - we could sponsor/stage such uploads to the PPA, though, but then we're back where we currently started - too many things to upload, not enough manpower to drive it.

Even if I were driving this, it'd turn with burnout eventually.

 
 1.5) People who need sponsoring can perform step 1) and subscribe
 ~ubuntu-sponsors *once there is a debdiff/package on the bug to upload*.
 
 2) The ~ubuntu-backporters team reviews uploads from the queue and
accepts/rejects,
 in much the same way as the SRU team does currently.
I suggest that we discuss and then define specifically what aspects of
the upload are the responsibility of the uploader and the backporter
team isn't expected to review. If we agree to cut down on what the
backporter team has to check, then hopefully that will help with
velocity.

 3) Any changes to the package which are required for backporting must
be minor
 in nature or otherwise required, and will be reviewed specifically by the
 Backporters Team prior to backport acceptance.
I suggest we define how to communicate these aspects with the
backporters team, so they don't have to rediscover the changes by a
potentially more time consuming review.

This can be a requirement during the backport process.  During a backport request, even in the 'current' broken process, we need to make notes about the package and whether any changes are required for building in the backported release.  Typically, these'd be determined by the person seeking the backport via PPA testing, etc. and then defined in the backport request.

We can possibly do a similar thing here - if no 'changes' are defined and the backport FTBFS in a PPA (such as a staging/proposed PPA like described above or like the Security team has), then reject the backport as-is because it fails to build.  Any requests that require changes that aren't defined could be rejected because of FTBFS due to undefined changes necessary to get the package to build.  Putting the requirement on defining changes to the individuals requesting/testing would offload that from the Backporters team.

This would also be part of the requests process - proof has to be made that the package builds and works in the target backported release with the Backports repo enabled (which as far as I can tell PPAs can be configured to do so).

------

An additional point of discussion is that if we can remove 'random end user
requesting' from the equation, that would also lighten up the backports
queues. When an end user wants to make a request, they should field the
request via [hidden email] or a similar mailing list for
community input on the request; if a developer or sponsor wants to assist
for that request, then with the revisions to the process above, the
backport
process can move forward without major impact to the queue and without
adding additional major stressors to the Backporters team, as those with
upload rights would already be able to do the 'upload' steps.
Perhaps reuse one of the existing lists, like devel-discuss? One thing
to be careful of here though is setting expectations - we need to make
sure that users don't think that backports will magically appear if they
are requested.

This would work on devel-discuss, in my opinion, but as you said we need to be careful of setting expectations

This said, we can also define the process and make a note in nice bold text on the Wiki somewhere that "even if you request a backport, there is no guarantee that it will actually be backported - backports require someone to drive the individual backport itself and if nobody steps up to back the request and do the required legwork per this process to get the backport reviewed and ready for inclusion, the request will likely go unfulfilled."



Thomas


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel