Releasing Alphas and Betas without "freezing"

[Rick Spencer-2]

Hello all,

At UDS I had some "hallway discussions" about why we freeze for Alphas
and Betas, and the fact that I think it is time to drop this practice
and rather focus on making Ubuntu good quality each day. Sadly, there
was no session on this, thus this email to this list for discussion.

I think it is time drop our "Freeze" practices for the alphas and
betas. Here is my reasoning:

1. We are developing tools that allow us to efficiently use -proposed
in a way that will ensure we will not have partially built or
incompatible components in the release pocket ... ever. Including days
we release Alphas and Betas:

These blueprints tools to ensure that Ubuntu is not uninstallable or
have other problems due to partially built components and such:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary
https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-proposed

I have been assured that the tools necessary to automate the work of
moving components correctly from -proposed to the release will be
ready before Alpha 2.

2. We are investing heavily in the daily quality of Ubuntu. For example ...
We run the same automated tests on an alpha as we run on a daily:
https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/

We tend to archive issues each day:
http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html

We ran all the manual ISO tests *before* we released Alpha 1, and we
have the capability of doing this at will:
http://iso.qa.ubuntu.com/qatracker/milestones/221/builds

In short, freezing the archive before an alpha or beta should not
actually be contributing to either ensuring the installability of
Ubuntu images or ensuring the quality of these images. This implies,
therefore, that all the work around freezing, and all the productivity
lost during a freeze, actually subtracts from the quality of Ubuntu by
reducing our overall velocity for both features and bug fixes, since
every day the image is good quality, and Alpha or Beta should be just
that day's image tagged appropriately.

AIUI, A1 was delivered in such a manner, though without the tooling to
ensure that moving from -proposed to the release pocket was efficient
and automated.

Cheers, Rick

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Stéphane Graber-2]

On 06/15/2012 10:12 AM, Rick Spencer wrote:

> Hello all,
>
> At UDS I had some "hallway discussions" about why we freeze for Alphas
> and Betas, and the fact that I think it is time to drop this practice
> and rather focus on making Ubuntu good quality each day. Sadly, there
> was no session on this, thus this email to this list for discussion.
>
> I think it is time drop our "Freeze" practices for the alphas and
> betas. Here is my reasoning:
>
> 1. We are developing tools that allow us to efficiently use -proposed
> in a way that will ensure we will not have partially built or
> incompatible components in the release pocket ... ever. Including days
> we release Alphas and Betas:
>
> These blueprints tools to ensure that Ubuntu is not uninstallable or
> have other problems due to partially built components and such:
> https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary
> https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-proposed
>
> I have been assured that the tools necessary to automate the work of
> moving components correctly from -proposed to the release will be
> ready before Alpha 2.
>
> 2. We are investing heavily in the daily quality of Ubuntu. For example ...
> We run the same automated tests on an alpha as we run on a daily:
> https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/
>
> We tend to archive issues each day:
> http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html
>
> We ran all the manual ISO tests *before* we released Alpha 1, and we
> have the capability of doing this at will:
> http://iso.qa.ubuntu.com/qatracker/milestones/221/builds
>
> In short, freezing the archive before an alpha or beta should not
> actually be contributing to either ensuring the installability of
> Ubuntu images or ensuring the quality of these images. This implies,
> therefore, that all the work around freezing, and all the productivity
> lost during a freeze, actually subtracts from the quality of Ubuntu by
> reducing our overall velocity for both features and bug fixes, since
> every day the image is good quality, and Alpha or Beta should be just
> that day's image tagged appropriately.
>
> AIUI, A1 was delivered in such a manner, though without the tooling to
> ensure that moving from -proposed to the release pocket was efficient
> and automated.
>
> Cheers, Rick

Hi Rick,

We certainly want to allow people to upload stuff to -proposed during a
milestone week, but I don't agree that we should automatically copy from
-proposed to the release pocket during a milestone week.

We usually try to release all our images with the same versions of the
packages, considering it takes us hours to rebuild everything, having
seeded packages land during that time, will lead to images having
different version of packages.

As for what happened with Alpha 1, we simply asked everyone to upload
their packages to -proposed and then cherry picked the packages we
actually needed for the release from -proposed and copied them into the
release pocket before rebuilding the images (we did that 3 times).


As I understand it, the plan going forward is to have the release pocket
be an alias of -proposed on upload, so that everything always lands into
-proposed.
After something lands in -proposed, is properly built and passes
whatever other criteria we'll have, the package will be automatically
copied to the release pocket.

That last part (copy to the release pocket) would be what we'd block
during a milestone week for any package that's seeded. These would be
copied on a case by case basis by the release team and the images
rebuilt afterwards.

That'd essentially allow any non-seeded package to still flow to the
release pocket and be available for everyone.
All the others will be available for people running with -proposed
enabled or will be available when we manually copy them to the release
pocket or right after we release the milestone and we copy everything
left in -proposed to the release pocket.

--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Rick Spencer-2]

>> Cheers, Rick
>
> Hi Rick,
>
> We certainly want to allow people to upload stuff to -proposed during a
> milestone week, but I don't agree that we should automatically copy from
> -proposed to the release pocket during a milestone week.
>
> We usually try to release all our images with the same versions of the
> packages, considering it takes us hours to rebuild everything, having
> seeded packages land during that time, will lead to images having
> different version of packages.
>
> As for what happened with Alpha 1, we simply asked everyone to upload
> their packages to -proposed and then cherry picked the packages we
> actually needed for the release from -proposed and copied them into the
> release pocket before rebuilding the images (we did that 3 times).

If you have to go in and cherry pick packages to copy over, would it
not make more sense to simply automatically copy everything over?
Everything will be properly built before it is copied over anyway,
right?

>
>
> As I understand it, the plan going forward is to have the release pocket
> be an alias of -proposed on upload, so that everything always lands into
> -proposed.
> After something lands in -proposed, is properly built and passes
> whatever other criteria we'll have, the package will be automatically
> copied to the release pocket.
>
> That last part (copy to the release pocket) would be what we'd block
> during a milestone week for any package that's seeded. These would be
> copied on a case by case basis by the release team and the images
> rebuilt afterwards.

This sounds exactly like a freeze. You upload a package and it doesn't
land in the distro for a week. That's a serious drag on velocity, and
I don't see that it buys us anything but lost productivity and busy
work as I described in my initial mail.

The essential point is, Ubuntu should be good every day. There should
be nothing special about an alpha or beta, it's simply the daily image
on some chosen day. Making them special doesn't buy us anything, but
has costs.

>
> That'd essentially allow any non-seeded package to still flow to the
> release pocket and be available for everyone.
> All the others will be available for people running with -proposed
> enabled or will be available when we manually copy them to the release
> pocket or right after we release the milestone and we copy everything
> left in -proposed to the release pocket.
I thought it was specifically an anti-goal for people to run proposed
during the development release. It would just expose developers to all
the problems that we have without having proposed at all, and
furthermore means that some developers are using proposed, while
others aren't, splitting attention and sewing confusion.

Cheers, Rick

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Friday, June 15, 2012 10:28:55 AM Stéphane Graber wrote:

> On 06/15/2012 10:12 AM, Rick Spencer wrote:
> > Hello all,
> >
> > At UDS I had some "hallway discussions" about why we freeze for Alphas
> > and Betas, and the fact that I think it is time to drop this practice
> > and rather focus on making Ubuntu good quality each day. Sadly, there
> > was no session on this, thus this email to this list for discussion.
> >
> > I think it is time drop our "Freeze" practices for the alphas and
> > betas. Here is my reasoning:
> >
> > 1. We are developing tools that allow us to efficiently use -proposed
> > in a way that will ensure we will not have partially built or
> > incompatible components in the release pocket ... ever. Including days
> > we release Alphas and Betas:
> >
> > These blueprints tools to ensure that Ubuntu is not uninstallable or
> > have other problems due to partially built components and such:
> > https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-interme
> > diary
> > https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-propo
> > sed
> >
> > I have been assured that the tools necessary to automate the work of
> > moving components correctly from -proposed to the release will be
> > ready before Alpha 2.
> >
> > 2. We are investing heavily in the daily quality of Ubuntu. For example
> > ...
> > We run the same automated tests on an alpha as we run on a daily:
> > https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/
> >
> > We tend to archive issues each day:
> > http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html
> >
> > We ran all the manual ISO tests *before* we released Alpha 1, and we
> > have the capability of doing this at will:
> > http://iso.qa.ubuntu.com/qatracker/milestones/221/builds
> >
> > In short, freezing the archive before an alpha or beta should not
> > actually be contributing to either ensuring the installability of
> > Ubuntu images or ensuring the quality of these images. This implies,
> > therefore, that all the work around freezing, and all the productivity
> > lost during a freeze, actually subtracts from the quality of Ubuntu by
> > reducing our overall velocity for both features and bug fixes, since
> > every day the image is good quality, and Alpha or Beta should be just
> > that day's image tagged appropriately.
> >
> > AIUI, A1 was delivered in such a manner, though without the tooling to
> > ensure that moving from -proposed to the release pocket was efficient
> > and automated.
> >
> > Cheers, Rick
>
> Hi Rick,
>
> We certainly want to allow people to upload stuff to -proposed during a
> milestone week, but I don't agree that we should automatically copy from
> -proposed to the release pocket during a milestone week.
>
> We usually try to release all our images with the same versions of the
> packages, considering it takes us hours to rebuild everything, having
> seeded packages land during that time, will lead to images having
> different version of packages.
>
> As for what happened with Alpha 1, we simply asked everyone to upload
> their packages to -proposed and then cherry picked the packages we
> actually needed for the release from -proposed and copied them into the
> release pocket before rebuilding the images (we did that 3 times).
>
>
> As I understand it, the plan going forward is to have the release pocket
> be an alias of -proposed on upload, so that everything always lands into
> -proposed.
> After something lands in -proposed, is properly built and passes
> whatever other criteria we'll have, the package will be automatically
> copied to the release pocket.
>
> That last part (copy to the release pocket) would be what we'd block
> during a milestone week for any package that's seeded. These would be
> copied on a case by case basis by the release team and the images
> rebuilt afterwards.
>
> That'd essentially allow any non-seeded package to still flow to the
> release pocket and be available for everyone.
> All the others will be available for people running with -proposed
> enabled or will be available when we manually copy them to the release
> pocket or right after we release the milestone and we copy everything
> left in -proposed to the release pocket.

This looks complicated.  Maybe it will be easier in practice.

It also looks like a big shift of work from developers to the release team.  
Currently it's up to developers to decide what needs uploading to get to the
milestone.  During a soft freeze there's no action from the release team
except coordination.  With this mechanism, developers can upload whatever and
it's up to the release team to figure out.

I can see this being particularly a problem where a small fix is needed for the
milestone, but the developer's work would naturally flow to a larger update.  
With no freeze and it being up to the release team, as Rick says, developer
flow would continue and the release team would get stuck without good choices
(I'm not sure if in this context there's a mechanism to do direct uploads to
the release pocket?).

In Debian terms I'm seeing release-proposed as like unstable and release as
testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?

Scott K


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Friday, June 15, 2012 04:41:51 PM Rick Spencer wrote:
...
> The essential point is, Ubuntu should be good every day. There should
> be nothing special about an alpha or beta, it's simply the daily image
> on some chosen day. Making them special doesn't buy us anything, but
> has costs.
...
Then cancel the whole milestone process, grab a daily and call it done.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Rick Spencer-2]

On Fri, Jun 15, 2012 at 4:48 PM, Scott Kitterman <[hidden email]> wrote:
> On Friday, June 15, 2012 04:41:51 PM Rick Spencer wrote:
> ...
>> The essential point is, Ubuntu should be good every day. There should
>> be nothing special about an alpha or beta, it's simply the daily image
>> on some chosen day. Making them special doesn't buy us anything, but
>> has costs.
> ...
> Then cancel the whole milestone process, grab a daily and call it done.
Kinda what I'm saying, yeah. I think we probably still need milestones
for targeting bug fixes and features, but we could change that to be
keyed off of months. I think there are also widespread external
expectations that there are alphas and betas in a software project so
I'm not quite ready to advocate for "no alpha and betas" at all.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Friday, June 15, 2012 04:57:57 PM Rick Spencer wrote:
> On Fri, Jun 15, 2012 at 4:48 PM, Scott Kitterman <[hidden email]>
wrote:

> > On Friday, June 15, 2012 04:41:51 PM Rick Spencer wrote:
> > ...
> >
> >> The essential point is, Ubuntu should be good every day. There should
> >> be nothing special about an alpha or beta, it's simply the daily image
> >> on some chosen day. Making them special doesn't buy us anything, but
> >> has costs.
> >
> > ...
> > Then cancel the whole milestone process, grab a daily and call it done.
>
> Kinda what I'm saying, yeah. I think we probably still need milestones
> for targeting bug fixes and features, but we could change that to be
> keyed off of months. I think there are also widespread external
> expectations that there are alphas and betas in a software project so
> I'm not quite ready to advocate for "no alpha and betas" at all.

I don't think you get to have it both ways.  Either we stabilize an image and
put a stamp on it and we need some kind of freeze or we don't.  Trying to let
developers continue to do their work while ignoring the milestone just pushes
the problem of getting things fixed for the milestone on the release team.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Stéphane Graber-2]

On 06/15/2012 10:46 AM, Scott Kitterman wrote:

> On Friday, June 15, 2012 10:28:55 AM Stéphane Graber wrote:
>> On 06/15/2012 10:12 AM, Rick Spencer wrote:
>>> Hello all,
>>>
>>> At UDS I had some "hallway discussions" about why we freeze for Alphas
>>> and Betas, and the fact that I think it is time to drop this practice
>>> and rather focus on making Ubuntu good quality each day. Sadly, there
>>> was no session on this, thus this email to this list for discussion.
>>>
>>> I think it is time drop our "Freeze" practices for the alphas and
>>> betas. Here is my reasoning:
>>>
>>> 1. We are developing tools that allow us to efficiently use -proposed
>>> in a way that will ensure we will not have partially built or
>>> incompatible components in the release pocket ... ever. Including days
>>> we release Alphas and Betas:
>>>
>>> These blueprints tools to ensure that Ubuntu is not uninstallable or
>>> have other problems due to partially built components and such:
>>> https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-interme
>>> diary
>>> https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-propo
>>> sed
>>>
>>> I have been assured that the tools necessary to automate the work of
>>> moving components correctly from -proposed to the release will be
>>> ready before Alpha 2.
>>>
>>> 2. We are investing heavily in the daily quality of Ubuntu. For example
>>> ...
>>> We run the same automated tests on an alpha as we run on a daily:
>>> https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/
>>>
>>> We tend to archive issues each day:
>>> http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html
>>>
>>> We ran all the manual ISO tests *before* we released Alpha 1, and we
>>> have the capability of doing this at will:
>>> http://iso.qa.ubuntu.com/qatracker/milestones/221/builds
>>>
>>> In short, freezing the archive before an alpha or beta should not
>>> actually be contributing to either ensuring the installability of
>>> Ubuntu images or ensuring the quality of these images. This implies,
>>> therefore, that all the work around freezing, and all the productivity
>>> lost during a freeze, actually subtracts from the quality of Ubuntu by
>>> reducing our overall velocity for both features and bug fixes, since
>>> every day the image is good quality, and Alpha or Beta should be just
>>> that day's image tagged appropriately.
>>>
>>> AIUI, A1 was delivered in such a manner, though without the tooling to
>>> ensure that moving from -proposed to the release pocket was efficient
>>> and automated.
>>>
>>> Cheers, Rick
>>
>> Hi Rick,
>>
>> We certainly want to allow people to upload stuff to -proposed during a
>> milestone week, but I don't agree that we should automatically copy from
>> -proposed to the release pocket during a milestone week.
>>
>> We usually try to release all our images with the same versions of the
>> packages, considering it takes us hours to rebuild everything, having
>> seeded packages land during that time, will lead to images having
>> different version of packages.
>>
>> As for what happened with Alpha 1, we simply asked everyone to upload
>> their packages to -proposed and then cherry picked the packages we
>> actually needed for the release from -proposed and copied them into the
>> release pocket before rebuilding the images (we did that 3 times).
>>
>>
>> As I understand it, the plan going forward is to have the release pocket
>> be an alias of -proposed on upload, so that everything always lands into
>> -proposed.
>> After something lands in -proposed, is properly built and passes
>> whatever other criteria we'll have, the package will be automatically
>> copied to the release pocket.
>>
>> That last part (copy to the release pocket) would be what we'd block
>> during a milestone week for any package that's seeded. These would be
>> copied on a case by case basis by the release team and the images
>> rebuilt afterwards.
>>
>> That'd essentially allow any non-seeded package to still flow to the
>> release pocket and be available for everyone.
>> All the others will be available for people running with -proposed
>> enabled or will be available when we manually copy them to the release
>> pocket or right after we release the milestone and we copy everything
>> left in -proposed to the release pocket.
>
> This looks complicated.  Maybe it will be easier in practice.

Well, it'd be easier during hard freeze as things that aren't supposed
to be frozen would still be automatically let through (but still
preventing any archive skew).

> It also looks like a big shift of work from developers to the release team.  
> Currently it's up to developers to decide what needs uploading to get to the
> milestone.  During a soft freeze there's no action from the release team
> except coordination.  With this mechanism, developers can upload whatever and
> it's up to the release team to figure out.

Yes, that's a bit of extra work, though it also gives us the guarantee
that we won't have any accidental upload that might break the archive or
require a respin.

Packages that need to land in the release pocket for a milestone already
need to be listed on the release team pad (as respin trigger), so I
don't think it was much more work for alpha-1 to do the copy from
-proposed for these. It certainly added a little delay (extra publisher
cycle), but that's pretty much it.

> I can see this being particularly a problem where a small fix is needed for the
> milestone, but the developer's work would naturally flow to a larger update.  
> With no freeze and it being up to the release team, as Rick says, developer
> flow would continue and the release team would get stuck without good choices
> (I'm not sure if in this context there's a mechanism to do direct uploads to
> the release pocket?).

I don't have a good answer for that one, but direct upload to the
release pocket would seem to be the easy way around that problem.

> In Debian terms I'm seeing release-proposed as like unstable and release as
> testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?
>
> Scott K


--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Sebastien Bacher]

Le 15/06/2012 17:05, Scott Kitterman a écrit :
> I don't think you get to have it both ways.  Either we stabilize an image and
> put a stamp on it and we need some kind of freeze or we don't.  Trying to let
> developers continue to do their work while ignoring the milestone just pushes
> the problem of getting things fixed for the milestone on the release team.
Hey,

Yeah, you are right there, so if we get working dailies every day do we
still need alphas at all?

Ideally we would have the automatic testing flagging isos "green" when
they have no issue (with the goal to always have them good) and we would
recommend people to just pick the current green iso.

Can we just drop the image rolling part of milestones? We still probably
want fixed checkpoints in the cycle to review the features, etc but they
don't especially need to be linked with a special image...

Cheers,
Sebastien Bacher

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Robbie Williamson-2]

On 06/15/2012 10:26 AM, Sebastien Bacher wrote:

> Le 15/06/2012 17:05, Scott Kitterman a écrit :
>> I don't think you get to have it both ways.  Either we stabilize an
>> image and
>> put a stamp on it and we need some kind of freeze or we don't.  Trying
>> to let
>> developers continue to do their work while ignoring the milestone just
>> pushes
>> the problem of getting things fixed for the milestone on the release
>> team.
> Hey,
>
> Yeah, you are right there, so if we get working dailies every day do we
> still need alphas at all?
>
> Ideally we would have the automatic testing flagging isos "green" when
> they have no issue (with the goal to always have them good) and we would
> recommend people to just pick the current green iso.
>
> Can we just drop the image rolling part of milestones? We still probably
> want fixed checkpoints in the cycle to review the features, etc but they
> don't especially need to be linked with a special image...
>

With our dailies, I've found that the milestones are most useful for
planning bug fix landings and feature deliverables. I'd be +1 on
dropping alphas all together.  If we need some specific test feedback on
a given image, we can always issue calls for testing like we've done in
the past.  However, if we drop alphas, I think we might want to keep a
single beta and consider an earlier RC in place of the Beta 2.  These
releases are typically aimed at getting the less developer savy/bug
tolerant users to test and provide feedback, so I could see perhaps a
more strenuous QA process put in place for them, i.e. for system
integrated/stress testing, versus the typical Unit/Functional automated
testing we have reporting to jenkins.  It would also be good for the
release team to have a couple practice runs before the real deal ;).

Just my $0.02

-Robbie

--
Robbie Williamson <[hidden email]>
robbiew[irc.freenode.net]

"Don't make me angry...you wouldn't like me when I'm angry."
 -Bruce Banner

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Nicholas Skaggs]

On 06/15/2012 11:19 AM, Stéphane Graber wrote:

On 06/15/2012 10:46 AM, Scott Kitterman wrote:

On Friday, June 15, 2012 10:28:55 AM Stéphane Graber wrote:

On 06/15/2012 10:12 AM, Rick Spencer wrote:

Hello all,

At UDS I had some "hallway discussions" about why we freeze for Alphas
and Betas, and the fact that I think it is time to drop this practice
and rather focus on making Ubuntu good quality each day. Sadly, there
was no session on this, thus this email to this list for discussion.

I think it is time drop our "Freeze" practices for the alphas and
betas. Here is my reasoning:

1. We are developing tools that allow us to efficiently use -proposed
in a way that will ensure we will not have partially built or
incompatible components in the release pocket ... ever. Including days
we release Alphas and Betas:

These blueprints tools to ensure that Ubuntu is not uninstallable or
have other problems due to partially built components and such:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-interme
diary
https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-propo
sed

I have been assured that the tools necessary to automate the work of
moving components correctly from -proposed to the release will be
ready before Alpha 2.

2. We are investing heavily in the daily quality of Ubuntu. For example
...
We run the same automated tests on an alpha as we run on a daily:
https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/

We tend to archive issues each day:
http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html

We ran all the manual ISO tests *before* we released Alpha 1, and we
have the capability of doing this at will:
http://iso.qa.ubuntu.com/qatracker/milestones/221/builds

In short, freezing the archive before an alpha or beta should not
actually be contributing to either ensuring the installability of
Ubuntu images or ensuring the quality of these images. This implies,
therefore, that all the work around freezing, and all the productivity
lost during a freeze, actually subtracts from the quality of Ubuntu by
reducing our overall velocity for both features and bug fixes, since
every day the image is good quality, and Alpha or Beta should be just
that day's image tagged appropriately.

AIUI, A1 was delivered in such a manner, though without the tooling to
ensure that moving from -proposed to the release pocket was efficient
and automated.

Cheers, Rick

Hi Rick,

We certainly want to allow people to upload stuff to -proposed during a
milestone week, but I don't agree that we should automatically copy from
-proposed to the release pocket during a milestone week.

We usually try to release all our images with the same versions of the
packages, considering it takes us hours to rebuild everything, having
seeded packages land during that time, will lead to images having
different version of packages.

As for what happened with Alpha 1, we simply asked everyone to upload
their packages to -proposed and then cherry picked the packages we
actually needed for the release from -proposed and copied them into the
release pocket before rebuilding the images (we did that 3 times).


As I understand it, the plan going forward is to have the release pocket
be an alias of -proposed on upload, so that everything always lands into
-proposed.
After something lands in -proposed, is properly built and passes
whatever other criteria we'll have, the package will be automatically
copied to the release pocket.

That last part (copy to the release pocket) would be what we'd block
during a milestone week for any package that's seeded. These would be
copied on a case by case basis by the release team and the images
rebuilt afterwards.

That'd essentially allow any non-seeded package to still flow to the
release pocket and be available for everyone.
All the others will be available for people running with -proposed
enabled or will be available when we manually copy them to the release
pocket or right after we release the milestone and we copy everything
left in -proposed to the release pocket.

This looks complicated.  Maybe it will be easier in practice.

Well, it'd be easier during hard freeze as things that aren't supposed
to be frozen would still be automatically let through (but still
preventing any archive skew).

It also looks like a big shift of work from developers to the release team.  
Currently it's up to developers to decide what needs uploading to get to the 
milestone.  During a soft freeze there's no action from the release team 
except coordination.  With this mechanism, developers can upload whatever and 
it's up to the release team to figure out.

Yes, that's a bit of extra work, though it also gives us the guarantee
that we won't have any accidental upload that might break the archive or
require a respin.

Packages that need to land in the release pocket for a milestone already
need to be listed on the release team pad (as respin trigger), so I
don't think it was much more work for alpha-1 to do the copy from
-proposed for these. It certainly added a little delay (extra publisher
cycle), but that's pretty much it.

I can see this being particularly a problem where a small fix is needed for the 
milestone, but the developer's work would naturally flow to a larger update.  
With no freeze and it being up to the release team, as Rick says, developer 
flow would continue and the release team would get stuck without good choices 
(I'm not sure if in this context there's a mechanism to do direct uploads to 
the release pocket?).

I don't have a good answer for that one, but direct upload to the
release pocket would seem to be the easy way around that problem.

In Debian terms I'm seeing release-proposed as like unstable and release as 
testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?

Scott K 

From a community perspective, I would like to see us be able to concentrate our effort more towards making sure ubuntu is working properly throughout the cycle. Currently, we place focus on milestone testing for points of time, and we of course focus our testing on things deemed critical; for instance the installer gets quite a bit of focus at each milestone. I would rather see us having focused specific testing throughout the cycle, in addition to leveraging our community of people running the development version on a day to day basis.

So, given our "deliverable" at the end of the cycle is still an image, and our development teams are distributed and independent, I have a few thoughts:

What if we instead helped to enable our developers timelines for delivering new software in a quality manner? Things like being flexible on releasing, not freezing the archive, and enabling developers to do commit based unit testing

What if instead of several milestones, we had many small focused "releases" of software? I can imagine a managed timeline of deliveries with focused testing campaigns specific to the deliverables, and automated regression and integration testing happening to coincide.

How can we enable a better feedback loop for those who are running the development version of ubuntu everyday, and those who have a hand in producing it? Perhaps we could have the concept of a "beta" channel via the normal development archive, and turn proposed into "bleeding" edge with allowed breakage. Our metrics for quality then would be to deliver updates from proposed to the archive without rendering a users system to be un-usable for all essential hardware1.

Nicholas

1. Things that would render the system un-usable:
https://wiki.ubuntu.com/PrecisePangolin/DesktopCriticalPackages
--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Robert Ancell-3]

On 16/06/12 02:12, Rick Spencer wrote:
> In short, freezing the archive before an alpha or beta should not
> actually be contributing to either ensuring the installability of
> Ubuntu images or ensuring the quality of these images. This implies,
> therefore, that all the work around freezing, and all the productivity
> lost during a freeze, actually subtracts from the quality of Ubuntu by
> reducing our overall velocity for both features and bug fixes, since
> every day the image is good quality, and Alpha or Beta should be just
> that day's image tagged appropriately.
In particular I find the alpha freeze kills our velocity and I wonder
how more useful than a daily build the alpha release is (given it's so
early in the cycle anyway).  I'd support dropping the alpha and pointing
at the dailies.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Robert Collins-9]

On Sat, Jun 16, 2012 at 2:12 AM, Rick Spencer
<[hidden email]> wrote:
> Hello all,
>
> At UDS I had some "hallway discussions" about why we freeze for Alphas
> and Betas, and the fact that I think it is time to drop this practice
> and rather focus on making Ubuntu good quality each day. Sadly, there
> was no session on this, thus this email to this list for discussion.

+1 in general. One thing that occurs to me, is that I don't know what
the Alpha and Betas are *for* for us.... I mean: in a traditional
software product alpha releases would be used to guage customer
reaction to new features and changes, betas are used to assess
real-world defect rates (and once they drop low enough, general
release can happen).

We have landed substantial changes post-alpha-milestone in the past,
and we probably will continue to do so (e.g. Gnome releases, Unity
etc): so I'm not sure, *other* than defect rates, what Alpha does for
us.

I'm a huge fan of keeping trunk stable and release-quality always,
which makes the beta process still useful, but one that doesn't need
fixed beta releases, just get folk tracking trunk and reporting back.

-Rob

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Martin Pitt-4]

Scott Kitterman [2012-06-15 10:46 -0400]:
> In Debian terms I'm seeing release-proposed as like unstable and release as
> testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?

I don't think we want that. Our system will move it over as soon as
it's built, verified to not cause uninstallability, and pass the
automatic tests, but it will not wait any extra time (unlike Debian's
testing migration).

When preparing a milestone release, it is rather more important to
ensure to not let in a package that only built on half of the
architectures and causes uninstallability.

Martin
--
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Martin Pitt-4]

Sebastien Bacher [2012-06-15 17:26 +0200]:
> Can we just drop the image rolling part of milestones? We still
> probably want fixed checkpoints in the cycle to review the features,
> etc but they don't especially need to be linked with a special
> image...

Our automated tests are still waaaay to incomplete for this step. In
manual testing we have found quite a number of real deal-breaker bugs
which the automatic tests didn't pick up. We also need to test the
current images on a wider range of real iron; which is something our
automated QA could do one day, but doesn't right now.

So regular manual testing rounds are still required, and the points
when we do them might just as well be called "milestones".

Martin
--
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Monday, June 18, 2012 06:59:17 AM Martin Pitt wrote:

> Scott Kitterman [2012-06-15 10:46 -0400]:
> > In Debian terms I'm seeing release-proposed as like unstable and release
> > as
> > testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?
>
> I don't think we want that. Our system will move it over as soon as
> it's built, verified to not cause uninstallability, and pass the
> automatic tests, but it will not wait any extra time (unlike Debian's
> testing migration).
>
> When preparing a milestone release, it is rather more important to
> ensure to not let in a package that only built on half of the
> architectures and causes uninstallability.

This thought was in the context of dealing with uploads that were too invasive
to hit a milestone, but we needed a smaller fix to get into the milestone?  It
seems wasteful to remove from -proposed and reupload, but I guess that would
work.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Thierry Carrez-4]

Robbie Williamson wrote:

>> Yeah, you are right there, so if we get working dailies every day do we
>> still need alphas at all?
>>
>> Ideally we would have the automatic testing flagging isos "green" when
>> they have no issue (with the goal to always have them good) and we would
>> recommend people to just pick the current green iso.
>>
>> Can we just drop the image rolling part of milestones? We still probably
>> want fixed checkpoints in the cycle to review the features, etc but they
>> don't especially need to be linked with a special image...
>>
>
> With our dailies, I've found that the milestones are most useful for
> planning bug fix landings and feature deliverables. I'd be +1 on
> dropping alphas all together. [...]

I'd certainly agree that the main value of milestones is as target dates
and to create an internal cadence within a cycle. That said, I still
think there is value in singling out a given build once in a while to
serve (1) as a reference point ("this bug wasn't present at alpha2"),
(2) to encourage user testing ("please test beta1"), and (3) as an event
that builds excitement towards the upcoming release ("let me introduce
the last milestone before 12.10 final release").

The trick is to make the release process of the milestone as light as
possible, so that it does not generate a significant extra amount of
work, nor should it block development. In that respect, getting rid of
soft freezes sounds like a very good idea. Something like taking a daily
from the Tuesday and tagging it on Thursday as the milestone, unless a
critical issue gets discovered in the meantime that justifies to use a
later daily as the milestone.

Regards,

--
Thierry Carrez (ttx)

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Rick Spencer-2]

On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt <[hidden email]> wrote:

> Sebastien Bacher [2012-06-15 17:26 +0200]:
>> Can we just drop the image rolling part of milestones? We still
>> probably want fixed checkpoints in the cycle to review the features,
>> etc but they don't especially need to be linked with a special
>> image...
>
> Our automated tests are still waaaay to incomplete for this step. In
> manual testing we have found quite a number of real deal-breaker bugs
> which the automatic tests didn't pick up. We also need to test the
> current images on a wider range of real iron; which is something our
> automated QA could do one day, but doesn't right now.
>
> So regular manual testing rounds are still required, and the points
> when we do them might just as well be called "milestones".

But if the focus is testing, we should optimize the schedule around
testing. For example, I think Ubuntu would benefit from more frequent
"rounds" of such in depth testing than the current alpha/beta
milestones provide. (I think every 2 weeks would be a good cadence).

Cheers, Rick

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Moser-4]

On Mon, 18 Jun 2012, Thierry Carrez wrote:

> > With our dailies, I've found that the milestones are most useful for
> > planning bug fix landings and feature deliverables. I'd be +1 on
> > dropping alphas all together. [...]
>
> I'd certainly agree that the main value of milestones is as target dates
> and to create an internal cadence within a cycle. That said, I still
> think there is value in singling out a given build once in a while to
> serve (1) as a reference point ("this bug wasn't present at alpha2"),

I will +1 that.
I do not think it justifies alpha in itself, but it does justify keeping
*some* historic builds around for a longer period of time.

In cloud-images, anything we mark as milestone (which includes alphas) are
kept indefinitely.  The cost of a build is currently storage of ~2.1G.

The value is that people can easily reproduce that "this bug wasn't
present in alpha2", as compared to "I don't think this bug was present in
alpha2" (which is almost entirely useless).

I realize, that, because we do not keep historical content in the archive,
"alpha2" really only means "packages included in the image (or ISO) in
alpha2".  That said, I do think it is useful to keep around some
generally-functional builds to reference and bisect from.  (And I surely
wouldn't complain if someone said we should also then implement something
like snapshot.debian.org).


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Bryce Harrington-5]

On Mon, Jun 18, 2012 at 11:49:46AM +0200, Rick Spencer wrote:
> On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt <[hidden email]> wrote:
> > So regular manual testing rounds are still required, and the points
> > when we do them might just as well be called "milestones".
>
> But if the focus is testing, we should optimize the schedule around
> testing. For example, I think Ubuntu would benefit from more frequent
> "rounds" of such in depth testing than the current alpha/beta
> milestones provide. (I think every 2 weeks would be a good cadence).

Scott and Thierry mentioned the value of alphas as reference images,
such as for isolating when a regression entered the repo.  The main
problem I've seen with this is that the alphas are spaced just a bit too
far apart.  So, if we go to a 2 week cadence on the testing, and the
isos used in that testing were kept available through the release, that
would help on that point as well.

Bryce



--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel