Releasing Alphas and Betas without "freezing"

[Jono Bacon-3]

On Wed, Jun 20, 2012 at 11:39 AM, Scott Kitterman <[hidden email]> wrote:
>>I talked with Nick Skaggs this week and we are happy to commit to
>>manual testing every two weeks, starting a week on Thursday.
>>Originally I required that Nick *assured* testing of all mandatory
>>tests for each milestone, and I am asking him to provide the same
>>assurances every two weeks.
>
> For all flavors or just the Canonical ones?

For our current Ubuntu ISOs. Flavors currently are coordinating their
own testing efforts. They could either latch into the two week
cadence, or use their own cadence if desired.

    Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Thought I would chime in with my thoughts on freezes, snapshot testing,
and QA in general.

Freezes:
When I was testing the Arm images, there were times when I would go for
weeks without an image.  And if I remember correctly, there was a period
during Natty where there were no images between milestones due to pool
skew.  And there were image specific changes being made at the time
(jasper-initramfs, Ubiquity/oem-config, etc).  When those changes can't
be tested without an image, failures become critical at milestone,
causing a lot of needless stress.

Manual QA:
Having been in a QA role for most of my career in computers (+25 years),
I can site areas where automated QA can and will fail.  Sure, repetitive
tests can be automated, especially server stacks and background
libraries.  But how do you automate testing a GUI for usability
(Evolution STILL has option windows that flow past the bottom of my
netbook screen).  Plus there are tests that find the oddball combination
that most developers don't think users will select (enabling autologin &
encrypted home directory during install was found this way).  When
developing automation scripts for tests, they really should be designed
to test on all platforms, not just simulated/virtualized environments.
Otherwise others have to constantly reinvent the wheel to run the same
test on systems that don't support that test environment.

Snapshot retention:
I set up my own internal mirror so that I could keep daily images
between milestones.  This way if I missed something during daily testing
and we needed to determine how far back it went, I had a complete image
history to go back to (Mono/Banshee in Oneiric).  This also helped to
determine if one package or a combination of package updates caused
issues.

As to testing 15 arm images during milestones, my secret was to take a
daily image for one platform (panda) and focus on it over a period of
2-3 days, testing everything.  Most bugs in applications there were the
same on all arm platforms.  That left hardware specific testing on other
platforms to be done only when testing kernel/xorg updates, or (in the
case of Mono not supporting SMP on arm) reproducing a bug found on the
main test environment.  I would also try to reproduce the bug on
x86/amd64 to determine if the bug was arch specific.  Often times, an
application bug found on the panda would also be found on other
architectures.

While I no longer have the time to test Ubuntu images, I am saddened by
the failures I do see on a daily basis on the LTS release (anyone try to
run a remote desktop lately using rdesktop?).  These issues should have
been found prior to release.


--
--

Tobin Davis

The "cutting edge" is getting rather dull.
                -- Andy Purshottam


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Wednesday, June 20, 2012 02:41:00 PM Jono Bacon wrote:
> On Wed, Jun 20, 2012 at 11:39 AM, Scott Kitterman <[hidden email]>
wrote:

> >>I talked with Nick Skaggs this week and we are happy to commit to
> >>manual testing every two weeks, starting a week on Thursday.
> >>Originally I required that Nick *assured* testing of all mandatory
> >>tests for each milestone, and I am asking him to provide the same
> >>assurances every two weeks.
> >>
> > For all flavors or just the Canonical ones?
>
> For our current Ubuntu ISOs. Flavors currently are coordinating their
> own testing efforts. They could either latch into the two week
> cadence, or use their own cadence if desired.

I find it somewhat unfortunate that the "community" testing efforts exclude the
community sponsored flavors in the Ubuntu project.  I would have hoped that the
community team was not just about Canonical's products.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jono Bacon-3]

On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman <[hidden email]> wrote:
>> For our current Ubuntu ISOs. Flavors currently are coordinating their
>> own testing efforts. They could either latch into the two week
>> cadence, or use their own cadence if desired.
>
> I find it somewhat unfortunate that the "community" testing efforts exclude the
> community sponsored flavors in the Ubuntu project.  I would have hoped that the
> community team was not just about Canonical's products.

This shouldn't be a particularly big surprise; Canonical supports our
flavors with infrastructure, but we primarily focus our engineering
and community team staff members on Ubuntu.

If we had more resources we would love to provide help for the
flavors, and we are certainly happy to offer any guidance and advice,
with with our current resources and staffing, Nick doesn't have the
bandwidth to handle more the than the Ubuntu ISOs and associated
testing. Saying that, I know Nick is in contact with many of the
flavors to ensure they get the support they need to set up their own
comprehensive testing plans.

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote:
> On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman <[hidden email]>
wrote:

> >> For our current Ubuntu ISOs. Flavors currently are coordinating their
> >> own testing efforts. They could either latch into the two week
> >> cadence, or use their own cadence if desired.
> >
> > I find it somewhat unfortunate that the "community" testing efforts
> > exclude the community sponsored flavors in the Ubuntu project.  I would
> > have hoped that the community team was not just about Canonical's
> > products.
>
> This shouldn't be a particularly big surprise; Canonical supports our
> flavors with infrastructure, but we primarily focus our engineering
> and community team staff members on Ubuntu.
>
> If we had more resources we would love to provide help for the
> flavors, and we are certainly happy to offer any guidance and advice,
> with with our current resources and staffing, Nick doesn't have the
> bandwidth to handle more the than the Ubuntu ISOs and associated
> testing. Saying that, I know Nick is in contact with many of the
> flavors to ensure they get the support they need to set up their own
> comprehensive testing plans.

Perhaps I misunderstood, but I thought that you were saying this was about the
community team he had organized to support ISO testing.  Nothing to do with
Canonical resources.  I think that such a team should not be focused on just
Canonical products.

As a Kubuntu developer trying to get Kubuntu images tested for milestones,
I've often gotten a lot of help from Canonical people in Ubuntu QA.  I
appreciate that.  That's not what I'm talking about.

None of the non-Canonical flavors have the resources to pick up the pace of ISO
testing.  If Canonical is insistent on reorganizing the development process in
a way that works better for them (dropping milestones and more frequent
testing), you're going to leave us behind.

Fundamentally the development and testing model has to be something that the
entire project can support and I don't think it's unreasonable to expect that
the community team person assigned to work on QA would be trying to build a
team of community members to accomplish that.  

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jono Bacon-3]

On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman <[hidden email]> wrote:

> On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote:
>> On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman <[hidden email]>
> wrote:
>> >> For our current Ubuntu ISOs. Flavors currently are coordinating their
>> >> own testing efforts. They could either latch into the two week
>> >> cadence, or use their own cadence if desired.
>> >
>> > I find it somewhat unfortunate that the "community" testing efforts
>> > exclude the community sponsored flavors in the Ubuntu project.  I would
>> > have hoped that the community team was not just about Canonical's
>> > products.
>>
>> This shouldn't be a particularly big surprise; Canonical supports our
>> flavors with infrastructure, but we primarily focus our engineering
>> and community team staff members on Ubuntu.
>>
>> If we had more resources we would love to provide help for the
>> flavors, and we are certainly happy to offer any guidance and advice,
>> with with our current resources and staffing, Nick doesn't have the
>> bandwidth to handle more the than the Ubuntu ISOs and associated
>> testing. Saying that, I know Nick is in contact with many of the
>> flavors to ensure they get the support they need to set up their own
>> comprehensive testing plans.
>
> Perhaps I misunderstood, but I thought that you were saying this was about the
> community team he had organized to support ISO testing.  Nothing to do with
> Canonical resources.  I think that such a team should not be focused on just
> Canonical products.
>
> As a Kubuntu developer trying to get Kubuntu images tested for milestones,
> I've often gotten a lot of help from Canonical people in Ubuntu QA.  I
> appreciate that.  That's not what I'm talking about.

Well to be clear, Nick is one member of the Ubuntu community who is
focused on testing. Other people are welcome to coordinate testing
campaigns and get others interested and excited about testing, but
Nick's focus is explicitly on the Ubuntu ISOs. Of course, if community
members want to volunteer to help test the flavors then that is great.

I don't think it is unreasonable for Canonical to focus its resources
on Ubuntu as opposed to the flavors.

> None of the non-Canonical flavors have the resources to pick up the pace of ISO
> testing.  If Canonical is insistent on reorganizing the development process in
> a way that works better for them (dropping milestones and more frequent
> testing), you're going to leave us behind.
>
> Fundamentally the development and testing model has to be something that the
> entire project can support and I don't think it's unreasonable to expect that
> the community team person assigned to work on QA would be trying to build a
> team of community members to accomplish that.

To be clear, the testing cadence is up to whatever the flavor wants it
to be; I am not suggesting we impose this two-week cadence on anyone
other than me imposing it on Nick. :-)

My mail earlier in this thread was merely making it clear that from my
teams perspective, we are assigning resources on a two-week cadence.
Now, admittedly, a big reason why we can do this is because we pay
Nick to help coordinate this work.

Naturally our flavors generally don't pay people to coordinate this
kind of testing (maybe Blue Systems could invest here for Kubuntu?),
but there is no requirement for the flavors to test every two weeks.
If you folks want to test once a month or once every six weeks, then
go ahead and do that. Importantly though, as with all flavors, you
will need to coordinate this work yourselves within your community.

I see the milestones as quite disconnected from the testing: the
milestones are a useful means of consolidating efforts around
*something*, such as targeting work items and deadlines; it is just
that we happen to use these milestones as a means to release. I
personally don't see the point of the alpha releases...our users don't
use them, and our developers are running the daily development branch
anyway, so to me it makes sense to get rid of the milestones at some
point.

My goal in this cycle is to ensure we have a regular testing cadence
for Ubuntu and not based on milestones; if the Kubuntu team want to
have your own internal milestones for targeting work and testing, I
see no reason why you can't do that on your own schedule.

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Didier Roche-3]

Le 20/06/2012 08:07, Rick Spencer a écrit :

> I think this was a very productive discussion. We considered a lot of
> possibilities from a lot of angles.
>
> All told, I think there are four points under discussion. I'd like to
> tease them out so we can move forward.
>
> Question 1: shall we stop freezing the archive at milestones?
> I believe there is not 100% consensus on this point, but enough
> support to try it for Alpha 2, a la Theirry's suggestion.
> QA Team/Foundations Team, do we/will we have the tools in place for Alpha 2?

I'm supportive of those as well. We really have to shake ourselves if we
want to get in a state where we are clearly confident that no breakage
happens anymore on the devel release, and this is a good step forward to
help ourselves getting there quicker, even if we don't have the quality
tool enough, we can clearly back that up with manual testing meanwhile.

This cycle, the next step for Unity and all related components is that
each release is potentially the last one that is uploaded to ubuntu
until the finale release. So, each version is a possible release
candidate for Quantal. I do not want anymore to see half-backed unity
features coming in. This is only possible now thanks to the huge quality
increase we had last cycle and that we finally reached the feature level
that we can expect from a UI. So, all extras should be precise,
polished, reliable before getting into ubuntu.

So, removing the milestone freeze is completely aligned with that vision.
>
> Question 2: shall we stop having milestones altogether?
> This question arose in thread. I don't believe there is consensus for
> doing this suddenly in 12.10.
Milestones are great for the "marketing part" and engage people testing
a version because it's an alpha, or beta. However, our first alphas are
not alphas the same way the other products are IMHO (the alphas are more
for them "we implemented almost all new features we wanted, but it's not
stabilized at all". It can bring some confusion maybe?
>
> Question 3: shall we increase the rate of manual testing?
> This question also arose in the thread. I think there is widespread
> consensus that we should do this, and it is not actually related to
> the other questions.
> Community Team, is it feasible to increase the rate of full manual
> testing runs to every 2 weeks or similar?
It was a hard job to keep regular contributors (reporting high quality
results)  tight redoing serious testing every 2 weeks for unity
releases, but I'm completely confident Nick can do this job. :)
>
> Question 4: shall we keep snapshots of the development release so that
> we can "bisect" more easily and find when bugs were introduced?
> This question also arose, and also is not tied to the other questions.
> QA Team, is it feasible to keep a set of snap shots somewhere for this purpose?
>

That would really be awesome, especially if the reporting QA tools get
better and we can run an older iso under a vm in a minute to just test
something quickly :)

Cheers,
Didier

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Kate Stewart]

On Thu, 2012-06-21 at 10:11 +0100, Didier Roche wrote:
> This cycle, the next step for Unity and all related components is that
> each release is potentially the last one that is uploaded to ubuntu
> until the finale release. So, each version is a possible release
> candidate for Quantal. I do not want anymore to see half-backed unity
> features coming in. This is only possible now thanks to the huge quality
> increase we had last cycle and that we finally reached the feature level
> that we can expect from a UI. So, all extras should be precise,
> polished, reliable before getting into ubuntu.

+1  :)
>
> So, removing the milestone freeze is completely aligned with that vision.

Challenge is that we don't have a good schedule of when the Unity drops
are going to happen and what features are emerging when.  The other
applications and products that work on top of Unity need to interface
with it need to get synchronized in some way, so that effective system
testing can occur.  Having predictable times when we plan to release an
image is a forcing function, in that they at least keep us focused on
making sure we have system testing going on and that the community
flavors, that are part of the Ubuntu project, can system test their
emerging bits on the evolving infrastructure with some degree of
confidence.

Additional system testing and snapshotting of images at more than just
the currently scheduled points would, of course, be very welcome and
help with improving the quality, especially early in the 6 month
cycle.  ;)

> >
> > Question 3: shall we increase the rate of manual testing?
> > This question also arose in the thread. I think there is widespread
> > consensus that we should do this, and it is not actually related to
> > the other questions.
> > Community Team, is it feasible to increase the rate of full manual
> > testing runs to every 2 weeks or similar?
> It was a hard job to keep regular contributors (reporting high quality
> results)  tight redoing serious testing every 2 weeks for unity
> releases, but I'm completely confident Nick can do this job. :)

+1   :)   Big challenge for him and the QA team will be when the 12.04.1
testing has to happen in parallel with the quantal development testing.

> >
> > Question 4: shall we keep snapshots of the development release so that
> > we can "bisect" more easily and find when bugs were introduced?
> > This question also arose, and also is not tied to the other questions.
> > QA Team, is it feasible to keep a set of snap shots somewhere for this purpose?
> >
>
> That would really be awesome, especially if the reporting QA tools get
> better and we can run an older iso under a vm in a minute to just test
> something quickly :)

Am trying to think about what makes sense to keep around, and across
which products, and for how long, but if we can secure the disk space
for this, agree it would be useful to the developers and testers.  

Preliminary thoughts are:  
 * Try to keep all image that we characterize (ie. run system tests on,
get boot speed measurements, etc.) available for a *useful* period.

However since we'll never have infinite disk space, ;) and they are less
useful over time, possibly some sort of age out strategy like:
 * keep at least the last month's worth of these characterized images
available (ideally it would be nice to have a set of weekly ones ;) )
 * keep at least a monthly snapshots from each of the 6 month
development period around for historical reference.
 * keep these for all images that will be in the release manifest.

Seem reasonable?  better ideas?

Kate


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Wednesday, June 20, 2012 11:14:05 PM Jono Bacon wrote:
> My goal in this cycle is to ensure we have a regular testing cadence
> for Ubuntu and not based on milestones; if the Kubuntu team want to
> have your own internal milestones for targeting work and testing, I
> see no reason why you can't do that on your own schedule.

No.  We can't.  It requires Canonical resources to do a release.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Didier Roche-3]

Le 21/06/2012 14:34, Kate Stewart a écrit :

>> So, removing the milestone freeze is completely aligned with that vision.
> Challenge is that we don't have a good schedule of when the Unity drops
> are going to happen and what features are emerging when.  The other
> applications and products that work on top of Unity need to interface
> with it need to get synchronized in some way, so that effective system
> testing can occur.  Having predictable times when we plan to release an
> image is a forcing function, in that they at least keep us focused on
> making sure we have system testing going on and that the community
> flavors, that are part of the Ubuntu project, can system test their
> emerging bits on the evolving infrastructure with some degree of
> confidence.

Upstream is going through structural changes and having different people
working on technical debts. All of those, despite multiple demands and
priority requests for the distro is making hard to give a definitive
answer at the moment, unfortunately.

Didier

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Michael Casadevall-5]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/20/2012 11:14 PM, Jono Bacon wrote:
> On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman
<[hidden email]> wrote:
>> On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote:
>>> On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman
<[hidden email]>
>> wrote:
>>>>> For our current Ubuntu ISOs. Flavors currently are
>>>>> coordinating
their
>>>>> own testing efforts. They could either latch into the two
>>>>> week cadence, or use their own cadence if desired.
>>>>

All flavors as it stands right now are actually coordinated to a
common testing and QA cycle based around milestones, as set forth by
the release manager and the manifest. At each milestone, all images,
regardless of backing are tested. If they fail they're removed from
the manifest, and are excluded from the release.

>>>> I find it somewhat unfortunate that the "community" testing
>>>> efforts exclude the community sponsored flavors in the Ubuntu
>>>> project.  I
would
>>>> have hoped that the community team was not just about
>>>> Canonical's products.
>>>
>>> This shouldn't be a particularly big surprise; Canonical
>>> supports our flavors with infrastructure, but we primarily
>>> focus our engineering and community team staff members on
>>> Ubuntu.

No one is objecting to additional QA efforts dedicated to Canonical
images. That being said, I've yet to see a stated reason on why this
additional QA drive requires changing the milestone process. As Scott
clearly points out what is being proposed will change the
QA/milestone/release process will cause a fair bit of large grief for
all flavors.

I have regularly tested those images and others such as Studio and
Edubuntu on x86 and ports architectures when I have additional cycles
to spend, and spent several cycles making sure Kubuntu's armel images
were in a good state until hardware became more prevalent.

>>>
>>> If we had more resources we would love to provide help for the
>>> flavors, and we are certainly happy to offer any guidance and
>>> advice, with with our current resources and staffing, Nick
>>> doesn't have the bandwidth to handle more the than the Ubuntu
>>> ISOs and associated testing. Saying that, I know Nick is in
>>> contact with many of the flavors to ensure they get the support
>>> they need to set up their own comprehensive testing plans.
>>

The first I heard of any of this was when this email was sent to
u-devel. To be frank, changes of this magnitude should have been
discussed in seasons at UDS-Q, and not in hallway conversations where
many of the affected parties were not present.

Many images such of Kubuntu have worked to have the various milestones
and deadlines set in ways that allow them to incorperate the correct
versions of their upstream packages (i.e. KDE 4.9 release dates are
more or less timed that the RC and final will be available for Alpha 3
and Beta 1 respectively).

Personal opinions aside, I object to large-scale changes in release
planning after everyone already agreed to the current Quantal release
schedule.

>> Perhaps I misunderstood, but I thought that you were saying this
was about the
>> community team he had organized to support ISO testing.  Nothing
>> to
do with
>> Canonical resources.  I think that such a team should not be
focused on just
>> Canonical products.
>>
>> As a Kubuntu developer trying to get Kubuntu images tested for
milestones,

>> I've often gotten a lot of help from Canonical people in Ubuntu
>> QA.  I appreciate that.  That's not what I'm talking about.
>
> Well to be clear, Nick is one member of the Ubuntu community who
> is focused on testing. Other people are welcome to coordinate
> testing campaigns and get others interested and excited about
> testing, but Nick's focus is explicitly on the Ubuntu ISOs. Of
> course, if community members want to volunteer to help test the
> flavors then that is great.
>
> I don't think it is unreasonable for Canonical to focus its
> resources on Ubuntu as opposed to the flavors.
>

To what extent?

As it stands, what is proposed will not only hinder but likely cost
considerable QA coverage of the many images that are
"community-supported". It is clear that non-Canonical backed images
simply can not support a rolling-QA testing plan, and depend on the
milestones system.

By changing how a minority of how images are being tested, it will
only serve to create confusion, and complications. As a release
manager, am I now to track down each team, make sure from them that
they've met whatever QA schedule they've set for themselves, and then
release off that?

As it stands, the vast majority of image testing comes at milestones,
and often picks up people who are otherwise uninvolved in a flavors
development. There has often been calls to ask for additional testers
for X image in #ubuntu-testing should coverage be lacking, which have
been always answered.

>> None of the non-Canonical flavors have the resources to pick up
>> the
pace of ISO
>> testing.  If Canonical is insistent on reorganizing the
>> development
process in
>> a way that works better for them (dropping milestones and more
>> frequent testing), you're going to leave us behind.
>>
>> Fundamentally the development and testing model has to be
>> something
that the
>> entire project can support and I don't think it's unreasonable
>> to
expect that
>> the community team person assigned to work on QA would be trying
>> to
build a
>> team of community members to accomplish that.
>
> To be clear, the testing cadence is up to whatever the flavor wants
> it to be; I am not suggesting we impose this two-week cadence on
> anyone other than me imposing it on Nick. :-)
>

Then why change the existing system?

Nothing about the existing system will prevent the Canonical QA
efforts from implementing this.

> My mail earlier in this thread was merely making it clear that from
> my teams perspective, we are assigning resources on a two-week
> cadence. Now, admittedly, a big reason why we can do this is
> because we pay Nick to help coordinate this work.
>
> Naturally our flavors generally don't pay people to coordinate
> this kind of testing (maybe Blue Systems could invest here for
> Kubuntu?), but there is no requirement for the flavors to test
> every two weeks. If you folks want to test once a month or once
> every six weeks, then go ahead and do that. Importantly though, as
> with all flavors, you will need to coordinate this work yourselves
> within your community.
>

How does ARM factor into this?

I have already cited the failure by Canonical QA to test the
armhf+omap4 image for precise. Were it not for manual intervention,
and the normal Ubuntu QA testing by both myself and other members of
the community, I'm certain that we would have shipped an image that
was not only substandard, but completely broken.

It should be cited for completeness that this was caught at *Beta 2*,
and only because there were voiced concerns on image quality that I
sat and tested it extensively.

Where were the QA efforts by Canonical QA for all the previous
milestones? As it stands, I'm only aware of one person that was
responsible for all ARM images that were supported during precise.

Furthermore, speaking as the ARM Server Tech Lead in Canonical, the
PES team has to  manually QA our own images due to the fact that in
general, we are the only teams with access to the hardware. We do NOT
have the bandwidth required for this additional testing initiative.

> I see the milestones as quite disconnected from the testing: the
> milestones are a useful means of consolidating efforts around
> *something*, such as targeting work items and deadlines; it is
> just that we happen to use these milestones as a means to release.
> I personally don't see the point of the alpha releases...our users
> don't use them, and our developers are running the daily
> development branch anyway, so to me it makes sense to get rid of
> the milestones at some point.
>

I use milestones extremely regularly. Both to gauge our goals by
comparing what on the image is implemented to what is left and a
common place to gauge any bug fixes between milestones to make sure
we've fixed realistic issues between them. In addition, it provides a
time where a considerable part of the developer community comes
together to provide sanity checks on images.

In addition, as a rule (and excluding precise), during development
cycles, I generally only dist-upgrade my laptop during freezes from
milestone to milestone until after Debian Import Freeze due to archive
breakage and other issues.

Precise was considerably less broken during development due to the
simple fact we were syncing from Debian testing vs. unstable, which
ensured a minimum of package breaking due to Debian's strict policies
on how packages enter the testing archive.

> My goal in this cycle is to ensure we have a regular testing
> cadence for Ubuntu and not based on milestones; if the Kubuntu team
> want to have your own internal milestones for targeting work and
> testing, I see no reason why you can't do that on your own
> schedule.
>

I can see a very good one. We are forsaking consistency across all
flavors for Canonical's interests. This even affects the Ubuntu
Desktop and Server images on "community-supported" architectures as
there is no one who will be doing this work for armel or powerpc.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP40WuAAoJEHM+GkLSJHY5lg0P/3XszACvOqnle4L+Qx95wD0U
hGS8uS2dc72PzmYAAKEkNyU5YnD706AY7dyU440zC/MrBLcrxky2QiPL8LNyuowR
ZII+1KH5TAffDVru/NpJbOAZVAmghXmJJYVRpiWuLZfpFDcnAKXf4pyrfG5TzIz6
dxsejTMMWMsXJz07PmfHxXMgZXpQnT6d0K+k6eNB2QoxsbPKjRp8tdJUkwdElYsL
t2Vz7w/7cAEwl8Tf4LK5Vz4ti/NcIztLKRW6R3HBCdp14iK/Pon3Xe3eNycyfWZH
K2EHosqJEna8Ftktb+gbPABV54tH+MwcKUv3RbWLS3EK6TYI8xNGbJg5d02MQp9j
x+3K2zGPDced3NDFScI/OHZF0QmS3OfWp8fU0wPldjvYopYcn8IRc+YzPeYU0uKv
PPfAm0FkuxQ3aV8St5V9BLZO/+llKkFECSOfW4SKnjQ3zUG6VxjTmQPDlisrEM4W
U5oJqXAiFyGFbBI+KSiGB9zPmjH2sR9qNhZJlMZbfNY2unRn1I2kW/y8dJiz110e
GbS+UAN9OTtE5IsQBIYbhIq4gMWY0jx7b72TXUvcRJmfsmsLEveHgJIq4f/j5X1H
CnzaZUGAgUgGqiivx7kghB7Fb33ktPDXYO6NryLjFRLTtSsphzn73ePhYCw1cSn4
tz+PBu2bgheTDRuYB8Ri
=oXTd
-----END PGP SIGNATURE-----

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Rick Spencer-2]

On Thu, Jun 21, 2012 at 6:02 PM, Michael Casadevall
<[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 06/20/2012 11:14 PM, Jono Bacon wrote:
>> On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman
> <[hidden email]> wrote:
>>> On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote:
>>>> On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman
> <[hidden email]>
>>> wrote:
>>>>>> For our current Ubuntu ISOs. Flavors currently are
>>>>>> coordinating
> their
>>>>>> own testing efforts. They could either latch into the two
>>>>>> week cadence, or use their own cadence if desired.
>>>>>
>
> All flavors as it stands right now are actually coordinated to a
> common testing and QA cycle based around milestones, as set forth by
> the release manager and the manifest. At each milestone, all images,
> regardless of backing are tested. If they fail they're removed from
> the manifest, and are excluded from the release.
>
>>>>> I find it somewhat unfortunate that the "community" testing
>>>>> efforts exclude the community sponsored flavors in the Ubuntu
>>>>> project.  I
> would
>>>>> have hoped that the community team was not just about
>>>>> Canonical's products.
>>>>
>>>> This shouldn't be a particularly big surprise; Canonical
>>>> supports our flavors with infrastructure, but we primarily
>>>> focus our engineering and community team staff members on
>>>> Ubuntu.
>
> No one is objecting to additional QA efforts dedicated to Canonical
> images. That being said, I've yet to see a stated reason on why this
> additional QA drive requires changing the milestone process. As Scott
> clearly points out what is being proposed will change the
> QA/milestone/release process will cause a fair bit of large grief for
> all flavors.

Nothing is being proposed by Jono other than saying they will strive
to increase the testing cadence for Ubuntu, which as you state is not
related to alphas and betas.

Cheers, Rick

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jono Bacon-3]

On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
<[hidden email]> wrote:

>>>> If we had more resources we would love to provide help for the
>>>> flavors, and we are certainly happy to offer any guidance and
>>>> advice, with with our current resources and staffing, Nick
>>>> doesn't have the bandwidth to handle more the than the Ubuntu
>>>> ISOs and associated testing. Saying that, I know Nick is in
>>>> contact with many of the flavors to ensure they get the support
>>>> they need to set up their own comprehensive testing plans.
>>>
>
> The first I heard of any of this was when this email was sent to
> u-devel. To be frank, changes of this magnitude should have been
> discussed in seasons at UDS-Q, and not in hallway conversations where
> many of the affected parties were not present.

What changes of magnitude? We had already made it pretty clear that my
team is not focused on flavors (historically we have never focused on
flavors in our committed work items and always on Ubuntu), and all I
am outlining here is the cadence in how Nick we be conducting his
work. This is purely a way in which I choose to manage my resources to
serve Ubuntu best.

Yes, these plans were not discussed at UDS, because they have evolved
since UDS, but Nick quite clearly laid out his testing strategy at UDS
(which is not to dissimilar from this...he had scheduled testing plans
for milestones and interim dailies).

> Many images such of Kubuntu have worked to have the various milestones
> and deadlines set in ways that allow them to incorperate the correct
> versions of their upstream packages (i.e. KDE 4.9 release dates are
> more or less timed that the RC and final will be available for Alpha 3
> and Beta 1 respectively).
>
> Personal opinions aside, I object to large-scale changes in release
> planning after everyone already agreed to the current Quantal release
> schedule.

The Kubuntu team are welcome to determine whatever milestones they
want - no one is suggesting anything needs to change for the flavors
in their testing cadence. I am purely stating how Nick will be
working: the only output of this work is assured mandatory test case
testing and this testing uncovering any bugs. As I said earlier, this
work is independent of whether we remove milestones or not.

>> I don't think it is unreasonable for Canonical to focus its
>> resources on Ubuntu as opposed to the flavors.
>>
>
> To what extent?

To the extent that Canonical provides investment in Ubuntu, as part of
this investment is paying Nick's salary, and as Nick's manager I
choose how we resource his time and efforts. How we resource Nick is
not a community decision.

> As it stands, what is proposed will not only hinder but likely cost
> considerable QA coverage of the many images that are
> "community-supported". It is clear that non-Canonical backed images
> simply can not support a rolling-QA testing plan, and depend on the
> milestones system.
>
> By changing how a minority of how images are being tested, it will
> only serve to create confusion, and complications. As a release
> manager, am I now to track down each team, make sure from them that
> they've met whatever QA schedule they've set for themselves, and then
> release off that?
>
> As it stands, the vast majority of image testing comes at milestones,
> and often picks up people who are otherwise uninvolved in a flavors
> development. There has often been calls to ask for additional testers
> for X image in #ubuntu-testing should coverage be lacking, which have
> been always answered.

As I have said a few times now...If a flavor can't maintain the same
level of testing, the flavor can just do the testing it can cope with.
This might mean just doing the testing with the current cadence of
milestones: just because I am ramping up our manual testing efforts
with Nick does not mean anyone else has to change their own testing
cadence.

Flavors are in charge of their own destiny, and this includes testing cadence.

>> To be clear, the testing cadence is up to whatever the flavor wants
>> it to be; I am not suggesting we impose this two-week cadence on
>> anyone other than me imposing it on Nick. :-)
>>
>
> Then why change the existing system?
>
> Nothing about the existing system will prevent the Canonical QA
> efforts from implementing this.

You are conflating two different things: testing cadence and
milestones. To be clear: the cadence of Nick's testing has nothing to
do with milestones and whether we choose to have them or not have
them. My only point here is that I am disconnecting Nick's cadence
from milestones so that if we do choose to not have them in the
future, nothing changes.

>> Naturally our flavors generally don't pay people to coordinate
>> this kind of testing (maybe Blue Systems could invest here for
>> Kubuntu?), but there is no requirement for the flavors to test
>> every two weeks. If you folks want to test once a month or once
>> every six weeks, then go ahead and do that. Importantly though, as
>> with all flavors, you will need to coordinate this work yourselves
>> within your community.
>>
>
> How does ARM factor into this?
>
> I have already cited the failure by Canonical QA to test the
> armhf+omap4 image for precise. Were it not for manual intervention,
> and the normal Ubuntu QA testing by both myself and other members of
> the community, I'm certain that we would have shipped an image that
> was not only substandard, but completely broken.
>
> It should be cited for completeness that this was caught at *Beta 2*,
> and only because there were voiced concerns on image quality that I
> sat and tested it extensively.
>
> Where were the QA efforts by Canonical QA for all the previous
> milestones? As it stands, I'm only aware of one person that was
> responsible for all ARM images that were supported during precise.
>
> Furthermore, speaking as the ARM Server Tech Lead in Canonical, the
> PES team has to  manually QA our own images due to the fact that in
> general, we are the only teams with access to the hardware. We do NOT
> have the bandwidth required for this additional testing initiative.

From my teams perspective, we were never asked to test ARM (and thus
left it our of Nick's targeting), although this is something that we
are going to be helping with moving forward.

>> I see the milestones as quite disconnected from the testing: the
>> milestones are a useful means of consolidating efforts around
>> *something*, such as targeting work items and deadlines; it is
>> just that we happen to use these milestones as a means to release.
>> I personally don't see the point of the alpha releases...our users
>> don't use them, and our developers are running the daily
>> development branch anyway, so to me it makes sense to get rid of
>> the milestones at some point.
>>
>
> I use milestones extremely regularly. Both to gauge our goals by
> comparing what on the image is implemented to what is left and a
> common place to gauge any bug fixes between milestones to make sure
> we've fixed realistic issues between them. In addition, it provides a
> time where a considerable part of the developer community comes
> together to provide sanity checks on images.

As has been said before, no-one is suggesting milestones for targeting
work are removed....

> In addition, as a rule (and excluding precise), during development
> cycles, I generally only dist-upgrade my laptop during freezes from
> milestone to milestone until after Debian Import Freeze due to archive
> breakage and other issues.

...and here lies the point: you should be able to upgrade *anytime*;
not just at certain points. This is the gist behind Rick's points
about assuring stability in our development release.

> Precise was considerably less broken during development due to the
> simple fact we were syncing from Debian testing vs. unstable, which
> ensured a minimum of package breaking due to Debian's strict policies
> on how packages enter the testing archive.

That is one of the reasons why Precise was less broken, but there was
also automated testing, acceptance criteria, increased manual testing,
and a stronger focus on QA.

>> My goal in this cycle is to ensure we have a regular testing
>> cadence for Ubuntu and not based on milestones; if the Kubuntu team
>> want to have your own internal milestones for targeting work and
>> testing, I see no reason why you can't do that on your own
>> schedule.
>>
>
> I can see a very good one. We are forsaking consistency across all
> flavors for Canonical's interests. This even affects the Ubuntu
> Desktop and Server images on "community-supported" architectures as
> there is no one who will be doing this work for armel or powerpc.

This is not about "Canonicals' interests" this is about building
efficiency in how we build Ubuntu; this efficiency benefits not just
Canonical staff, but our wider community of participants who can
assure a more stable development release.

Nothing is being taken away here: flavors are still free to coordinate
their testing as they see fit.

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Rick Spencer-2]

On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon <[hidden email]>
>
> The Kubuntu team are welcome to determine whatever milestones they
> want - no one is suggesting anything needs to change for the flavors
> in their testing cadence. I am purely stating how Nick will be
> working: the only output of this work is assured mandatory test case
> testing and this testing uncovering any bugs. As I said earlier, this
> work is independent of whether we remove milestones or not.

I think the terminology here is the source of some confusion. I think
when you (Jono) say "milestones" people hear specifically the already
defined "alpha and beta" Milestones.  I don't think you mean that. I
think you are using the term "milestones" differently, to just mean
"chosen dates for testing".

I think it's fair to say that flavors can choose a higher frequency
for a testing cadence than just the alpha and beta dates,

Cheers, Rick

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jono Bacon-3]

On Thu, Jun 21, 2012 at 9:52 AM, Rick Spencer
<[hidden email]> wrote:

> On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon <[hidden email]>
>>
>> The Kubuntu team are welcome to determine whatever milestones they
>> want - no one is suggesting anything needs to change for the flavors
>> in their testing cadence. I am purely stating how Nick will be
>> working: the only output of this work is assured mandatory test case
>> testing and this testing uncovering any bugs. As I said earlier, this
>> work is independent of whether we remove milestones or not.
>
> I think the terminology here is the source of some confusion. I think
> when you (Jono) say "milestones" people hear specifically the already
> defined "alpha and beta" Milestones.  I don't think you mean that. I
> think you are using the term "milestones" differently, to just mean
> "chosen dates for testing".
>
> I think it's fair to say that flavors can choose a higher frequency
> for a testing cadence than just the alpha and beta dates,

Agreed: to be clear, when I say "milestones" I am referring to a
"point in time when something happens", and in the context of this
discussion testing I mean "the point in time when a testing run kicks
off". Apologies for the confusion.

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Thursday, June 21, 2012 06:52:58 PM Rick Spencer wrote:

> On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon <[hidden email]>
>
> > The Kubuntu team are welcome to determine whatever milestones they
> > want - no one is suggesting anything needs to change for the flavors
> > in their testing cadence. I am purely stating how Nick will be
> > working: the only output of this work is assured mandatory test case
> > testing and this testing uncovering any bugs. As I said earlier, this
> > work is independent of whether we remove milestones or not.
>
> I think the terminology here is the source of some confusion. I think
> when you (Jono) say "milestones" people hear specifically the already
> defined "alpha and beta" Milestones.  I don't think you mean that. I
> think you are using the term "milestones" differently, to just mean
> "chosen dates for testing".
>
> I think it's fair to say that flavors can choose a higher frequency
> for a testing cadence than just the alpha and beta dates,

Yes, but that's not what's being discussed.  What's being discusses is more
testing at shorter intervals in order to abolish the alphas and the betas.  
Once they are gone, then we don't have the milestones to organize around
anymore.  It takes Canonical employees to execute a release milestone.  Doing
an Alpha/Beta without Canonical support isn't currently something that can be
done.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Michael Casadevall-5]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/21/2012 09:43 AM, Jono Bacon wrote:

> On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
> <[hidden email]> wrote:
>>>>> If we had more resources we would love to provide help for
>>>>> the flavors, and we are certainly happy to offer any
>>>>> guidance and advice, with with our current resources and
>>>>> staffing, Nick doesn't have the bandwidth to handle more
>>>>> the than the Ubuntu ISOs and associated testing. Saying
>>>>> that, I know Nick is in contact with many of the flavors to
>>>>> ensure they get the support they need to set up their own
>>>>> comprehensive testing plans.
>>>>
>>
>> The first I heard of any of this was when this email was sent to
>> u-devel. To be frank, changes of this magnitude should have been
>> discussed in seasons at UDS-Q, and not in hallway conversations
>> where many of the affected parties were not present.
>
> What changes of magnitude? We had already made it pretty clear that
> my team is not focused on flavors (historically we have never
> focused on flavors in our committed work items and always on
> Ubuntu), and all I am outlining here is the cadence in how Nick we
> be conducting his work. This is purely a way in which I choose to
> manage my resources to serve Ubuntu best.
>
> Yes, these plans were not discussed at UDS, because they have
> evolved since UDS, but Nick quite clearly laid out his testing
> strategy at UDS (which is not to dissimilar from this...he had
> scheduled testing plans for milestones and interim dailies).
>

It has already been established by Rick that we can increase Canonical
QA efforts without changing the milestone process. I'm embedding
Rick's proposal here:

On 06/17/2012 11:36 PM, Rick Spencer wrote:
> (I changed the subject to better represent this branch of the
conversation)

>
> This discussion suggests that we don't need to release special
> alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A
> trial run (or 2) of spinning ISOs 3. Development targets
>
> Therefore, I propose we: 1. Stop with the alphas and betas and win
> back all of the development effort 2. *Increase* the cadence of ISO
> testing to whatever we want or whatever the community team can
> manage 3. Spin a trial ISO near what is not beta time (maybe around
> current kernel freeze?) 4. Spin ISOs for release candidates 5.
> Maintain the current Alpha and Beta designations as development
> targets only (i.e. don't spin a special image for them).
>
> Cheers, Rick

The justification for this as I understand it is that the main Ubuntu
image is moving to more consistent and constant QA practice, thus when
combined with the active use of -proposed during freezes, and other
tools would render the freezes and associates images moot.

We've already established that we can increase QA frequency without
changing the freeze/milestone process, and once the -proposed queue is
fully functional, no development effort will be lost for those who do
not wish to partake in testing; its simply a matter then of enabling
 -proposed on ones machine

>> Many images such of Kubuntu have worked to have the various
>> milestones and deadlines set in ways that allow them to
>> incorperate the correct versions of their upstream packages (i.e.
>> KDE 4.9 release dates are more or less timed that the RC and
>> final will be available for Alpha 3 and Beta 1 respectively).
>>
>> Personal opinions aside, I object to large-scale changes in
>> release planning after everyone already agreed to the current
>> Quantal release schedule.
>
> The Kubuntu team are welcome to determine whatever milestones they
> want - no one is suggesting anything needs to change for the
> flavors in their testing cadence. I am purely stating how Nick will
> be working: the only output of this work is assured mandatory test
> case testing and this testing uncovering any bugs. As I said
> earlier, this work is independent of whether we remove milestones
> or not.
>

The Kubuntu team wants to have standard alpha and beta images (i.e.
the system that is in place and currently works for them).

As a member of the release team, and someone who has seen the current
system of freeze/milestone release catch image breaking bugs, I want
the official desktop/server images to be cross-checked by the
community by the same process that everyone else uses.

>>> I don't think it is unreasonable for Canonical to focus its
>>> resources on Ubuntu as opposed to the flavors.
>>>
>>
>> To what extent?
>
> To the extent that Canonical provides investment in Ubuntu, as part
> of this investment is paying Nick's salary, and as Nick's manager I
> choose how we resource his time and efforts. How we resource Nick
> is not a community decision.
>

I have no argument that Canonical directs Nick's work in whatever way
we see fit. However, as someone paid by Canonical to ensure the high
quality of our ARM images, I have a vested interest in knowing if I'm
covered these efforts.

>> As it stands, what is proposed will not only hinder but likely
>> cost considerable QA coverage of the many images that are
>> "community-supported". It is clear that non-Canonical backed
>> images simply can not support a rolling-QA testing plan, and
>> depend on the milestones system.
>>
>> By changing how a minority of how images are being tested, it
>> will only serve to create confusion, and complications. As a
>> release manager, am I now to track down each team, make sure from
>> them that they've met whatever QA schedule they've set for
>> themselves, and then release off that?
>>
>> As it stands, the vast majority of image testing comes at
>> milestones, and often picks up people who are otherwise
>> uninvolved in a flavors development. There has often been calls
>> to ask for additional testers for X image in #ubuntu-testing
>> should coverage be lacking, which have been always answered.
>
> As I have said a few times now...If a flavor can't maintain the
> same level of testing, the flavor can just do the testing it can
> cope with. This might mean just doing the testing with the current
> cadence of milestones: just because I am ramping up our manual
> testing efforts with Nick does not mean anyone else has to change
> their own testing cadence.
>

As you stated later, ARM is not directly apart of these efforts as of
this moment. As such, the ARM images must remain on the current
freeze/release process.

> Flavors are in charge of their own destiny, and this includes
> testing cadence.
>

I believe the practical upshot of this is given that community-flavors
do not wish to abandon the system that is simply easer to remove
Ubuntu ISOs off the tracker (or otherwise marked as non-essential/low
priority) at milestone time and allow everyone else to carry on as is.

Any community image that wishes to follow in Canonical's stead is
welcome to bring that up with the release manager, and after making
sure a proper process of QA reporting is ensured will then be excused
from the standard ISO tracking (and will not take part of any image
milestones except final).

To make sure that we don't release a sub-par product, all flavors
should undergo full testing at the end of the cycle, with the
understanding that should they not meet release standards, they can
and will be dropped.

>>> To be clear, the testing cadence is up to whatever the flavor
>>> wants it to be; I am not suggesting we impose this two-week
>>> cadence on anyone other than me imposing it on Nick. :-)
>>>
>>
>> Then why change the existing system?
>>
>> Nothing about the existing system will prevent the Canonical QA
>> efforts from implementing this.
>
> You are conflating two different things: testing cadence and
> milestones. To be clear: the cadence of Nick's testing has nothing
> to do with milestones and whether we choose to have them or not
> have them. My only point here is that I am disconnecting Nick's
> cadence from milestones so that if we do choose to not have them in
> the future, nothing changes.
>

No, I'm not. The former is being used to justify the removal of the
later, and the current system of alpha/betas.

>>> Naturally our flavors generally don't pay people to coordinate
>>> this kind of testing (maybe Blue Systems could invest here for
>>> Kubuntu?), but there is no requirement for the flavors to test
>>> every two weeks. If you folks want to test once a month or once
>>> every six weeks, then go ahead and do that. Importantly though,
>>> as with all flavors, you will need to coordinate this work
>>> yourselves within your community.
>>>
>>
>> How does ARM factor into this?
>>
>> I have already cited the failure by Canonical QA to test the
>> armhf+omap4 image for precise. Were it not for manual
>> intervention, and the normal Ubuntu QA testing by both myself and
>> other members of the community, I'm certain that we would have
>> shipped an image that was not only substandard, but completely
>> broken.
>>
>> It should be cited for completeness that this was caught at *Beta
>> 2*, and only because there were voiced concerns on image quality
>> that I sat and tested it extensively.
>>
>> Where were the QA efforts by Canonical QA for all the previous
>> milestones? As it stands, I'm only aware of one person that was
>> responsible for all ARM images that were supported during
>> precise.
>>
>> Furthermore, speaking as the ARM Server Tech Lead in Canonical,
>> the PES team has to  manually QA our own images due to the fact
>> that in general, we are the only teams with access to the
>> hardware. We do NOT have the bandwidth required for this
>> additional testing initiative.
>
> From my teams perspective, we were never asked to test ARM (and
> thus left it our of Nick's targeting), although this is something
> that we are going to be helping with moving forward.
>

Good to know.

>>> I see the milestones as quite disconnected from the testing:
>>> the milestones are a useful means of consolidating efforts
>>> around *something*, such as targeting work items and deadlines;
>>> it is just that we happen to use these milestones as a means to
>>> release. I personally don't see the point of the alpha
>>> releases...our users don't use them, and our developers are
>>> running the daily development branch anyway, so to me it makes
>>> sense to get rid of the milestones at some point.
>>>
>>
>> I use milestones extremely regularly. Both to gauge our goals by
>> comparing what on the image is implemented to what is left and a
>> common place to gauge any bug fixes between milestones to make
>> sure we've fixed realistic issues between them. In addition, it
>> provides a time where a considerable part of the developer
>> community comes together to provide sanity checks on images.
>
> As has been said before, no-one is suggesting milestones for
> targeting work are removed....
>

The confusion on the term milestone was already cleared up. Just for
the record, when I say milestone, I consider it milestone images
(mostly because that's when we confirm during QA testing that
everything slated for that milestone went in, and wasn't accidently
mismarked as completed or such).

>> In addition, as a rule (and excluding precise), during
>> development cycles, I generally only dist-upgrade my laptop
>> during freezes from milestone to milestone until after Debian
>> Import Freeze due to archive breakage and other issues.
>
> ...and here lies the point: you should be able to upgrade
> *anytime*; not just at certain points. This is the gist behind
> Rick's points about assuring stability in our development release.
>

True, this reason for the milestones existing right now will be
resolved. That being said, it is only one part of the larger picture,
and many that do run the development releases upgrade on a much more
frequent timeframe.

>> Precise was considerably less broken during development due to
>> the simple fact we were syncing from Debian testing vs. unstable,
>> which ensured a minimum of package breaking due to Debian's
>> strict policies on how packages enter the testing archive.
>
> That is one of the reasons why Precise was less broken, but there
> was also automated testing, acceptance criteria, increased manual
> testing, and a stronger focus on QA.
>

I'm awaiting an explanation on how a Canonical supported image then
was nearly released in a broken state at Beta 2.

One of the biggest hassles of running a development release is
packages going into an uninstallable state or otherwise skewing.
Syncing from testing solved this issue since our upstream for the
period was always in a consistent state and thus we did not have
issues such as transitions to worry about.

Once our equivalent of these tools are in place, then yes, we will
have a development release that can be upgraded both safely and at will.

>>> My goal in this cycle is to ensure we have a regular testing
>>> cadence for Ubuntu and not based on milestones; if the Kubuntu
>>> team want to have your own internal milestones for targeting
>>> work and testing, I see no reason why you can't do that on your
>>> own schedule.
>>>
>>
>> I can see a very good one. We are forsaking consistency across
>> all flavors for Canonical's interests. This even affects the
>> Ubuntu Desktop and Server images on "community-supported"
>> architectures as there is no one who will be doing this work for
>> armel or powerpc.
>
> This is not about "Canonicals' interests" this is about building
> efficiency in how we build Ubuntu; this efficiency benefits not
> just Canonical staff, but our wider community of participants who
> can assure a more stable development release.
>
> Nothing is being taken away here: flavors are still free to
> coordinate their testing as they see fit.
>
> Jono
>

Scott has made it clear that if the current system of alpha/betas is
abolished, they will not be able to manage to maintain their current
level of quality. Kubuntu (and by extension, the other flavors) depend
on the current system as is.

As ARM and PowerPC is not actively part of these new QA efforts, it is
clear that we are also dependent on the current system to keep quality.

It should be noted though that as long as one architecture in an image
can't proscribe to the new testing method, the packageset relating to
that image in the release pocket must be frozen as we have no way of
freezing on a per-architecture basis.


Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP42NxAAoJEHM+GkLSJHY5O+8P/3u2LIwVLtP2ybxBxLcL9viv
qJhxMezclZerpNAB55sjrOYTr6NdN8J6bBkHOPFlXOkdSeRB9D1CSbsvWDm8CGFJ
a5lpgLVWBz4qaC5/YHB7vgcV/Huv91lWd96h0i48gmWbF3nsIbwgJnl8+gs2F0g+
cDLIY8Q01eZJTDjE0XVZr0Cnhwm6bGgFOIT4Sbi9T1dXyqdi8q79aq3EuaA7LNhY
XxayVssuowwTHIWRrtS18uYYcM/GH9jKDFX9aNof+3F9tOWA7uktNqEqhYohdxY6
9hGu6ZoMWKayX50oQ/RQT6Al2jaxamPgjvKStLnXmexsGkALJGKl++wG5uC5hdhk
qrhrBud+ExHerhvFVNPbu0y936hph58ha2OeabVNt7LtimUQYqdXFLaVN+rEJ/V/
8tSaDbq+1x5kzlq51gHHjPLjKav7ALrSp2ZT9SdTQo91D0lvwMdD8uXz9pwC4d3p
LaAr3uGOFMyZiawGfpFWs724lcS84hC1ia+BQjWjzbV+PSyJtyEFjJqgawTF07Nn
Qa+E5yYsciE4WGTijxevsVJykRpojVx+bcRtGKCZjCu9j/ZOkClCOGlWG5E+rBRe
7fkPESOXkOKJveNnc/9XceUOkvpb2ijU4j9iPV2D7HxyVSgOuh2Q+PmKiDYOSktY
0HKERvgwqMn4O3OiQBcT
=vuaU
-----END PGP SIGNATURE-----

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Wednesday, June 20, 2012 11:14:05 PM Jono Bacon wrote:
> Well to be clear, Nick is one member of the Ubuntu community who is
> focused on testing. Other people are welcome to coordinate testing
> campaigns and get others interested and excited about testing, but
> Nick's focus is explicitly on the Ubuntu ISOs. Of course, if community
> members want to volunteer to help test the flavors then that is great.
>
> I don't think it is unreasonable for Canonical to focus its resources
> on Ubuntu as opposed to the flavors.

I'm crystal clear that the Canonical community team's QA effort is focused on
trying to get the broader community to do QA on Canonical products.  I think
that's quite unfortunate.  Rather than just trying to get free labor for
Canonical, I would have hoped you wanted to make QA better for the entire
Ubuntu project.

This is in marked contrast to Daniel Holbach's efforts (which I've been
watching and appreciating, but not had much time to get involved with) to
bring new blood into the Ubuntu development process.  He's pursing the kind of
holistic approach I'd hope to see from your entire team.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Scott Kitterman-3]

On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 06/21/2012 09:43 AM, Jono Bacon wrote:
> > On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
> >
> > <[hidden email]> wrote:
> >>>>> If we had more resources we would love to provide help for
> >>>>> the flavors, and we are certainly happy to offer any
> >>>>> guidance and advice, with with our current resources and
> >>>>> staffing, Nick doesn't have the bandwidth to handle more
> >>>>> the than the Ubuntu ISOs and associated testing. Saying
> >>>>> that, I know Nick is in contact with many of the flavors to
> >>>>> ensure they get the support they need to set up their own
> >>>>> comprehensive testing plans.
> >>
> >> The first I heard of any of this was when this email was sent to
> >> u-devel. To be frank, changes of this magnitude should have been
> >> discussed in seasons at UDS-Q, and not in hallway conversations
> >> where many of the affected parties were not present.
> >
> > What changes of magnitude? We had already made it pretty clear that
> > my team is not focused on flavors (historically we have never
> > focused on flavors in our committed work items and always on
> > Ubuntu), and all I am outlining here is the cadence in how Nick we
> > be conducting his work. This is purely a way in which I choose to
> > manage my resources to serve Ubuntu best.
> >
> > Yes, these plans were not discussed at UDS, because they have
> > evolved since UDS, but Nick quite clearly laid out his testing
> > strategy at UDS (which is not to dissimilar from this...he had
> > scheduled testing plans for milestones and interim dailies).
>
> It has already been established by Rick that we can increase Canonical
> QA efforts without changing the milestone process. I'm embedding
> Rick's proposal here:
>
> On 06/17/2012 11:36 PM, Rick Spencer wrote:
> > (I changed the subject to better represent this branch of the
>
> conversation)
>
> > This discussion suggests that we don't need to release special
> > alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A
> > trial run (or 2) of spinning ISOs 3. Development targets
> >
> > Therefore, I propose we: 1. Stop with the alphas and betas and win
> > back all of the development effort 2. *Increase* the cadence of ISO
> > testing to whatever we want or whatever the community team can
> > manage 3. Spin a trial ISO near what is not beta time (maybe around
> > current kernel freeze?) 4. Spin ISOs for release candidates 5.
> > Maintain the current Alpha and Beta designations as development
> > targets only (i.e. don't spin a special image for them).
> >
> > Cheers, Rick
>
> The justification for this as I understand it is that the main Ubuntu
> image is moving to more consistent and constant QA practice, thus when
> combined with the active use of -proposed during freezes, and other
> tools would render the freezes and associates images moot.
>
> We've already established that we can increase QA frequency without
> changing the freeze/milestone process, and once the -proposed queue is
> fully functional, no development effort will be lost for those who do
> not wish to partake in testing; its simply a matter then of enabling
>  -proposed on ones machine
>
> >> Many images such of Kubuntu have worked to have the various
> >> milestones and deadlines set in ways that allow them to
> >> incorperate the correct versions of their upstream packages (i.e.
> >> KDE 4.9 release dates are more or less timed that the RC and
> >> final will be available for Alpha 3 and Beta 1 respectively).
> >>
> >> Personal opinions aside, I object to large-scale changes in
> >> release planning after everyone already agreed to the current
> >> Quantal release schedule.
> >
> > The Kubuntu team are welcome to determine whatever milestones they
> > want - no one is suggesting anything needs to change for the
> > flavors in their testing cadence. I am purely stating how Nick will
> > be working: the only output of this work is assured mandatory test
> > case testing and this testing uncovering any bugs. As I said
> > earlier, this work is independent of whether we remove milestones
> > or not.
>
> The Kubuntu team wants to have standard alpha and beta images (i.e.
> the system that is in place and currently works for them).
>
> As a member of the release team, and someone who has seen the current
> system of freeze/milestone release catch image breaking bugs, I want
> the official desktop/server images to be cross-checked by the
> community by the same process that everyone else uses.
>
> >>> I don't think it is unreasonable for Canonical to focus its
> >>> resources on Ubuntu as opposed to the flavors.
> >>
> >> To what extent?
> >
> > To the extent that Canonical provides investment in Ubuntu, as part
> > of this investment is paying Nick's salary, and as Nick's manager I
> > choose how we resource his time and efforts. How we resource Nick
> > is not a community decision.
>
> I have no argument that Canonical directs Nick's work in whatever way
> we see fit. However, as someone paid by Canonical to ensure the high
> quality of our ARM images, I have a vested interest in knowing if I'm
> covered these efforts.
>
> >> As it stands, what is proposed will not only hinder but likely
> >> cost considerable QA coverage of the many images that are
> >> "community-supported". It is clear that non-Canonical backed
> >> images simply can not support a rolling-QA testing plan, and
> >> depend on the milestones system.
> >>
> >> By changing how a minority of how images are being tested, it
> >> will only serve to create confusion, and complications. As a
> >> release manager, am I now to track down each team, make sure from
> >> them that they've met whatever QA schedule they've set for
> >> themselves, and then release off that?
> >>
> >> As it stands, the vast majority of image testing comes at
> >> milestones, and often picks up people who are otherwise
> >> uninvolved in a flavors development. There has often been calls
> >> to ask for additional testers for X image in #ubuntu-testing
> >> should coverage be lacking, which have been always answered.
> >
> > As I have said a few times now...If a flavor can't maintain the
> > same level of testing, the flavor can just do the testing it can
> > cope with. This might mean just doing the testing with the current
> > cadence of milestones: just because I am ramping up our manual
> > testing efforts with Nick does not mean anyone else has to change
> > their own testing cadence.
>
> As you stated later, ARM is not directly apart of these efforts as of
> this moment. As such, the ARM images must remain on the current
> freeze/release process.
>
> > Flavors are in charge of their own destiny, and this includes
> > testing cadence.
>
> I believe the practical upshot of this is given that community-flavors
> do not wish to abandon the system that is simply easer to remove
> Ubuntu ISOs off the tracker (or otherwise marked as non-essential/low
> priority) at milestone time and allow everyone else to carry on as is.
>
> Any community image that wishes to follow in Canonical's stead is
> welcome to bring that up with the release manager, and after making
> sure a proper process of QA reporting is ensured will then be excused
> from the standard ISO tracking (and will not take part of any image
> milestones except final).
>
> To make sure that we don't release a sub-par product, all flavors
> should undergo full testing at the end of the cycle, with the
> understanding that should they not meet release standards, they can
> and will be dropped.
>
> >>> To be clear, the testing cadence is up to whatever the flavor
> >>> wants it to be; I am not suggesting we impose this two-week
> >>> cadence on anyone other than me imposing it on Nick. :-)
> >>
> >> Then why change the existing system?
> >>
> >> Nothing about the existing system will prevent the Canonical QA
> >> efforts from implementing this.
> >
> > You are conflating two different things: testing cadence and
> > milestones. To be clear: the cadence of Nick's testing has nothing
> > to do with milestones and whether we choose to have them or not
> > have them. My only point here is that I am disconnecting Nick's
> > cadence from milestones so that if we do choose to not have them in
> > the future, nothing changes.
>
> No, I'm not. The former is being used to justify the removal of the
> later, and the current system of alpha/betas.
>
> >>> Naturally our flavors generally don't pay people to coordinate
> >>> this kind of testing (maybe Blue Systems could invest here for
> >>> Kubuntu?), but there is no requirement for the flavors to test
> >>> every two weeks. If you folks want to test once a month or once
> >>> every six weeks, then go ahead and do that. Importantly though,
> >>> as with all flavors, you will need to coordinate this work
> >>> yourselves within your community.
> >>
> >> How does ARM factor into this?
> >>
> >> I have already cited the failure by Canonical QA to test the
> >> armhf+omap4 image for precise. Were it not for manual
> >> intervention, and the normal Ubuntu QA testing by both myself and
> >> other members of the community, I'm certain that we would have
> >> shipped an image that was not only substandard, but completely
> >> broken.
> >>
> >> It should be cited for completeness that this was caught at *Beta
> >> 2*, and only because there were voiced concerns on image quality
> >> that I sat and tested it extensively.
> >>
> >> Where were the QA efforts by Canonical QA for all the previous
> >> milestones? As it stands, I'm only aware of one person that was
> >> responsible for all ARM images that were supported during
> >> precise.
> >>
> >> Furthermore, speaking as the ARM Server Tech Lead in Canonical,
> >> the PES team has to  manually QA our own images due to the fact
> >> that in general, we are the only teams with access to the
> >> hardware. We do NOT have the bandwidth required for this
> >> additional testing initiative.
> >
> > From my teams perspective, we were never asked to test ARM (and
> > thus left it our of Nick's targeting), although this is something
> > that we are going to be helping with moving forward.
>
> Good to know.
>
> >>> I see the milestones as quite disconnected from the testing:
> >>> the milestones are a useful means of consolidating efforts
> >>> around *something*, such as targeting work items and deadlines;
> >>> it is just that we happen to use these milestones as a means to
> >>> release. I personally don't see the point of the alpha
> >>> releases...our users don't use them, and our developers are
> >>> running the daily development branch anyway, so to me it makes
> >>> sense to get rid of the milestones at some point.
> >>
> >> I use milestones extremely regularly. Both to gauge our goals by
> >> comparing what on the image is implemented to what is left and a
> >> common place to gauge any bug fixes between milestones to make
> >> sure we've fixed realistic issues between them. In addition, it
> >> provides a time where a considerable part of the developer
> >> community comes together to provide sanity checks on images.
> >
> > As has been said before, no-one is suggesting milestones for
> > targeting work are removed....
>
> The confusion on the term milestone was already cleared up. Just for
> the record, when I say milestone, I consider it milestone images
> (mostly because that's when we confirm during QA testing that
> everything slated for that milestone went in, and wasn't accidently
> mismarked as completed or such).
>
> >> In addition, as a rule (and excluding precise), during
> >> development cycles, I generally only dist-upgrade my laptop
> >> during freezes from milestone to milestone until after Debian
> >> Import Freeze due to archive breakage and other issues.
> >
> > ...and here lies the point: you should be able to upgrade
> > *anytime*; not just at certain points. This is the gist behind
> > Rick's points about assuring stability in our development release.
>
> True, this reason for the milestones existing right now will be
> resolved. That being said, it is only one part of the larger picture,
> and many that do run the development releases upgrade on a much more
> frequent timeframe.
>
> >> Precise was considerably less broken during development due to
> >> the simple fact we were syncing from Debian testing vs. unstable,
> >> which ensured a minimum of package breaking due to Debian's
> >> strict policies on how packages enter the testing archive.
> >
> > That is one of the reasons why Precise was less broken, but there
> > was also automated testing, acceptance criteria, increased manual
> > testing, and a stronger focus on QA.
>
> I'm awaiting an explanation on how a Canonical supported image then
> was nearly released in a broken state at Beta 2.
>
> One of the biggest hassles of running a development release is
> packages going into an uninstallable state or otherwise skewing.
> Syncing from testing solved this issue since our upstream for the
> period was always in a consistent state and thus we did not have
> issues such as transitions to worry about.
>
> Once our equivalent of these tools are in place, then yes, we will
> have a development release that can be upgraded both safely and at will.
>
> >>> My goal in this cycle is to ensure we have a regular testing
> >>> cadence for Ubuntu and not based on milestones; if the Kubuntu
> >>> team want to have your own internal milestones for targeting
> >>> work and testing, I see no reason why you can't do that on your
> >>> own schedule.
> >>
> >> I can see a very good one. We are forsaking consistency across
> >> all flavors for Canonical's interests. This even affects the
> >> Ubuntu Desktop and Server images on "community-supported"
> >> architectures as there is no one who will be doing this work for
> >> armel or powerpc.
> >
> > This is not about "Canonicals' interests" this is about building
> > efficiency in how we build Ubuntu; this efficiency benefits not
> > just Canonical staff, but our wider community of participants who
> > can assure a more stable development release.
> >
> > Nothing is being taken away here: flavors are still free to
> > coordinate their testing as they see fit.
> >
> > Jono
>
> Scott has made it clear that if the current system of alpha/betas is
> abolished, they will not be able to manage to maintain their current
> level of quality. Kubuntu (and by extension, the other flavors) depend
> on the current system as is.
>
> As ARM and PowerPC is not actively part of these new QA efforts, it is
> clear that we are also dependent on the current system to keep quality.
>
> It should be noted though that as long as one architecture in an image
> can't proscribe to the new testing method, the packageset relating to
> that image in the release pocket must be frozen as we have no way of
> freezing on a per-architecture basis.

+1.

If Ubuntu wants to skip the Alpha/Beta milestones because Canonical thinks
it's other QA efforts are sufficient, I think they should feel free to do it, but
the existing structure needs to stay in place for the rest of us.

Scott K

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jono Bacon-3]

On Thu, Jun 21, 2012 at 11:11 AM, Scott Kitterman <[hidden email]> wrote:

>> I don't think it is unreasonable for Canonical to focus its resources
>> on Ubuntu as opposed to the flavors.
>
> I'm crystal clear that the Canonical community team's QA effort is focused on
> trying to get the broader community to do QA on Canonical products.  I think
> that's quite unfortunate.  Rather than just trying to get free labor for
> Canonical, I would have hoped you wanted to make QA better for the entire
> Ubuntu project.
>
> This is in marked contrast to Daniel Holbach's efforts (which I've been
> watching and appreciating, but not had much time to get involved with) to
> bring new blood into the Ubuntu development process.  He's pursing the kind of
> holistic approach I'd hope to see from your entire team.

We *are* working to "make QA better for the entire Ubuntu project",
but the point is that our focus is on *Ubuntu* and our specific
efforts don't extend to coordinating flavor testing. This doesn't mean
we are ignoring our flavors, or are not happy to offer advice or
guidance, but my team (Daniel included) is not focusing their efforts
on helping specific flavors achieve their goals.

I myself am surprised that you find this surprising: while many of our
efforts and programmes can bring value to the flavors (e.g. general
developer growth, working with upstreams, translations work etc), we
have rarely if ever assigned staff time to delivering on flavor work
items.

This is purely and simple about resourcing. Canonical is a company,
and it needs to invest its resources carefully: sure, we would love to
support all the flavors with more staff time (not just Kubuntu), but
we simply don't have the resources to do so. Importantly, though, we
are not stopping flavors from doing this work themselves...we are
still providing the infrastructure and help and guidance we can offer
in doing this work.

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel