git-ubuntu 0.7.1 in edge/candidate/stable channels

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

git-ubuntu 0.7.1 in edge/candidate/stable channels

Nish Aravamudan
Hello all!

I am  happy to announce the release of git-ubuntu 0.7.1 to all channels
today.

This is a massive update to our code as well as a refocus on the
import side of things. While regressions in other subcommands is not
intentional, it is certainly possible right now -- our goal is to get
all of main imported, by slowly ramping our phasing (we're aiming for
10% first, to see how it affects Launchpad, etc.).

To that end, I'm not going to provide highlights this time around.
Almost all of the code has been changed since the last tagged release!
We are working on improving code quality, by adding unit tests (thanks
to Robie for extensive work in this area).

I am about to kick off a reimport of all source packages in our
whitelist and 1% of main. Then the default Git repositories for the
working set of source packages will be updated to point at our
repositories. Finally, I will restart our keep-up job with those same
parameters. I will send follow-up e-mails as we get through each of
those steps.

Because of this reimport, if you have pending MPs against previously
imported repositories, they will need to be resubmitted (our importer
code should put a comment in appropriately!). What you will need to do
(roughly) is:

git tag <tagname> <old MP target>
git fetch pkg (which should force update)
git rebase --onto <MP target> <tagname> HEAD

We are effectively transplanting changes we had relative to <old merge
target> to now be based upon the updated pkg/<merge target> branch.
(Note that you do not need to create a tag for this, I just find it
handy to give descriptive names for commits; you can just use the commit
hash).

If you have any questions about this, please let me know.

Thanks!
-Nish

--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On 28.02.2018 [21:33:38 -0800], Nish Aravamudan wrote:
> Hello all!
>
> I am  happy to announce the release of git-ubuntu 0.7.1 to all channels
> today.

I encountered some unfortunate issues with snapcraft that hit
xenial-updates just as I was building (LP: #1752481) and then realized
we needed to snap the archive keyrings in order to verify recent archive
files (LP: #1752656). I then made a typo fixing the latter, so we're now
at 0.7.3. This should build successfully in all channels shortly.

Thanks to Sergio Schvezov for the snapcraft assist and Steve Langasek
for the keyring suggestion.

Thanks,
Nish

--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On Thu, Mar 1, 2018 at 5:20 PM, Nish Aravamudan
<[hidden email]> wrote:

> On 28.02.2018 [21:33:38 -0800], Nish Aravamudan wrote:
>> Hello all!
>>
>> I am  happy to announce the release of git-ubuntu 0.7.1 to all channels
>> today.
>
> I encountered some unfortunate issues with snapcraft that hit
> xenial-updates just as I was building (LP: #1752481) and then realized
> we needed to snap the archive keyrings in order to verify recent archive
> files (LP: #1752656). I then made a typo fixing the latter, so we're now
> at 0.7.3. This should build successfully in all channels shortly.
>
> Thanks to Sergio Schvezov for the snapcraft assist and Steve Langasek
> for the keyring suggestion.

Well, it didn't quite go as planned. I found another bug (so we're at
0.7.4) and also the snapcraft fix didn't resolve all the issues. So
I'm temporarily using a PPA that reverts the xenial-updates of
snapcraft back to 2.35.

Kyle Fazzari is working on a proper fix, but if it's not able to get
done soon, we might back out the snapcraft SRU in xenial-updates.

-Nish

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On Fri, Mar 2, 2018 at 2:00 PM, Nish Aravamudan
<[hidden email]> wrote:

> On Thu, Mar 1, 2018 at 5:20 PM, Nish Aravamudan
> <[hidden email]> wrote:
>> On 28.02.2018 [21:33:38 -0800], Nish Aravamudan wrote:
>>> Hello all!
>>>
>>> I am  happy to announce the release of git-ubuntu 0.7.1 to all channels
>>> today.
>>
>> I encountered some unfortunate issues with snapcraft that hit
>> xenial-updates just as I was building (LP: #1752481) and then realized
>> we needed to snap the archive keyrings in order to verify recent archive
>> files (LP: #1752656). I then made a typo fixing the latter, so we're now
>> at 0.7.3. This should build successfully in all channels shortly.
>>
>> Thanks to Sergio Schvezov for the snapcraft assist and Steve Langasek
>> for the keyring suggestion.
>
> Well, it didn't quite go as planned. I found another bug (so we're at
> 0.7.4) and also the snapcraft fix didn't resolve all the issues. So
> I'm temporarily using a PPA that reverts the xenial-updates of
> snapcraft back to 2.35.
>
> Kyle Fazzari is working on a proper fix, but if it's not able to get
> done soon, we might back out the snapcraft SRU in xenial-updates.

For completeness, we ended up backing out the snapcraft SRU via a
re-upload of 2.35 to xenial-updates. The snapcraft team is working on
fixes to the bugs found.

Meanwhile, a mass reimport ran last week and the existing whitelist +
1% of main was reimported. This resulted in 905 successful imports and
5 failed imports. The latter 5 will presumably be broken for a while,
as we focus on upping our phasing before resolving those issues (LP:
#1754898).

I am working right now on an issue with our default repository
repointing script.

Additionally, roughly 150 manual imports remain in the
~usd-import-team space, but those source packages were never added to
the whitelist. They need to be reimported, as well (and added to the
whitelist). I am waiting to ensure that there are no pending MPs for
those source packages before reimporting.

Once that is done, we will start the 'keep-up' script again, which
will spend some time catching up the repositories based upon Launchpad
publishes since the mass reimport.

Finally, we will then look to bump our phasing to 2% (of main), to
approximate the disk usage as the phasing increases.

Thanks for your patience as we continue to work through this process.

-Nish

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On 12.03.2018 [11:59:11 -0700], Nish Aravamudan wrote:

> On Fri, Mar 2, 2018 at 2:00 PM, Nish Aravamudan
> <[hidden email]> wrote:
> > On Thu, Mar 1, 2018 at 5:20 PM, Nish Aravamudan
> > <[hidden email]> wrote:
> >> On 28.02.2018 [21:33:38 -0800], Nish Aravamudan wrote:
> >>> Hello all!
> >>>
> >>> I am  happy to announce the release of git-ubuntu 0.7.1 to all channels
> >>> today.
> >>
> >> I encountered some unfortunate issues with snapcraft that hit
> >> xenial-updates just as I was building (LP: #1752481) and then realized
> >> we needed to snap the archive keyrings in order to verify recent archive
> >> files (LP: #1752656). I then made a typo fixing the latter, so we're now
> >> at 0.7.3. This should build successfully in all channels shortly.
> >>
> >> Thanks to Sergio Schvezov for the snapcraft assist and Steve Langasek
> >> for the keyring suggestion.
> >
> > Well, it didn't quite go as planned. I found another bug (so we're at
> > 0.7.4) and also the snapcraft fix didn't resolve all the issues. So
> > I'm temporarily using a PPA that reverts the xenial-updates of
> > snapcraft back to 2.35.
> >
> > Kyle Fazzari is working on a proper fix, but if it's not able to get
> > done soon, we might back out the snapcraft SRU in xenial-updates.
>
> For completeness, we ended up backing out the snapcraft SRU via a
> re-upload of 2.35 to xenial-updates. The snapcraft team is working on
> fixes to the bugs found.
>
> Meanwhile, a mass reimport ran last week and the existing whitelist +
> 1% of main was reimported. This resulted in 905 successful imports and
> 5 failed imports. The latter 5 will presumably be broken for a while,
> as we focus on upping our phasing before resolving those issues (LP:
> #1754898).

I have removed the repositories for the failure cases. They would not
update with the normal 'keep the repositories current' anyways, and we
would rather have consistent repository contents.

> I am working right now on an issue with our default repository
> repointing script.

This was PEBKAC.

> Additionally, roughly 150 manual imports remain in the
> ~usd-import-team space, but those source packages were never added to
> the whitelist. They need to be reimported, as well (and added to the
> whitelist). I am waiting to ensure that there are no pending MPs for
> those source packages before reimporting.

This was completed this morning.

> Once that is done, we will start the 'keep-up' script again, which
> will spend some time catching up the repositories based upon Launchpad
> publishes since the mass reimport.

This has been started and is churning through roughly 11 days of
publishing backlog.

> Finally, we will then look to bump our phasing to 2% (of main), to
> approximate the disk usage as the phasing increases.

I am looking into this step now.

Thanks,
Nish

--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On Thu, Mar 15, 2018 at 2:45 PM, Nish Aravamudan
<[hidden email]> wrote:
> On 12.03.2018 [11:59:11 -0700], Nish Aravamudan wrote:
>> On Fri, Mar 2, 2018 at 2:00 PM, Nish Aravamudan
>> <[hidden email]> wrote:

<snip>

>> Once that is done, we will start the 'keep-up' script again, which
>> will spend some time catching up the repositories based upon Launchpad
>> publishes since the mass reimport.
>
> This has been started and is churning through roughly 11 days of
> publishing backlog.
>
>> Finally, we will then look to bump our phasing to 2% (of main), to
>> approximate the disk usage as the phasing increases.
>
> I am looking into this step now.

The keep-up script ... well, caught up, and we successfully increased
our phasing of main to 2%. We are now increasing again to 10%.

Thanks,
Nish

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On Fri, Mar 16, 2018 at 11:39 AM, Nish Aravamudan
<[hidden email]> wrote:

> On Thu, Mar 15, 2018 at 2:45 PM, Nish Aravamudan
> <[hidden email]> wrote:
>> On 12.03.2018 [11:59:11 -0700], Nish Aravamudan wrote:
>>> On Fri, Mar 2, 2018 at 2:00 PM, Nish Aravamudan
>>> <[hidden email]> wrote:
>
> <snip>
>
>>> Once that is done, we will start the 'keep-up' script again, which
>>> will spend some time catching up the repositories based upon Launchpad
>>> publishes since the mass reimport.
>>
>> This has been started and is churning through roughly 11 days of
>> publishing backlog.
>>
>>> Finally, we will then look to bump our phasing to 2% (of main), to
>>> approximate the disk usage as the phasing increases.
>>
>> I am looking into this step now.
>
> The keep-up script ... well, caught up, and we successfully increased
> our phasing of main to 2%. We are now increasing again to 10%.

We successfully went through 10% and 20% of main, and have updated the
default Git repository alias for imported packages to point to the
imported repositories (so if you clone them, we should see minimal
on-disk usage in Launchpad).

We are now starting a 50% of main run, which I expect to take a while.

Thanks,
Nish

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: git-ubuntu 0.7.3 in edge/candidate/stable channels [Was Re: git-ubuntu 0.7.1 in edge/candidate/stable channels]

Nish Aravamudan
On Tue, Mar 27, 2018 at 3:10 PM, Nish Aravamudan
<[hidden email]> wrote:

> On Fri, Mar 16, 2018 at 11:39 AM, Nish Aravamudan
> <[hidden email]> wrote:
>> On Thu, Mar 15, 2018 at 2:45 PM, Nish Aravamudan
>> <[hidden email]> wrote:
>>> On 12.03.2018 [11:59:11 -0700], Nish Aravamudan wrote:
>>>> On Fri, Mar 2, 2018 at 2:00 PM, Nish Aravamudan
>>>> <[hidden email]> wrote:
>>
>> <snip>
>>
>>>> Once that is done, we will start the 'keep-up' script again, which
>>>> will spend some time catching up the repositories based upon Launchpad
>>>> publishes since the mass reimport.
>>>
>>> This has been started and is churning through roughly 11 days of
>>> publishing backlog.
>>>
>>>> Finally, we will then look to bump our phasing to 2% (of main), to
>>>> approximate the disk usage as the phasing increases.
>>>
>>> I am looking into this step now.
>>
>> The keep-up script ... well, caught up, and we successfully increased
>> our phasing of main to 2%. We are now increasing again to 10%.
>
> We successfully went through 10% and 20% of main, and have updated the
> default Git repository alias for imported packages to point to the
> imported repositories (so if you clone them, we should see minimal
> on-disk usage in Launchpad).
>
> We are now starting a 50% of main run, which I expect to take a while.

I'm excited to announce that we have completed our first 100% phasing
of main round of imports.

We have a handful of failures that are being monitored in
https://bugs.launchpad.net/usd-importer/+bug/1754898, but for those
packages not in our blacklist, we expect all of them to be imported
and kept current now. They are also setup to be the default Git
repository for the corresponding source package.

Thanks!
-Nish

>
> Thanks,
> Nish

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel