[Feedback needed] New tool for proposed migration help

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Feedback needed] New tool for proposed migration help

Mathieu Trudel-Lapierre-4
Hi!

Lukasz and I wrote a new tool to attempt to help with proposed migration
(well, we did that at least a year ago). I just added it to
lp:ubuntu-dev-tools (apologies, this was done without much consultation
prior to this email). Now we need help to make this tool even better.
Your feedback is welcome.

Proposed migration is great work for new contributors wishing to work on
something, say, to get upload privileges in Ubuntu. The main problem is
that aside from figuring out how to collaborate with others looking at
the packages, handling the possible parallel work on the same packages,
how do you find out what to do about any particular package stuck in
proposed?

That's why we wrote 'ubuntu-archive-assistant' (lp:ubuntu-dev-tools).
You can use a CLI command to check what packages are in proposed, and
what you can do to help them migrate. Obviously, this is far from
complete just yet given the large number of different things that could
block a package's migration, but I'm hoping it's already a fair way to
help users, and that more people can contribute and help making it better.

I blogged about it already[1] and the entry should already be visible on
planet.ubuntu.com. Here's essentially a copy of what I wrote on there as
usage examples...

Without any further options than "ubuntu-archive-assistant proposed", it
will list packages in -proposed (sorted by age, from oldest to newest)
and let you pick:

$ ./ubuntu-archive-assistant proposed
No source package name was provided. The following packages are blocked
in proposed:

(1) gnome-shell-extension-multi-monitors (Age: 338 days)
(2) node-is-glob (Age: 278 days)
(3) node-concat-with-sourcemaps (Age: 264 days)
(4) node-postcss (Age: 231 days)
(5) node-source-map (Age: 229 days)
(6) android-platform-system-core (Age: 226 days)
(7) libdigidocpp (Age: 226 days)
(8) qesteidutil (Age: 225 days)
(9) schleuder (Age: 218 days)
(10) ncbi-blast+ (Age: 216 days)
(11) node-postcss-filter-plugins (Age: 213 days)
(12) node-postcss-load-options (Age: 213 days)
(13) node-postcss-load-plugins (Age: 213 days)
(14) node-postcss-minify-font-values (Age: 213 days)
(15) node-postcss-load-config (Age: 209 days)
(16) live-config (Age: 207 days)
Page -1-. Press any key for next page or Q to select a package.
Which package do you want to look at? 9
Next steps for schleuder 3.2.2-1:
  Fix missing builds: amd64
     https://launchpad.net/ubuntu/+source/schleuder/3.2.2-1


If you specify which package you want to look at, it will give you the
specifics for that package:

$ ./ubuntu-archive-assistant proposed -s qesteidutil
Next steps for qesteidutil 0.3.1-0ubuntu4:
  Fix missing builds: amd64, arm64, armhf, i386, ppc64el, s390x
     https://launchpad.net/ubuntu/+source/qesteidutil/0.3.1-0ubuntu4

$ ./ubuntu-archive-assistant proposed -s android-platform-system-core
Next steps for android-platform-system-core 1:7.0.0+r33-2build1:
  Fix missing builds: amd64, arm64, armhf, i386
    
https://launchpad.net/ubuntu/+source/android-platform-system-core/1:7.0.0+r33-2build1



You can even get more information about the next steps for a package, by
enabling --verbose:

$ ./ubuntu-archive-assistant proposed -s live-config
Next steps for live-config 5.20180224:
  Fix unsatisfiable dependencies in live-config:

$ ./ubuntu-archive-assistant proposed --verbose -s live-config
live-config is not considered ✘
Next steps for live-config 5.20180224:
  Fix unsatisfiable dependencies in live-config:
    sysvinit-core | sysvinit (<< 2.88dsf-44) can not be satisfied on amd64 ✘
      sysvinit-core only exists in Debian ✘
 

We've covered the most common excuses (I think), and it will still need
work to integrate 'update-output-helper' and to generally "parse"
update_output.txt and get users more information, but I'm hoping more
eyes on the project will help improve it, and that it might already be
of some uses to people who want to see why their packages haven't
migrated yet.

So... Feedback welcome :)


--
Mathieu Trudel-Lapierre <[hidden email]>
Freenode: cyphermox, Jabber: [hidden email]
4096R/65B58DA1 818A D123 0992 275B 23C2  CF89 C67B B4D6 65B5 8DA1




--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Mattia Rizzolo-2
On Thu, Sep 20, 2018 at 10:57:49AM +0200, Mathieu Trudel-Lapierre wrote:
> Lukasz and I wrote a new tool to attempt to help with proposed migration
> (well, we did that at least a year ago). I just added it to
> lp:ubuntu-dev-tools (apologies, this was done without much consultation
> prior to this email). Now we need help to make this tool even better.
> Your feedback is welcome.

It strikes me as a clourful and interactive (and kinda more nice that
way) version of devscripts' grep-excuses.
It feels like those two things could be merged together and gain a lot
from each other (especially the plain grep-excuses…).  I think it that
much that I even opened a bug... https://bugs.debian.org/909247  :)

> work to integrate 'update-output-helper' and to generally "parse"
> update_output.txt and get users more information

Well, IME that's when most of annoying issues come into being.
Reading update_excuses.html is very easy; sure it may be handier to not
have to load the whole page each time, but it's too bothersome.  OTOH
understanding update_output.txt is much harder, and often one needs to
use other tools (dose, …) to understand the failure reasons.

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Mathieu Trudel-Lapierre
On 09/20/18 12:37, Mattia Rizzolo wrote:
>
>> work to integrate 'update-output-helper' and to generally "parse"
>> update_output.txt and get users more information
> Well, IME that's when most of annoying issues come into being.
> Reading update_excuses.html is very easy; sure it may be handier to not
> have to load the whole page each time, but it's too bothersome.  OTOH
> understanding update_output.txt is much harder, and often one needs to
> use other tools (dose, …) to understand the failure reasons.

I agree. In fact, we started a discussion on that with britney upstream,
to figure out a way of getting a machine-readable format of
update-output.txt. I suggested YAML, because that's easy to parse and
also fairly easy for a user to read.

That said, we do already catch some of the common issues: missing
builds, unresolvable dependencies, packages that require MIRs, etc. What
we can't display are required rebuilds of reverse-dependencies. That's
where integrating update-output-helper would help, too.

Much of the issue here is to keep anything that britney has to generate
to be as simple as possible and not inject too much logic that would
slow down its processing or use up more memory -- britney has enough to
do already. In that sense, I thought of just massaging the output in
text-mode it currently does, to just add enough whitespace to the
printfs to pass things as YAML (or you know, the right characters in any
case...)

--
Mathieu Trudel-Lapierre <[hidden email]>
Freenode: cyphermox, Jabber: [hidden email]
4096R/65B58DA1 818A D123 0992 275B 23C2  CF89 C67B B4D6 65B5 8DA1



--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Dan Streetman
In reply to this post by Mathieu Trudel-Lapierre-4
On Thu, Sep 20, 2018 at 4:58 AM Mathieu Trudel-Lapierre
<[hidden email]> wrote:
>
> Hi!
>
> Lukasz and I wrote a new tool to attempt to help with proposed migration
> (well, we did that at least a year ago). I just added it to
> lp:ubuntu-dev-tools (apologies, this was done without much consultation
> prior to this email). Now we need help to make this tool even better.
> Your feedback is welcome.

cool!

however, looking at the commit, it appears it's totally isolated and
uses none of the common code from ubuntutools/

that's unfortunate and some of what you've done duplicates what's
already under ubuntutools/, and some has better alternatives to how
you implemented it.

For example, your MIRReview.get_source_package is already implemented
at ubuntutools/archive.py, with much better (i.e. not using apt-cache
call-out) binary->source resolution.  Calling out to apt-cache won't
work right if the binary->source mapping is different for the target
release than your locally running release.  Likewise in
ProposedMigration.get_source_package() and .get_pkg_archive_path().
The .package_in_distro() method callout to madison does text
processing, but archive.py has a full already-json-parsed Madison
class to do all that for you.

In LaunchpadInstance, i'm not clear why you override the default
launchpad_dir (~/.launchpadlb) with a subdir (~/.launchpadlib/cache)?
This will create a cache dir of
~/.launchpadlib/cache/api.launchpad.net/cache instead of the normal
~/.launchpadlib/api.launchpad.net/cache.  In fact i'm not sure why you
need this class at all, instead of using what ubuntutools/ already
provides...

The URLRetrieverWithProgress class likewise is basically already
provided by ubuntutools/archive.py; and this class doesn't appear used
anywhere, either.

Thanks!

>
> Proposed migration is great work for new contributors wishing to work on
> something, say, to get upload privileges in Ubuntu. The main problem is
> that aside from figuring out how to collaborate with others looking at
> the packages, handling the possible parallel work on the same packages,
> how do you find out what to do about any particular package stuck in
> proposed?
>
> That's why we wrote 'ubuntu-archive-assistant' (lp:ubuntu-dev-tools).
> You can use a CLI command to check what packages are in proposed, and
> what you can do to help them migrate. Obviously, this is far from
> complete just yet given the large number of different things that could
> block a package's migration, but I'm hoping it's already a fair way to
> help users, and that more people can contribute and help making it better.
>
> I blogged about it already[1] and the entry should already be visible on
> planet.ubuntu.com. Here's essentially a copy of what I wrote on there as
> usage examples...
>
> Without any further options than "ubuntu-archive-assistant proposed", it
> will list packages in -proposed (sorted by age, from oldest to newest)
> and let you pick:
>
> $ ./ubuntu-archive-assistant proposed
> No source package name was provided. The following packages are blocked
> in proposed:
>
> (1) gnome-shell-extension-multi-monitors (Age: 338 days)
> (2) node-is-glob (Age: 278 days)
> (3) node-concat-with-sourcemaps (Age: 264 days)
> (4) node-postcss (Age: 231 days)
> (5) node-source-map (Age: 229 days)
> (6) android-platform-system-core (Age: 226 days)
> (7) libdigidocpp (Age: 226 days)
> (8) qesteidutil (Age: 225 days)
> (9) schleuder (Age: 218 days)
> (10) ncbi-blast+ (Age: 216 days)
> (11) node-postcss-filter-plugins (Age: 213 days)
> (12) node-postcss-load-options (Age: 213 days)
> (13) node-postcss-load-plugins (Age: 213 days)
> (14) node-postcss-minify-font-values (Age: 213 days)
> (15) node-postcss-load-config (Age: 209 days)
> (16) live-config (Age: 207 days)
> Page -1-. Press any key for next page or Q to select a package.
> Which package do you want to look at? 9
> Next steps for schleuder 3.2.2-1:
>   Fix missing builds: amd64
>      https://launchpad.net/ubuntu/+source/schleuder/3.2.2-1
>
>
> If you specify which package you want to look at, it will give you the
> specifics for that package:
>
> $ ./ubuntu-archive-assistant proposed -s qesteidutil
> Next steps for qesteidutil 0.3.1-0ubuntu4:
>   Fix missing builds: amd64, arm64, armhf, i386, ppc64el, s390x
>      https://launchpad.net/ubuntu/+source/qesteidutil/0.3.1-0ubuntu4
>
> $ ./ubuntu-archive-assistant proposed -s android-platform-system-core
> Next steps for android-platform-system-core 1:7.0.0+r33-2build1:
>   Fix missing builds: amd64, arm64, armhf, i386
>
> https://launchpad.net/ubuntu/+source/android-platform-system-core/1:7.0.0+r33-2build1
>
>
>
> You can even get more information about the next steps for a package, by
> enabling --verbose:
>
> $ ./ubuntu-archive-assistant proposed -s live-config
> Next steps for live-config 5.20180224:
>   Fix unsatisfiable dependencies in live-config:
>
> $ ./ubuntu-archive-assistant proposed --verbose -s live-config
> live-config is not considered ✘
> Next steps for live-config 5.20180224:
>   Fix unsatisfiable dependencies in live-config:
>     sysvinit-core | sysvinit (<< 2.88dsf-44) can not be satisfied on amd64 ✘
>       sysvinit-core only exists in Debian ✘
>
>
> We've covered the most common excuses (I think), and it will still need
> work to integrate 'update-output-helper' and to generally "parse"
> update_output.txt and get users more information, but I'm hoping more
> eyes on the project will help improve it, and that it might already be
> of some uses to people who want to see why their packages haven't
> migrated yet.
>
> So... Feedback welcome :)
>
>
> --
> Mathieu Trudel-Lapierre <[hidden email]>
> Freenode: cyphermox, Jabber: [hidden email]
> 4096R/65B58DA1 818A D123 0992 275B 23C2  CF89 C67B B4D6 65B5 8DA1
>
>
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Mathieu Trudel-Lapierre-4
On 09/20/18 16:30, Dan Streetman wrote:
[...]
> however, looking at the commit, it appears it's totally isolated and
> uses none of the common code from ubuntutools/
>
> that's unfortunate and some of what you've done duplicates what's
> already under ubuntutools/, and some has better alternatives to how
> you implemented it.

I disagree on some aspects that it has "better alternatives". Some of
this is doing the exact same thing, just going about it a slightly
different way.

To be clear, this code started in a completely separate tree, and this
is the first commit to ubuntu-dev-tools of its codebase. I wasn't using
ubuntutools because it was "not available", not really practical to use
and keep the tree (and a snap) to use as few dependencies as possible. N

pull-lp-source use was added afterwards, and the 'mir' subcommand is
special in that I wouldn't expect it to be used by anyone but the MIR
review team -- it doesn't do much else than display a bug, find the
right bug based on a fuzzy search of the package (binary or source) to
review; and then drop you to a shell to do code review. It's certainly
reusable, but things were done in a way to scratch my own itches for the
purpose of MIR review.

Now, all this was pushed to ubuntu-dev-tools as a way to get more eyes
and more use -- so when you see things you disagree with, it's
absolutely fine for you to go ahead and fix the code. :)

Kindly,

--
Mathieu Trudel-Lapierre <[hidden email]>
Freenode: cyphermox, Jabber: [hidden email]
4096R/65B58DA1 818A D123 0992 275B 23C2  CF89 C67B B4D6 65B5 8DA1


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Mattia Rizzolo-2
On Fri, Sep 21, 2018 at 09:29:55AM +0200, Mathieu Trudel-Lapierre wrote:
> Now, all this was pushed to ubuntu-dev-tools as a way to get more eyes
> and more use -- so when you see things you disagree with, it's
> absolutely fine for you to go ahead and fix the code. :)

If that was your goal, then I'd say it would have been much nicer to do
so in a separate branch, rather than pushing a ton of code to master.

> I wasn't using
> ubuntutools because it was "not available", not really practical to use
> and keep the tree (and a snap) to use as few dependencies as possible

You are talking about a tool to be used by ubuntu developers.  Asking
for python3-ubuntutools to be installed is entirely reasonable, and I
doubt you'll find anybody buying the "as few dependencies as possible"
argument in this context...

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback needed] New tool for proposed migration help

Mathieu Trudel-Lapierre
On Fri, Sep 21, 2018 at 4:08 AM Mattia Rizzolo <[hidden email]> wrote:
[...]
> If that was your goal, then I'd say it would have been much nicer to do
> so in a separate branch, rather than pushing a ton of code to master.
>

It's usable right now. In that sense, I don't see a difference in it
being in a completely separate tree vs. in a branch other than master
in lp:ubuntu-dev-tools.

To be clear, I'm not throwing this over the fence. It's usable right
now to anyone who clones lp:ubuntu-dev-tools, and we'll add fixes and
converge the code (albeit in separate branches, to merge once it
works, etc.) as an iterative process.


Mathieu Trudel-Lapierre <[hidden email]>
Freenode: cyphermox, Jabber: [hidden email]
4096R/65B58DA1 818A D123 0992 275B 23C2  CF89 C67B B4D6 65B5 8DA1

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel