kernel-series-info.yaml redux

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

kernel-series-info.yaml redux

Andy Whitcroft-3
We have been talking various forums about issues we have with the
kernel-series-info.yaml file.  This file is nominally meant to describe
all series of Ubuntu and the kernels contained therein.  We are starting
to want to record more information about the kernels such as adding
repository information for mainline-builds and cve-autotriage, adding snap
tracking information for shankbot etc.  As we try and shoehorn these into
the existing data format it is clear that the file is the wrong shape to
make this easy and extensible.

Currently the data really only considers series.  We then are storing
package specific information as per series dictionaries further keyed by
the package.  With multiple attributes this is just becoming hard to read
and even harder to validate.  We have multiple lists which contain
either debian packages, or primary source packages.  There are many
subtle interactions such as if a package is in derivative-packages and
not in packages then the source is not supported; nice and obvious.
Packages which are rebased on sources in the same series are specified
in a completely different manner to those in a different series; ugg.

I have taken a step back and looked at what we are trying to encode.  We
are trying to store attributes of a series; name, development, supported
etc.  We are also trying to store attributes for a source; supported,
version, related debian packges, related snap packages, and what we are
derived from.

To this end I defined a new kernel-series.yaml format and converted over
the existing data to the new form.  Below is an example record for
17.10.  Note that each kernel is referred to by the primary source
package name.  Everything related to that source package sits below it.
We have one list of dependent packages (.debs), there is also a snap
package list when needed.  The relationship between packages is always
expressed as a derived-from attribute pointing to a series and source
pair:

    # 17.10 (artful)
    '17.10':
        name: artful
        development: false
        supported: true
        sources:
            linux:
                versions: ['4.11.0', '4.12.0', '4.13.0']
                packages:
                    linux:
                        repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
                    linux-meta:
                        type: meta
                        repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
                    linux-signed:
                        type: signed
                        repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
                supported: true

            linux-raspi2:
                packages:
                    linux-raspi2:
                        repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
                    linux-meta-raspi2:
                        type: meta
                        repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
                supported: true
                derived-from: [ '17.10', 'linux' ]

Snaps are represented in a similar form to packages:

    sources:
        linux:
            versions: ['4.4.0']
            packages:
                linux:
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
                linux-meta:
                    type: meta
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
                linux-signed:
                    type: signed
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
            snaps:
                caracalla-kernel:
                pc-kernel:
                    primary: true
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
                    gated: false
                    stable: true
                    qa: false
                    hw-cert: true
                    arches: ['amd64', 'i386']
                stlouis-kernel:
            supported: true

The full new format file is available[1] and contains full documentation
on the meaning of the current fields.  The aim is to make this a sensible
form to add new attributes as we need them, such as the proposed-only
blocker for shanky is clearly a source related attribute and would get
into the sources/linux attribute set; proposed-only: true.

We have a few wrinkles with any attempt to change the format of this file.
The biggest of which is that the live ubuntu.py library consumes the
existing ktl/kernel-series-info.yaml from the launchpad git repository
directly.  This means it is essentially impossible to change the
original file without first changing all of the consumers to have new
code.  For this reason I am proposing to place the new V2 format file in
the (slightly more sensibly named) info/kernel-series.yaml in the same
repository.  We have converters to produce the old V1 format file from
that to keep un-upgraded consumers working in the short term.

The second wrinkle is that the current implementation of ubuntu.py
hands out the entire data structure for direct inspection.  This is
something else I would like to improve but in the short term it means
that even with an updated ubuntu.py we would need to expose the data in
the old form.  I am proposing to use the converter above to convert the
file as it is loaded into the old form so unconverted consumers will
still function but would be unable to access new attributes.

I am further proposing to start a small project to create proper
accessors for the various data elements in this data so that we can
convert consumers over to those.  This would be ongoing once we have
concensus that the new format makes sense to us.

In parallel I would like to start converting some of the other edge code
over to this data, such as mainline builds which have their own out of
date lists of repositories and the like.

Comments appreciated on the new format, its extensibility, and the
overall plan and direction.

-apw

[1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: kernel-series-info.yaml redux

brad.figg
On Tue, Nov 14, 2017 at 03:02:25PM +0000, Andy Whitcroft wrote:

> We have been talking various forums about issues we have with the
> kernel-series-info.yaml file.  This file is nominally meant to describe
> all series of Ubuntu and the kernels contained therein.  We are starting
> to want to record more information about the kernels such as adding
> repository information for mainline-builds and cve-autotriage, adding snap
> tracking information for shankbot etc.  As we try and shoehorn these into
> the existing data format it is clear that the file is the wrong shape to
> make this easy and extensible.
>
> Currently the data really only considers series.  We then are storing
> package specific information as per series dictionaries further keyed by
> the package.  With multiple attributes this is just becoming hard to read
> and even harder to validate.  We have multiple lists which contain
> either debian packages, or primary source packages.  There are many
> subtle interactions such as if a package is in derivative-packages and
> not in packages then the source is not supported; nice and obvious.
> Packages which are rebased on sources in the same series are specified
> in a completely different manner to those in a different series; ugg.
>
> I have taken a step back and looked at what we are trying to encode.  We
> are trying to store attributes of a series; name, development, supported
> etc.  We are also trying to store attributes for a source; supported,
> version, related debian packges, related snap packages, and what we are
> derived from.
>
> To this end I defined a new kernel-series.yaml format and converted over
> the existing data to the new form.  Below is an example record for
> 17.10.  Note that each kernel is referred to by the primary source
> package name.  Everything related to that source package sits below it.
> We have one list of dependent packages (.debs), there is also a snap
> package list when needed.  The relationship between packages is always
> expressed as a derived-from attribute pointing to a series and source
> pair:
>
>     # 17.10 (artful)
>     '17.10':
> name: artful
> development: false
> supported: true
> sources:
>    linux:
> versions: ['4.11.0', '4.12.0', '4.13.0']
> packages:
>    linux:
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
>    linux-meta:
> type: meta
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
>    linux-signed:
> type: signed
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
> supported: true
>
>    linux-raspi2:
> packages:
>    linux-raspi2:
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
>    linux-meta-raspi2:
> type: meta
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
> supported: true
> derived-from: [ '17.10', 'linux' ]
>
> Snaps are represented in a similar form to packages:
>
>     sources:
>         linux:
>             versions: ['4.4.0']
>             packages:
>                 linux:
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
>                 linux-meta:
>                     type: meta
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
>                 linux-signed:
>                     type: signed
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
>             snaps:
>                 caracalla-kernel:
>                 pc-kernel:
>                     primary: true
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
>                     gated: false
>                     stable: true
>                     qa: false
>                     hw-cert: true
>                     arches: ['amd64', 'i386']
>                 stlouis-kernel:
>             supported: true
>
> The full new format file is available[1] and contains full documentation
> on the meaning of the current fields.  The aim is to make this a sensible
> form to add new attributes as we need them, such as the proposed-only
> blocker for shanky is clearly a source related attribute and would get
> into the sources/linux attribute set; proposed-only: true.
>
> We have a few wrinkles with any attempt to change the format of this file.
> The biggest of which is that the live ubuntu.py library consumes the
> existing ktl/kernel-series-info.yaml from the launchpad git repository
> directly.  This means it is essentially impossible to change the
> original file without first changing all of the consumers to have new
> code.  For this reason I am proposing to place the new V2 format file in
> the (slightly more sensibly named) info/kernel-series.yaml in the same
> repository.  We have converters to produce the old V1 format file from
> that to keep un-upgraded consumers working in the short term.

What do you think about defining a new class KernelSeries (for example)
which consumes the new format directly? That class defines the new accessors.
As we go through our existing code base it will be easy to see what remaining
code still imports the old code. In the end, after everything has been
converted over we simply delete ktl/ubuntu.py and move on.

>
> The second wrinkle is that the current implementation of ubuntu.py
> hands out the entire data structure for direct inspection.  This is
> something else I would like to improve but in the short term it means
> that even with an updated ubuntu.py we would need to expose the data in
> the old form.  I am proposing to use the converter above to convert the
> file as it is loaded into the old form so unconverted consumers will
> still function but would be unable to access new attributes.
>
> I am further proposing to start a small project to create proper
> accessors for the various data elements in this data so that we can
> convert consumers over to those.  This would be ongoing once we have
> concensus that the new format makes sense to us.
>
> In parallel I would like to start converting some of the other edge code
> over to this data, such as mainline builds which have their own out of
> date lists of repositories and the like.
>
> Comments appreciated on the new format, its extensibility, and the
> overall plan and direction.
>
> -apw
>
> [1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml
>
> --
> kernel-team mailing list
> [hidden email]
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

--
Brad Figg [hidden email] http://www.canonical.com

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: kernel-series-info.yaml redux

Andy Whitcroft-3
On Tue, Nov 14, 2017 at 07:40:30AM -0800, Brad Figg wrote:
[...]
> What do you think about defining a new class KernelSeries (for example)
> which consumes the new format directly? That class defines the new accessors.
> As we go through our existing code base it will be easy to see what remaining
> code still imports the old code. In the end, after everything has been
> converted over we simply delete ktl/ubuntu.py and move on.

Likely we could just put KernelSeries in ktl/ubuntu.py.  As we almost
exclusivly 'from ktl.ubuntu import Ubuntu' and the new would be 'from
ktl.ubuntu import KernelSeries' sort of thing.  But yes a new class
seems most sensible.

[...]

-apw

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: kernel-series-info.yaml redux

Stefan Bader-2
In reply to this post by Andy Whitcroft-3
On 14.11.2017 16:02, Andy Whitcroft wrote:

> We have been talking various forums about issues we have with the
> kernel-series-info.yaml file.  This file is nominally meant to describe
> all series of Ubuntu and the kernels contained therein.  We are starting
> to want to record more information about the kernels such as adding
> repository information for mainline-builds and cve-autotriage, adding snap
> tracking information for shankbot etc.  As we try and shoehorn these into
> the existing data format it is clear that the file is the wrong shape to
> make this easy and extensible.
>
> Currently the data really only considers series.  We then are storing
> package specific information as per series dictionaries further keyed by
> the package.  With multiple attributes this is just becoming hard to read
> and even harder to validate.  We have multiple lists which contain
> either debian packages, or primary source packages.  There are many
> subtle interactions such as if a package is in derivative-packages and
> not in packages then the source is not supported; nice and obvious.
> Packages which are rebased on sources in the same series are specified
> in a completely different manner to those in a different series; ugg.
>
> I have taken a step back and looked at what we are trying to encode.  We
> are trying to store attributes of a series; name, development, supported
> etc.  We are also trying to store attributes for a source; supported,
> version, related debian packges, related snap packages, and what we are
> derived from.
>
> To this end I defined a new kernel-series.yaml format and converted over
> the existing data to the new form.  Below is an example record for
> 17.10.  Note that each kernel is referred to by the primary source
> package name.  Everything related to that source package sits below it.
> We have one list of dependent packages (.debs), there is also a snap
> package list when needed.  The relationship between packages is always
> expressed as a derived-from attribute pointing to a series and source
> pair:
The layout feels sensible from the layering. At this time of day, the only thing
that trips me a bit is the naming of "sources" but would not know a better name
either. "kernels" is probably not clearer as "sources"...

-Stefan

>
>     # 17.10 (artful)
>     '17.10':
> name: artful
> development: false
> supported: true
> sources:
>    linux:
> versions: ['4.11.0', '4.12.0', '4.13.0']
> packages:
>    linux:
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
>    linux-meta:
> type: meta
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
>    linux-signed:
> type: signed
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
> supported: true
>
>    linux-raspi2:
> packages:
>    linux-raspi2:
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
>    linux-meta-raspi2:
> type: meta
> repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
> supported: true
> derived-from: [ '17.10', 'linux' ]
>
> Snaps are represented in a similar form to packages:
>
>     sources:
>         linux:
>             versions: ['4.4.0']
>             packages:
>                 linux:
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
>                 linux-meta:
>                     type: meta
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
>                 linux-signed:
>                     type: signed
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
>             snaps:
>                 caracalla-kernel:
>                 pc-kernel:
>                     primary: true
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
>                     gated: false
>                     stable: true
>                     qa: false
>                     hw-cert: true
>                     arches: ['amd64', 'i386']
>                 stlouis-kernel:
>             supported: true
>
> The full new format file is available[1] and contains full documentation
> on the meaning of the current fields.  The aim is to make this a sensible
> form to add new attributes as we need them, such as the proposed-only
> blocker for shanky is clearly a source related attribute and would get
> into the sources/linux attribute set; proposed-only: true.
>
> We have a few wrinkles with any attempt to change the format of this file.
> The biggest of which is that the live ubuntu.py library consumes the
> existing ktl/kernel-series-info.yaml from the launchpad git repository
> directly.  This means it is essentially impossible to change the
> original file without first changing all of the consumers to have new
> code.  For this reason I am proposing to place the new V2 format file in
> the (slightly more sensibly named) info/kernel-series.yaml in the same
> repository.  We have converters to produce the old V1 format file from
> that to keep un-upgraded consumers working in the short term.
>
> The second wrinkle is that the current implementation of ubuntu.py
> hands out the entire data structure for direct inspection.  This is
> something else I would like to improve but in the short term it means
> that even with an updated ubuntu.py we would need to expose the data in
> the old form.  I am proposing to use the converter above to convert the
> file as it is loaded into the old form so unconverted consumers will
> still function but would be unable to access new attributes.
>
> I am further proposing to start a small project to create proper
> accessors for the various data elements in this data so that we can
> convert consumers over to those.  This would be ongoing once we have
> concensus that the new format makes sense to us.
>
> In parallel I would like to start converting some of the other edge code
> over to this data, such as mainline builds which have their own out of
> date lists of repositories and the like.
>
> Comments appreciated on the new format, its extensibility, and the
> overall plan and direction.
>
> -apw
>
> [1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml
>


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kernel-series-info.yaml redux

brad.figg
In reply to this post by Andy Whitcroft-3
On Tue, Nov 14, 2017 at 04:02:07PM +0000, Andy Whitcroft wrote:

> On Tue, Nov 14, 2017 at 07:40:30AM -0800, Brad Figg wrote:
> [...]
> > What do you think about defining a new class KernelSeries (for example)
> > which consumes the new format directly? That class defines the new accessors.
> > As we go through our existing code base it will be easy to see what remaining
> > code still imports the old code. In the end, after everything has been
> > converted over we simply delete ktl/ubuntu.py and move on.
>
> Likely we could just put KernelSeries in ktl/ubuntu.py.  As we almost
> exclusivly 'from ktl.ubuntu import Ubuntu' and the new would be 'from
> ktl.ubuntu import KernelSeries' sort of thing.  But yes a new class
> seems most sensible.

I was just thinking that as we replace old with new the import of ktl/ubuntu.py
would disappear and anything still importing it needed to be converted.

>
> [...]
>
> -apw

--
Brad Figg [hidden email] http://www.canonical.com

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team