Re: system-tap support in Ubuntu kernel

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: system-tap support in Ubuntu kernel

Amit Kucheria-6
Hi Sebastien,

On 10 Jun 25, Sebastien Jan wrote:

> Hi Bryan,
>
> Just a short query regarding kernel config.
> Some of our teams are regularly making performance tests and some monitoring and use system-tap for this task.
> The Ubuntu config already includes some flags enabling a basic system-tap support out of the box.
> However, for most of our system-tap based tests, an additional config flag has to be activated: CONFIG_BLK_DEV_IO_TRACE
> This flag activates several additional tracers into the kernel config (see below for detailed config impact).
>
> We are looking at if we shall include these additional flags in our default kernel images, or if we shall make special builds dedicated to system-tap usage (including this additional flag).
>
> The criterion behind this choice are:
>  - kernel size: I checked - increase of a bit less than 5%
>  - performances: I have no idea
>
> => So the questions are:
> 1) I know that the ubuntu kernel team makes reviews of the flags to integrate into the ubuntu kernel. Were these flags investigated, and was there a rational for not taking them?

Yes they are reviewed at UDS. I think someone on the kernel team will know
the details about this specific decision, the mailing list is now cc'ed.
Chase and Andy have been looking at the tracing problem in general.

> 2) Would you take these flags into the ti-omap4 branch of the maverick tree?

I am curious to know if there is something in systemtap that cannot be
measured using ftrace? As I understand the winds blowing in kernel-land,
ftrace is the new favourite amongst all the tracing subsystems and is already
upstream.

From personal experience, systemtap has always been very cumbersome to use,
ftrace is a lot easier. Also, the runtime overhead of ftrace is close to zero (really
zero?) when it is not enabled.

> Please feel free to forward to other Ubuntu kernel folks, as appropriate.

Done.

> Thanks,
> Sebastien
>
> ---
> Complete changes implied by activating the CONFIG_BLK_DEV_IO_TRACE flag:
> -ENABLE_DEFAULT_TRACERS n
>  BINARY_PRINTF n -> y
>  BLK_DEV_IO_TRACE n -> y
> +CONTEXT_SWITCH_TRACER y
> +EVENT_TRACING y
> +FTRACE_STARTUP_TEST n
> +GENERIC_TRACER y
> +NET_DROP_MONITOR n
> +NOP_TRACER y
> +STACKTRACE y
> +TRACEPOINTS y
> +TRACING y

--
----------------------------------------------------------------------
Amit Kucheria, Kernel Engineer || [hidden email]
----------------------------------------------------------------------

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: system-tap support in Ubuntu kernel

Chase Douglas-5
On Sun, 2010-06-27 at 20:50 +0300, Amit Kucheria wrote:

> On 10 Jun 25, Sebastien Jan wrote:
> > Just a short query regarding kernel config.
> > Some of our teams are regularly making performance tests and some monitoring and use system-tap for this task.
> > The Ubuntu config already includes some flags enabling a basic system-tap support out of the box.
> > However, for most of our system-tap based tests, an additional config flag has to be activated: CONFIG_BLK_DEV_IO_TRACE
> > This flag activates several additional tracers into the kernel config (see below for detailed config impact).
> >
> > We are looking at if we shall include these additional flags in our default kernel images, or if we shall make special builds dedicated to system-tap usage (including this additional flag).
> >
> > The criterion behind this choice are:
> >  - kernel size: I checked - increase of a bit less than 5%
> >  - performances: I have no idea
> >
> > => So the questions are:
> > 1) I know that the ubuntu kernel team makes reviews of the flags to integrate into the ubuntu kernel. Were these flags investigated, and was there a rational for not taking them?
>
> Yes they are reviewed at UDS. I think someone on the kernel team will know
> the details about this specific decision, the mailing list is now cc'ed.
> Chase and Andy have been looking at the tracing problem in general.

I've been watching perf and ftrace for the Ubuntu kernel. I had
questions about the usage of some config options, so I inquired
upstream:

http://lkml.org/lkml/2010/5/25/382

> > 2) Would you take these flags into the ti-omap4 branch of the maverick tree?
>
> I am curious to know if there is something in systemtap that cannot be
> measured using ftrace? As I understand the winds blowing in kernel-land,
> ftrace is the new favourite amongst all the tracing subsystems and is already
> upstream.
>
> From personal experience, systemtap has always been very cumbersome to use,
> ftrace is a lot easier. Also, the runtime overhead of ftrace is close to zero (really
> zero?) when it is not enabled.

The problem here is that ARM still cannot dynamically enable and disable
ftrace. Thus, enabling it for ARM has a performance hit.

The only thing I am aware of that systemtap has over ftrace and perf
probe (new in maverick) is the ability to modify code execution (more
than just printing values of variables and registers). However, my gut
tells me this isn't used very often.

> > Complete changes implied by activating the CONFIG_BLK_DEV_IO_TRACE flag:
> > -ENABLE_DEFAULT_TRACERS n
> >  BINARY_PRINTF n -> y
> >  BLK_DEV_IO_TRACE n -> y
> > +CONTEXT_SWITCH_TRACER y
> > +EVENT_TRACING y
> > +FTRACE_STARTUP_TEST n
> > +GENERIC_TRACER y
> > +NET_DROP_MONITOR n
> > +NOP_TRACER y
> > +STACKTRACE y
> > +TRACEPOINTS y
> > +TRACING y

See the following email for what's enabled in our common builds:

http://lkml.org/lkml/2010/6/8/319

We then have architecture and flavour overrides, so it's best to check
our git tree for all the relevant config options.

-- Chase


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: system-tap support in Ubuntu kernel

Bryan Wu-5
On 06/29/2010 12:32 AM, Chase Douglas wrote:

> On Sun, 2010-06-27 at 20:50 +0300, Amit Kucheria wrote:
>> On 10 Jun 25, Sebastien Jan wrote:
>>> Just a short query regarding kernel config.
>>> Some of our teams are regularly making performance tests and some monitoring and use system-tap for this task.
>>> The Ubuntu config already includes some flags enabling a basic system-tap support out of the box.
>>> However, for most of our system-tap based tests, an additional config flag has to be activated: CONFIG_BLK_DEV_IO_TRACE
>>> This flag activates several additional tracers into the kernel config (see below for detailed config impact).
>>>
>>> We are looking at if we shall include these additional flags in our default kernel images, or if we shall make special builds dedicated to system-tap usage (including this additional flag).
>>>
>>> The criterion behind this choice are:
>>>  - kernel size: I checked - increase of a bit less than 5%
>>>  - performances: I have no idea
>>>
>>> => So the questions are:
>>> 1) I know that the ubuntu kernel team makes reviews of the flags to integrate into the ubuntu kernel. Were these flags investigated, and was there a rational for not taking them?
>>
>> Yes they are reviewed at UDS. I think someone on the kernel team will know
>> the details about this specific decision, the mailing list is now cc'ed.
>> Chase and Andy have been looking at the tracing problem in general.
>
> I've been watching perf and ftrace for the Ubuntu kernel. I had
> questions about the usage of some config options, so I inquired
> upstream:
>
> http://lkml.org/lkml/2010/5/25/382
>
>>> 2) Would you take these flags into the ti-omap4 branch of the maverick tree?
>>
>> I am curious to know if there is something in systemtap that cannot be
>> measured using ftrace? As I understand the winds blowing in kernel-land,
>> ftrace is the new favourite amongst all the tracing subsystems and is already
>> upstream.
>>
>> From personal experience, systemtap has always been very cumbersome to use,
>> ftrace is a lot easier. Also, the runtime overhead of ftrace is close to zero (really
>> zero?) when it is not enabled.
>
> The problem here is that ARM still cannot dynamically enable and disable
> ftrace. Thus, enabling it for ARM has a performance hit.
>

Right, during Maverick cycle, we might not be able to enable ftrace for armel
and omap due to miss dynamic ftrace feature.

> The only thing I am aware of that systemtap has over ftrace and perf
> probe (new in maverick) is the ability to modify code execution (more
> than just printing values of variables and registers). However, my gut
> tells me this isn't used very often.
>
>>> Complete changes implied by activating the CONFIG_BLK_DEV_IO_TRACE flag:

CONFIG_BLK_DEV_IO_TRACE=y in our master branch

>>> -ENABLE_DEFAULT_TRACERS n

Not existed in master, either.

>>>  BINARY_PRINTF n -> y

CONFIG_BINARY_PRINTF=y in master

>>>  BLK_DEV_IO_TRACE n -> y

ditto.

>>> +CONTEXT_SWITCH_TRACER y

ditto.

>>> +EVENT_TRACING y

ditto.

>>> +FTRACE_STARTUP_TEST n

Not set in master

>>> +GENERIC_TRACER y

CONFIG_GENERIC_TRACER=y in master

>>> +NET_DROP_MONITOR n

Not set in master

>>> +NOP_TRACER y

CONFIG_NOP_TRACER=y in master

>>> +STACKTRACE y

CONFIG_STACKTRACE=y in master

>>> +TRACEPOINTS y

CONFIG_TRACEPOINTS=y in master

>>> +TRACING y

CONFIG_TRACING=y in master

>
> See the following email for what's enabled in our common builds:
>
> http://lkml.org/lkml/2010/6/8/319
>
> We then have architecture and flavour overrides, so it's best to check
> our git tree for all the relevant config options.
>

After review these config options in master, I think it's OK for us to enable
CONFIG_BLK_DEV_IO_TRACE=y. All of these config options will be the same as
master branch config.

So Sebastien, please set them as you proposed here.

Thanks,
--
Bryan Wu <[hidden email]>
Kernel Developer    +86.138-1617-6545 Mobile
Ubuntu Kernel Team | Hardware Enablement Team
Canonical Ltd.      www.canonical.com
Ubuntu - Linux for human beings | www.ubuntu.com

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: system-tap support in Ubuntu kernel

Amit Kucheria-6
In reply to this post by Chase Douglas-5
On 10 Jun 28, Chase Douglas wrote:
>
> The problem here is that ARM still cannot dynamically enable and disable
> ftrace. Thus, enabling it for ARM has a performance hit.

Do you know if this is being actively worked on?

--
----------------------------------------------------------------------
Amit Kucheria, Kernel Engineer || [hidden email]
----------------------------------------------------------------------

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

Re: system-tap support in Ubuntu kernel

Bryan Wu-5
On 06/29/2010 01:46 PM, Amit Kucheria wrote:
> On 10 Jun 28, Chase Douglas wrote:
>>
>> The problem here is that ARM still cannot dynamically enable and disable
>> ftrace. Thus, enabling it for ARM has a performance hit.
>
> Do you know if this is being actively worked on?
>

IIRC, there are some patches in arm-linux mail list, which add dynamic ftrace
feature. But they are not in mainline yet.

Thanks,
-Bryan

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team