mlocate - what is it good for?

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

mlocate - what is it good for?

Brian Murray-5
The Ubuntu Foundations team was recently looking at an issue with
mlocate[1] and the effect it has on all users of Ubuntu. While that
specific issue is fixable there are also issues[2,3] with keeping
PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
questioning the usefulness of installing mlocate by default on systems
at all. We believe that find is an adequate replacement for mlocate but
want to hear from you about use cases where it may not be. I'll start
with a personal example:

"I don't remember (because I need to know so infrequently) where the
meta-release file is cached on disk by update-manager and use locate to
find it. The find command itself is inadequate because the cached file
exists in both /home and /var."

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880507
[2] http://launchpad.net/bugs/827841
[3] http://launchpad.net/bugs/1823518

Thanks,
--
Brian Murray

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Neal McBurnett
I use mlocate multiple times a day.
Find is way too slow and inconvenient for finding files in a big
set of filesystems, compared to properly configuring mlocate.

It also seems that the bugs should be addressed (and have in some
cases been addressed) whether or not find is installed by default.

Thanks,

Neal McBurnett                 http://neal.mcburnett.org/

On Wed, May 22, 2019 at 11:59:57AM -0700, Brian Murray wrote:

> The Ubuntu Foundations team was recently looking at an issue with
> mlocate[1] and the effect it has on all users of Ubuntu. While that
> specific issue is fixable there are also issues[2,3] with keeping
> PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
> questioning the usefulness of installing mlocate by default on systems
> at all. We believe that find is an adequate replacement for mlocate but
> want to hear from you about use cases where it may not be. I'll start
> with a personal example:
>
> "I don't remember (because I need to know so infrequently) where the
> meta-release file is cached on disk by update-manager and use locate to
> find it. The find command itself is inadequate because the cached file
> exists in both /home and /var."
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880507
> [2] http://launchpad.net/bugs/827841
> [3] http://launchpad.net/bugs/1823518
>
> Thanks,
> --
> Brian Murray
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Andrea Corbellini-4
In reply to this post by Brian Murray-5
Hi Brian,

I personally use 'locate' all the time, and rarely use 'find' if 'locate' is available.

The major problem I have with find is that it's slow. Even restricting the search space to a specific directory (as opposed to searching everything under "/") can be slow. For example, searching my home directory takes a while, due to the large number of Python virtual envs that I have.

Sometimes searching a specific directory simply does not work with find due to the presence of symlinks. You have to wait for find to complete (which might take a while), realize that maybe symlinks are involved (which is not obvious) and re-run find with -L, hoping it will work.

Searching from the root directory "/" with find is not just slow, but also clobbers the terminal with "Permission Denied" or "Operation Not Permitted" errors. You must remember to use 2>/dev/null (hoping that won't hide any useful errors).

With locate instead you can search the entire filesystem and get your results in a few milliseconds, without thinking about edge cases.

Also, if you want to always exclude certain filesystems or directories, with find you have to play with -prune or -xdev or similar, and repeat the same arguments every time. With locate, you can just edit one configuration file once and you're done. In my locate configuration I exclude things like my LXC/LXD containers: I never want results involving those.

Those are the reasons why I wish locate was enabled everywhere, not just Ubuntu :-)

Hope this helps!


On Wed, May 22, 2019, 20:00 Brian Murray <[hidden email]> wrote:
The Ubuntu Foundations team was recently looking at an issue with
mlocate[1] and the effect it has on all users of Ubuntu. While that
specific issue is fixable there are also issues[2,3] with keeping
PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
questioning the usefulness of installing mlocate by default on systems
at all. We believe that find is an adequate replacement for mlocate but
want to hear from you about use cases where it may not be. I'll start
with a personal example:

"I don't remember (because I need to know so infrequently) where the
meta-release file is cached on disk by update-manager and use locate to
find it. The find command itself is inadequate because the cached file
exists in both /home and /var."

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880507
[2] http://launchpad.net/bugs/827841
[3] http://launchpad.net/bugs/1823518

Thanks,
--
Brian Murray

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Julian Andres Klode
In reply to this post by Neal McBurnett
On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> I use mlocate multiple times a day.
> Find is way too slow and inconvenient for finding files in a big
> set of filesystems, compared to properly configuring mlocate.

Specifically, the filesystem must be huge or on a slow medium. It might make
sense to move it out of standard and elsewhere, as I don't think it's
necessarily needed everywhere, such as laptops.

Consider my laptop, fairly standard, 512 GB NVME SSD, about 250G allocated,
containing about 1435134 files. mlocate foo takes 1s, find / -mount
-name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
(but there is a davfs mount of an internet server, so things might be
screwed up a bit).

19s to find something is perfectly workable, also you don't usually
find from /, but you have an idea where things are, so it will be much
faster.

I think mlocate only really makes sense on data storage servers with
huge disks, or on machines with HDDs. I therefore do not think the
overhead of building the index is warranted for most users. It might
make sense to keep mlocate in always-on tasks, like servers, but get
rid of it from desktop scenarios.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Andrea Corbellini-4
On Wed, May 22, 2019 at 8:47 PM Julian Andres Klode <[hidden email]> wrote:
On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> I use mlocate multiple times a day.
> Find is way too slow and inconvenient for finding files in a big
> set of filesystems, compared to properly configuring mlocate.

Specifically, the filesystem must be huge or on a slow medium. It might make
sense to move it out of standard and elsewhere, as I don't think it's
necessarily needed everywhere, such as laptops.

There's also the case of a blazing-fast medium, but high I/O load.
That's the case that I'm often facing when working on production servers. Using find there is very frustrating.

Consider my laptop, fairly standard, 512 GB NVME SSD, about 250G allocated,
containing about 1435134 files. mlocate foo takes 1s, find / -mount
-name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
(but there is a davfs mount of an internet server, so things might be
screwed up a bit).

19s to find something is perfectly workable, also you don't usually
find from /, but you have an idea where things are, so it will be much
faster.

19s for my standards are a lot, especially if I have to do multiple searches.
There have been cases where I dumped the output of "find /" into a file and resorted to search using grep.
 

I think mlocate only really makes sense on data storage servers with
huge disks, or on machines with HDDs. I therefore do not think the
overhead of building the index is warranted for most users. It might
make sense to keep mlocate in always-on tasks, like servers, but get
rid of it from desktop scenarios.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Marcos Alano
I not search things all the time, but when I do I use find-pipe-grep. Not because "locate" is an inferior solution, not at all, just a matter of habit.

On Wed, May 22, 2019 at 5:09 PM Andrea Corbellini <[hidden email]> wrote:
On Wed, May 22, 2019 at 8:47 PM Julian Andres Klode <[hidden email]> wrote:
On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> I use mlocate multiple times a day.
> Find is way too slow and inconvenient for finding files in a big
> set of filesystems, compared to properly configuring mlocate.

Specifically, the filesystem must be huge or on a slow medium. It might make
sense to move it out of standard and elsewhere, as I don't think it's
necessarily needed everywhere, such as laptops.

There's also the case of a blazing-fast medium, but high I/O load.
That's the case that I'm often facing when working on production servers. Using find there is very frustrating.

Consider my laptop, fairly standard, 512 GB NVME SSD, about 250G allocated,
containing about 1435134 files. mlocate foo takes 1s, find / -mount
-name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
(but there is a davfs mount of an internet server, so things might be
screwed up a bit).

19s to find something is perfectly workable, also you don't usually
find from /, but you have an idea where things are, so it will be much
faster.

19s for my standards are a lot, especially if I have to do multiple searches.
There have been cases where I dumped the output of "find /" into a file and resorted to search using grep.
 

I think mlocate only really makes sense on data storage servers with
huge disks, or on machines with HDDs. I therefore do not think the
overhead of building the index is warranted for most users. It might
make sense to keep mlocate in always-on tasks, like servers, but get
rid of it from desktop scenarios.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel


--
Marcos H. Alano
Linux System Administrator
[hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Steve Langasek-6
In reply to this post by Neal McBurnett
Hi Neal,

On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> I use mlocate multiple times a day.
> Find is way too slow and inconvenient for finding files in a big
> set of filesystems, compared to properly configuring mlocate.

Does "properly configuring mlocate" mean you are using something other than
the default, generic config?  If you are already configuring the mlocate
package on your systems, it doesn't seem onerous to also have to install the
package because it's not installed by default; do you agree?


> It also seems that the bugs should be addressed (and have in some
> cases been addressed) whether or not find is installed by default.
>
> Thanks,
>
> Neal McBurnett                 http://neal.mcburnett.org/
>
> On Wed, May 22, 2019 at 11:59:57AM -0700, Brian Murray wrote:
> > The Ubuntu Foundations team was recently looking at an issue with
> > mlocate[1] and the effect it has on all users of Ubuntu. While that
> > specific issue is fixable there are also issues[2,3] with keeping
> > PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
> > questioning the usefulness of installing mlocate by default on systems
> > at all. We believe that find is an adequate replacement for mlocate but
> > want to hear from you about use cases where it may not be. I'll start
> > with a personal example:
> >
> > "I don't remember (because I need to know so infrequently) where the
> > meta-release file is cached on disk by update-manager and use locate to
> > find it. The find command itself is inadequate because the cached file
> > exists in both /home and /var."
> >
> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880507
> > [2] http://launchpad.net/bugs/827841
> > [3] http://launchpad.net/bugs/1823518
> >
> > Thanks,
> > --
> > Brian Murray
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Seth Arnold
In reply to this post by Julian Andres Klode
On Wed, May 22, 2019 at 09:47:22PM +0200, Julian Andres Klode wrote:
> I think mlocate only really makes sense on data storage servers with
> huge disks, or on machines with HDDs. I therefore do not think the
> overhead of building the index is warranted for most users. It might
> make sense to keep mlocate in always-on tasks, like servers, but get
> rid of it from desktop scenarios.

In my early days of using Linux, I used locate dozens of times a day. I
might know the filename but not the pathname, or a part of a filename,
etc.

"locate XF86Config" was way easier than "find / -name '*XF86Config*' -print".

Sure, the find command is simpler today, but it still spews loads of
useless error messages unless you also add 2> /dev/null. (And maybe
you care about some but not all errors. Unlikely but possible.)

Now that I'm far more familiar with where files live I no longer
use locate for this purpose. Now I fall firmly in the other camp,
where locate is annoyingly slow and I will try my hand at writing a
replacement someday:

$ time locate thisdoesntexist

real 1m29.294s
user 1m27.649s
sys 0m1.644s

Anyway, I believe locate can have great value to all our users,
experienced or brand new, huge systems or small systems. I'd like
us to keep it in default installs.

Thanks

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Gunnar Hjalmarsson
In reply to this post by Brian Murray-5
On 2019-05-22 20:59, Brian Murray wrote:
> So we ended up questioning the usefulness of installing mlocate by
> default on systems at all. We believe that find is an adequate
> replacement for mlocate but want to hear from you about use cases
> where it may not be.

As a volunteer contributor and involved in support, I tend to use
locate() very often, and find() so seldom that I have to look up the
syntax everytime. If mlocate would be dropped from default, it would be
one of the first package I'd install when doing a fresh install.

Suitable to be added to ubuntu-dev-tools?

--
Gunnar Hjalmarsson
https://launchpad.net/~gunnarhj

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Bryan Quigley-2
In reply to this post by Brian Murray-5
The mentioned Debian bug #880507 is one of the advantages of moving from cron to systemd units -see proposed unit file in [1].  The timers in systemd are really perfect for this...

I use locate all the time.  I really like how much simpler it is then find.  I don't have to think to usually get what I'm looking for.  I'm going to experiment with using grep <filename> /var/lib/dpkg/info/*.list as the majority of files I run locate for are from packages..

From a product perspective it does make sense to remove:
 * Desktop has tracker installed by default.  The majority of users will use that instead as it's integrated with the GUI.
 * The majority of cloud/server deployments aren't interactive these days.  It's a waste there. 

They key group I can see wanting it are developers - who can install it.

My 2 cents,
Bryan



On Wed, May 22, 2019 at 7:09 PM Brian Murray <[hidden email]> wrote:
The Ubuntu Foundations team was recently looking at an issue with
mlocate[1] and the effect it has on all users of Ubuntu. While that
specific issue is fixable there are also issues[2,3] with keeping
PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
questioning the usefulness of installing mlocate by default on systems
at all. We believe that find is an adequate replacement for mlocate but
want to hear from you about use cases where it may not be. I'll start
with a personal example:

"I don't remember (because I need to know so infrequently) where the
meta-release file is cached on disk by update-manager and use locate to
find it. The find command itself is inadequate because the cached file
exists in both /home and /var."

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880507
[2] http://launchpad.net/bugs/827841
[3] http://launchpad.net/bugs/1823518

Thanks,
--
Brian Murray

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Julian Andres Klode
On Wed, May 22, 2019 at 11:25:45PM -0700, Bryan Quigley wrote:

> The mentioned Debian bug #880507 is one of the advantages of moving from
> cron to systemd units -see proposed unit file in [1].  The timers in
> systemd are really perfect for this...
>
> I use locate all the time.  I really like how much simpler it is then
> find.  I don't have to think to usually get what I'm looking for.  I'm
> going to experiment with using grep <filename> /var/lib/dpkg/info/*.list as
> the majority of files I run locate for are from packages..
>
> From a product perspective it does make sense to remove:
>  * Desktop has tracker installed by default.  The majority of users will
> use that instead as it's integrated with the GUI.

Did this change? It's not installed by default in bionic; and no meta
package depends on it, but it seems to have moved to main in disco.

I think both tracker and locate can offer a terrible user experience for
a lot of users, due to I/O load after resume/boot; but tracker also has
a ton of CPU usage extracting stuff from documents and putting the text
into its xapian (?) db.

>  * The majority of cloud/server deployments aren't interactive these days.
> It's a waste there.

ack - also, it sucks to have this I/O load once a day.

>
> They key group I can see wanting it are developers - who can install it.

ack


--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Jamie Strandboge-3
In reply to this post by Seth Arnold
On Wed, 22 May 2019, Seth Arnold wrote:

> Anyway, I believe locate can have great value to all our users,
> experienced or brand new, huge systems or small systems. I'd like
> us to keep it in default installs.
>
IMO, find has its place and so does mlocate. I still use locate myself fairly
regularly (perhaps a couple/few times a month?) on my laptop and I have been
known to adjust PRUNEPATHS. As said by another elsewhere, even if it was moved
out of standard we would still want to address the bugs so I'd just assume it
stay in standard.

--
Jamie Strandboge             | http://www.canonical.com

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Steve Langasek-6
In reply to this post by Julian Andres Klode
Hi Jim,

On Wed, May 22, 2019 at 03:51:14PM -0400, jim wrote:

> Seems to me for an individual user such
> as I (my laptop runs Ubuntu 18.04), locate
> and (the cumbersome) find are sufficient.
> To what extent does decision-making balance
> net ops and large system administrators as
> opposed to individual users?

Currently, this is seeded as part of ubuntu-standard, which is common to all
Ubuntu flavors.  If the discussion led to a determination that this still
made sense to include by default on desktops but not servers, or vice versa,
we could seed this as part of one Ubuntu flavor or the other.

My own sense is that this is not a server vs desktop thing; there are users
of locate, to be sure, but I believe they are a very small minority on both
desktop and server (small on desktop because the user will generally use the
gui instead; small on server because most server use is not interactive at
the shell).  I don't think the benefit of having locate available by default
justifies the daily disk thrashing / energy usage on every Ubuntu machine
everywhere.  I think it's not onerous for those who want to use locate to
manually install it the first time they need it on a machine.

But this thread is here to be a sanity check on that opinion, to understand
if the locate db being available by default either affects so many users, or
has so great an impact on some use cases, that we should consider leaving it
enabled by default despite the daily disk usage for everyone not using it.

> On 5/22/19 3:47 PM, Julian Andres Klode wrote:
> > On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> > > I use mlocate multiple times a day.
> > > Find is way too slow and inconvenient for finding files in a big
> > > set of filesystems, compared to properly configuring mlocate.
> > Specifically, the filesystem must be huge or on a slow medium. It might make
> > sense to move it out of standard and elsewhere, as I don't think it's
> > necessarily needed everywhere, such as laptops.
> >
> > Consider my laptop, fairly standard, 512 GB NVME SSD, about 250G allocated,
> > containing about 1435134 files. mlocate foo takes 1s, find / -mount
> > -name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
> > (but there is a davfs mount of an internet server, so things might be
> > screwed up a bit).
> >
> > 19s to find something is perfectly workable, also you don't usually
> > find from /, but you have an idea where things are, so it will be much
> > faster.
> >
> > I think mlocate only really makes sense on data storage servers with
> > huge disks, or on machines with HDDs. I therefore do not think the
> > overhead of building the index is warranted for most users. It might
> > make sense to keep mlocate in always-on tasks, like servers, but get
> > rid of it from desktop scenarios.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Sebastien Bacher-4
In reply to this post by Julian Andres Klode
Le 22/05/2019 à 21:47, Julian Andres Klode a écrit :
> containing about 1435134 files. mlocate foo takes 1s, find / -mount
> -name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
> (but there is a davfs mount of an internet server, so things might be

Well here on a few years old 80GB intel ssd doing

$ time find / -mount -name disco-desktop-amd64.iso

Takes over 30 seconds (on a machine which not doing any other work)
where mlocate takes around 1 second on the same machine... and I do
personally find 30 seconds to be too long.

Note that nautilus has a mlocate search provider which we have been
using by default until bionic, to avoid depending on tracker. Now we use
tracker so it's less important but some users do uninstall tracker
because they don't like having an indexer (and it still does have some
bugs) so it's nice to have the UI still responsive for those users.

Cheers,
Sebastien Bacher




--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Leroy Tennison-2
In reply to this post by Steve Langasek-6

I don't know if anyone has said it already but I want to applaud you for asking rather than just doing.


From: ubuntu-server <[hidden email]> on behalf of Steve Langasek <[hidden email]>
Sent: Thursday, May 23, 2019 12:48:35 PM
To: jim
Cc: [hidden email]; [hidden email]
Subject: [EXTERNAL] Re: mlocate - what is it good for?
 
Hi Jim,

On Wed, May 22, 2019 at 03:51:14PM -0400, jim wrote:

> Seems to me for an individual user such
> as I (my laptop runs Ubuntu 18.04), locate
> and (the cumbersome) find are sufficient.
> To what extent does decision-making balance
> net ops and large system administrators as
> opposed to individual users?

Currently, this is seeded as part of ubuntu-standard, which is common to all
Ubuntu flavors.  If the discussion led to a determination that this still
made sense to include by default on desktops but not servers, or vice versa,
we could seed this as part of one Ubuntu flavor or the other.

My own sense is that this is not a server vs desktop thing; there are users
of locate, to be sure, but I believe they are a very small minority on both
desktop and server (small on desktop because the user will generally use the
gui instead; small on server because most server use is not interactive at
the shell).  I don't think the benefit of having locate available by default
justifies the daily disk thrashing / energy usage on every Ubuntu machine
everywhere.  I think it's not onerous for those who want to use locate to
manually install it the first time they need it on a machine.

But this thread is here to be a sanity check on that opinion, to understand
if the locate db being available by default either affects so many users, or
has so great an impact on some use cases, that we should consider leaving it
enabled by default despite the daily disk usage for everyone not using it.

Harriscomputer

Leroy Tennison
Network Information/Cyber Security Specialist
E: [hidden email]


2220 Bush Dr
McKinney, Texas
75070
www.datavoiceint.com
 

This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. These companies are listed here.

If you prefer not to be contacted by Harris Operating Group please notify us.

 

This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.

 

> On 5/22/19 3:47 PM, Julian Andres Klode wrote:
> > On Wed, May 22, 2019 at 01:19:48PM -0600, Neal McBurnett wrote:
> > > I use mlocate multiple times a day.
> > > Find is way too slow and inconvenient for finding files in a big
> > > set of filesystems, compared to properly configuring mlocate.
> > Specifically, the filesystem must be huge or on a slow medium. It might make
> > sense to move it out of standard and elsewhere, as I don't think it's
> > necessarily needed everywhere, such as laptops.
> >
> > Consider my laptop, fairly standard, 512 GB NVME SSD, about 250G allocated,
> > containing about 1435134 files. mlocate foo takes 1s, find / -mount
> > -name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
> > (but there is a davfs mount of an internet server, so things might be
> > screwed up a bit).
> >
> > 19s to find something is perfectly workable, also you don't usually
> > find from /, but you have an idea where things are, so it will be much
> > faster.
> >
> > I think mlocate only really makes sense on data storage servers with
> > huge disks, or on machines with HDDs. I therefore do not think the
> > overhead of building the index is warranted for most users. It might
> > make sense to keep mlocate in always-on tasks, like servers, but get
> > rid of it from desktop scenarios.


--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Steve Langasek-6
In reply to this post by Sebastien Bacher-4
On Thu, May 23, 2019 at 08:25:20PM +0200, Sebastien Bacher wrote:
> Le 22/05/2019 à 21:47, Julian Andres Klode a écrit :
> > containing about 1435134 files. mlocate foo takes 1s, find / -mount
> > -name '*foo*' takes about 7-9 secs, or 19 seconds with all mount points
> > (but there is a davfs mount of an internet server, so things might be

> Well here on a few years old 80GB intel ssd doing

> $ time find / -mount -name disco-desktop-amd64.iso

> Takes over 30 seconds (on a machine which not doing any other work)
> where mlocate takes around 1 second on the same machine... and I do
> personally find 30 seconds to be too long.

I agree that 30 seconds is too long, but as a user of find I find this usage
surprising, why would you be searching the whole root disk for the file?  I
usually know a file like that is going to be in one of a few subtrees (my
homedir or /tmp or /var/lib/libvirt or something) and would be doing a more
targeted search.

Also since the locate database is only updated once a day, locate doesn't
help with this if the file I've lost is one I've downloaded today, and it
seems likely to me that files I'm looking for that aren't part of the system
data are fairly likely to fall into this category.

Anyway, agreed that 30s is too long.  Would you be annoyed if the solution
to that 30s being too long was "get tired of waiting, run 'locate', get told
it's not installed, apt install mlocate, run 'locate' again"?

> Note that nautilus has a mlocate search provider which we have been
> using by default until bionic, to avoid depending on tracker. Now we use
> tracker so it's less important but some users do uninstall tracker
> because they don't like having an indexer (and it still does have some
> bugs) so it's nice to have the UI still responsive for those users.

Well, I don't think this is an argument for keeping mlocate installed by
default on desktops, because effectively this means that you have TWO
indexers on your desktop system - both tracker and mlocate.  It looks like
nautilus currently depends on tracker, so I'm not sure how one would
uninstall it and usefully fall back to the mlocate backend anyway, but at
most I'd say this should be expressed as 'Depends: tracker | mlocate' in
nautilus, and not have mlocate kept around on the system updating its
database daily just in case a user removes tracker.

Thanks,
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Michael Hudson-Doyle
In reply to this post by Brian Murray-5
On Thu, 23 May 2019 at 09:07, Brian Murray <[hidden email]> wrote:
The Ubuntu Foundations team was recently looking at an issue with
mlocate[1] and the effect it has on all users of Ubuntu. While that
specific issue is fixable there are also issues[2,3] with keeping
PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
questioning the usefulness of installing mlocate by default on systems
at all. We believe that find is an adequate replacement for mlocate but
want to hear from you about use cases where it may not be.

I think this is a cattle vs pet sort of thing as well: on a system that being used for development or adminned by multiple humans, locate is quite useful. And the overhead of having to install it for systems like this seems acceptable (users like these are exactly the sort of users for whom installing a package is easy!).

On systems that are part of a fleet of automatically maintained machines, all locate does is waste IO bandwidth -- which might well be a shared resource if the systems are VMs.

My vote would be for removing it from the default install.

Cheers,
mwh

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Sebastien Bacher-4
In reply to this post by Steve Langasek-6
Hey Steve,

Le 24/05/2019 à 00:33, Steve Langasek a écrit :
>> Takes over 30 seconds (on a machine which not doing any other work)
>> where mlocate takes around 1 second on the same machine... and I do
>> personally find 30 seconds to be too long.
> I agree that 30 seconds is too long, but as a user of find I find this usage
> surprising, why would you be searching the whole root disk for the file?

I just used the same example than Julian gave so we would compare the
same things (but picked a file that I know existed on my disk), I agree
that the command/example doesn't make much sense.

> Anyway, agreed that 30s is too long.  Would you be annoyed if the solution
> to that 30s being too long was "get tired of waiting, run 'locate', get told
> it's not installed, apt install mlocate, run 'locate' again"?

No, I think it would be fine. Locate isn't going to be useful on "throw
away" installations/VMs since on a fresh install the index is not going
to be there, and if you have an installed machine you use then
installing the machine is a one action easy action.

> indexers on your desktop system - both tracker and mlocate.  It looks like
> nautilus currently depends on tracker, so I'm not sure how one would
> uninstall it and usefully fall back to the mlocate backend anyway, but at
> most I'd say this should be expressed as 'Depends: tracker | mlocate' in
> nautilus, and not have mlocate kept around on the system updating its
> database daily just in case a user removes tracker.

Right, GNOME sort of forced our hands there since several features in
nautilus now rely on tracker so yes I agree with you, our default
Desktop indexer is tracker and it's probably good enough for most users.
Those who need mlocate can go an install it from the archive, we
probably don't need it pre-installed on the desktop image.


Cheers,
Sebastien Bacher


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Steve Langasek-6
In reply to this post by Steve Langasek-6
On Thu, May 23, 2019 at 03:13:49PM -0400, Little Girl wrote:
> Hey there,

> Steve Langasek wrote:

> >I don't think the benefit of having locate available by default
> >justifies the daily disk thrashing / energy usage on every Ubuntu
> >machine everywhere.

> Just curious, but how bad (or excessive or whatever) is this disk
> thrashing / energy usage that you mentioned?

It is difficult to answer this in aggregate because the data is hard to come
by, and it definitely varies by type of disk and by number of files on the
system.  You can get a sense of the effect on your own system by running
e.g. 'sudo iostat /dev/sda 30' in one window and 'sudo
/etc/cron.daily/mlocate' in another.  On my otherwise-idle desktop with an
SSD, that only takes 3 seconds to complete and it only reads a few hundred
MB of data off the disk (in order to open every directory and stat every
file).

I only have about 1.5 million files on my disk.

On machines with a lot more files; or machines with spinning disks instead
of SSDs; or heavily loaded servers, the impact is bound to be much higher
that 3 seconds of I/O.

And in a cloud environment running many Ubuntu instances on top of the same
storage set, the fact that this job is kicked off by cron simultaneously on
all instances can have a much more noticeable impact on server performance.
(Indeed, the daily cron jobs causing noticeable I/O storms is precisely what
led us to examine what cron jobs we have and question their utility.)

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mlocate - what is it good for?

Steve Langasek-6
In reply to this post by Gunnar Hjalmarsson
On Thu, May 23, 2019 at 04:32:24AM +0200, Gunnar Hjalmarsson wrote:
> On 2019-05-22 20:59, Brian Murray wrote:
> > So we ended up questioning the usefulness of installing mlocate by
> > default on systems at all. We believe that find is an adequate
> > replacement for mlocate but want to hear from you about use cases
> > where it may not be.

> As a volunteer contributor and involved in support, I tend to use locate()
> very often, and find() so seldom that I have to look up the syntax
> everytime. If mlocate would be dropped from default, it would be one of the
> first package I'd install when doing a fresh install.

> Suitable to be added to ubuntu-dev-tools?

My $.02, I think it's orthogonal to ubuntu-dev-tools and is not used in
support of any of the tools in that package, so should not be added there.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
12