out of space on /root

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
85 messages Options
12345
Bob
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

out of space on /root

Bob
Using Ubuntu 16.10.  The system ran out of space on /root.  I found that
/var/log/syslog was 27gb in size.  I deleted all the syslog.* files and got the
system sort of working and copied /var/log/syslog to my home directory (/home
is on a different partition than /root).  I then deleted the syslog file and
rebooted.  Now things are working OK.  I would like to find out what caused
this problem but can not look at the syslog file as it is too large.  Any ideas
on how to display this file?

--
Robert Blair


A liberal is someone who feels a great debt to his fellow man, which debt he proposes to pay off with your money.  -- G. Gordon Liddy

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
Bob schreef op 06-03-2017 7:33:

> Using Ubuntu 16.10.  The system ran out of space on /root.  I found
> that
> /var/log/syslog was 27gb in size.  I deleted all the syslog.* files and
> got the
> system sort of working and copied /var/log/syslog to my home directory
> (/home
> is on a different partition than /root).  I then deleted the syslog
> file and
> rebooted.  Now things are working OK.  I would like to find out what
> caused
> this problem but can not look at the syslog file as it is too large.  
> Any ideas
> on how to display this file?

In general commands such as:

cat "file" | tail -n 500

will get you the last 500 lines.

You can make that 500 figure as large as you want.

Maybe with a file so large a better command would be

tail -n 500 "file" so that tail can do random seeks, but I'm not sure.


The above command will read the entire file from disk before tail will
filter out the last 500 lines.

What you can easily do is to use the "split" command to first reduce it
into smaller bits.

The idiomatic expression:


cat "file" | split -b 500M

would do the trick, although again, you can better directly name a file:


split -b 500M "filename".

Both should work (tail and split with a direct filename). If you use
split, you will need the same amount of space you are already using.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Simos Xenitellis-4
On Mon, Mar 6, 2017 at 8:53 AM, Xen <[hidden email]> wrote:

> Bob schreef op 06-03-2017 7:33:
>>
>> Using Ubuntu 16.10.  The system ran out of space on /root.  I found that
>> /var/log/syslog was 27gb in size.  I deleted all the syslog.* files and
>> got the
>> system sort of working and copied /var/log/syslog to my home directory
>> (/home
>> is on a different partition than /root).  I then deleted the syslog file
>> and
>> rebooted.  Now things are working OK.  I would like to find out what
>> caused
>> this problem but can not look at the syslog file as it is too large.  Any
>> ideas
>> on how to display this file?
>
>
> In general commands such as:
>
> cat "file" | tail -n 500
>
> will get you the last 500 lines.

Preferably, you can write straight ahead

tail -100 /var/log/syslog

What does it do?

1. "tail" shows the last "n" lines of a text file.
2. "-100" and "-n 100" are equivalent, both show the last 100 lines.
Substitute with any other number.
3. I think it is more performant not to start with "cat" because "cat"
would go reading through all the file
before "tail" will be able to prune all but the last 100 lines.
Perhaps "tail" has some optimizations to ignore reading the majority
of the 27GB syslog,
so it is better to to just "tail".

Simos

>
> You can make that 500 figure as large as you want.
>
> Maybe with a file so large a better command would be
>
> tail -n 500 "file" so that tail can do random seeks, but I'm not sure.
>
>
> The above command will read the entire file from disk before tail will
> filter out the last 500 lines.
>
> What you can easily do is to use the "split" command to first reduce it into
> smaller bits.
>
> The idiomatic expression:
>
>
> cat "file" | split -b 500M
>
> would do the trick, although again, you can better directly name a file:
>
>
> split -b 500M "filename".
>
> Both should work (tail and split with a direct filename). If you use split,
> you will need the same amount of space you are already using.
>
>
> --
> ubuntu-users mailing list
> [hidden email]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
Simos Xenitellis schreef op 06-03-2017 11:57:

>> In general commands such as:
>>
>> cat "file" | tail -n 500
>>
>> will get you the last 500 lines.
>
> Preferably, you can write straight ahead
>
> tail -100 /var/log/syslog

Yes, that's what I said. But his files are no longer in /var/log, so
that doesn't help.

> 3. I think it is more performant not to start with "cat" because "cat"
> would go reading through all the file

Yes, that's what I said.

> before "tail" will be able to prune all but the last 100 lines.
> Perhaps "tail" has some optimizations to ignore reading the majority
> of the 27GB syslog,

Yes, that's what I said.

> so it is better to to just "tail".

Yes, that's what I said ;-).

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Oliver Grawert
hi,
On Mo, 2017-03-06 at 14:10 +0100, Xen wrote:

>
> Simos Xenitellis schreef op 06-03-2017 11:57:
>
> >
> > >
> > > In general commands such as:
> > >
> > > cat "file" | tail -n 500
> > >
> > > will get you the last 500 lines.
> >
> > Preferably, you can write straight ahead
> >
> > tail -100 /var/log/syslog
>
> Yes, that's what I said. But his files are no longer in /var/log, so 
> that doesn't help.
>
no, you said to cat the file and then tail the result ... which means
reading the whole file into ram and pipe the output to read the last
500 lines ... while simos suggested to read the last 500 lines (and
only these) directly from the file (admittedly from the wrong place).

ciao
        oli
--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Ken D'Ambrosio
In reply to this post by Xen
This is a very rudimentary attempt to strip out stuff (timestamp, PID)
that would be largely unique per-line, and then pull out the most
frequently recurring messages:

export LOG=/var/log/syslog
cat $LOG | perl -e 'while (<>){s/^.{16}//; s/\[[0-9]+]//;print;}' | sort
| uniq -c | sort -n

Now, 27 GB would take a L-O-N-G time to sort through, so instead of "cat
$LOG" I'd probably change it to something like "tail -10000 $LOG".  (I
only bother with stuffing the logfile into $LOG so that it will fit in
the constraints of my ASCII mail compose window; apologies.)  The output
will look something like this:

[lots of onesy-twosy stuff here]
      84 foo spamd: prefork: child states: II
      86 foo spamd: spamd: setuid to spamd succeeded
     153 foo dovecot: imap(ken): Logged out in=94 out=987

The number in the left-hand column is the number of times that
particular message occurred.  You could pipe everything through "tail
-20" or somesuch, just to grab the bottom of the output, with,
presumably, the most relevant errors/messages.

Good luck!

-Ken


On 2017-03-06 08:10, Xen wrote:

> Simos Xenitellis schreef op 06-03-2017 11:57:
>
>>> In general commands such as:
>>>
>>> cat "file" | tail -n 500
>>>
>>> will get you the last 500 lines.
>>
>> Preferably, you can write straight ahead
>>
>> tail -100 /var/log/syslog
>
> Yes, that's what I said. But his files are no longer in /var/log, so
> that doesn't help.
>
>> 3. I think it is more performant not to start with "cat" because "cat"
>> would go reading through all the file
>
> Yes, that's what I said.
>
>> before "tail" will be able to prune all but the last 100 lines.
>> Perhaps "tail" has some optimizations to ignore reading the majority
>> of the 27GB syslog,
>
> Yes, that's what I said.
>
>> so it is better to to just "tail".
>
> Yes, that's what I said ;-).

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Paul Smith-2
In reply to this post by Xen
On Mon, 2017-03-06 at 14:10 +0100, Xen wrote:
> > so it is better to to just "tail".
>
> Yes, that's what I said ;-).

I think for helping new people it's better to provide the best answer
only and leave it at that, rather than provide a not-so-good answer
first and then say "maybe it would be better to do XYZ"... :)

The other example in this email is also a UUOC:

> cat "file" | split -b 500M

can be more efficiently written:

  split -b 500M < "file"

In general, cat can almost always be avoided giving a more efficient
(sometimes GREATLY more efficient depending on the situation, other
times just a little bit more efficient) result.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
In reply to this post by Oliver Grawert
Oliver Grawert schreef op 06-03-2017 15:23:

> hi,
> On Mo, 2017-03-06 at 14:10 +0100, Xen wrote:
>>
>> Simos Xenitellis schreef op 06-03-2017 11:57:
>>
>> >
>> > >
>> > > In general commands such as:
>> > >
>> > > cat "file" | tail -n 500
>> > >
>> > > will get you the last 500 lines.
>> >
>> > Preferably, you can write straight ahead
>> >
>> > tail -100 /var/log/syslog
>>
>> Yes, that's what I said. But his files are no longer in /var/log, so 
>> that doesn't help.
>>
>
> no, you said to cat the file and then tail the result ... which means
> reading the whole file into ram and pipe the output to read the last
> 500 lines ... while simos suggested to read the last 500 lines (and
> only these) directly from the file (admittedly from the wrong place).

Do I really have to quote?

> Maybe with a file so large a better command would be
>
> tail -n 500 "file" so that tail can do random seeks, but I'm not sure.

Why? Because the above command, namely

cat "file" | tail -n 500

will read the entire file from disk before tail will filter out the last
500 lines.

So I never said to cat the file, I said:

*In general* "cat "file" | tail -n 500 will get you the last 500 lines."
so that's not what I advised. That is the general form. That's the
idiomatic expression. Then I said "Maybe with a file so large" you had
better name the file directly.

Is it so hard to read?

It says it verbatim.

Down below, I say again:

"would do the trick, although again, you can better directly name a
file:"

split -b 500M "filename".

And:

"Both should work (tail and split with a direct filename)."



Is it so hard to read the entire message before you start responding?

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
In reply to this post by Paul Smith-2
Paul Smith schreef op 06-03-2017 15:37:
> On Mon, 2017-03-06 at 14:10 +0100, Xen wrote:
>> > so it is better to to just "tail".
>>
>> Yes, that's what I said ;-).
>
> I think for helping new people it's better to provide the best answer
> only and leave it at that, rather than provide a not-so-good answer
> first and then say "maybe it would be better to do XYZ"... :)

Maybe, but anyone really interested in the answer instead of only
responding to 'improve' it would probably read the entire message.

And that becomes more educational as well, although I admit my wording
was a bit ambiguous.

I prefer people to do a little thinking you know, but yeah.


> The other example in this email is also a UUOC:
>
>> cat "file" | split -b 500M
>
> can be more efficiently written:
>
>   split -b 500M < "file"

I disagree. The second expression is harder to write mentally and more
prone to error. There is barely a functional difference apart from the
extra cat process, but this doesn't really take any resources. It's
either Bash doing it, or Cat, there is no other difference.

It's much more natural to use the pipe verbatim because you can easily
expand it and do a lot more with it. In fact

< "file" split -b 500M would also be perfectly the same, but you see how
unreadable it becomes.

Meaning, when you start writing your expression, the pipe syntax takes
less mental resources at the start, I mean you have to think less about
how long your expression is going to be or whatever.

It is also more natural to feed something from left to right, but your
mileage may vary.

I just disagree with your sentiment, that's all. The saving of the extra
cat process is just not worth it. I hope we can each have our own style
here.

I mean, I hope you wouldn't seriously consider writing something like

grep text < file | sed | tac > file.

That makes for very unreadable and unmanageable code.

Not every program requests the "file" parameter in the same way, so in
general cat is ubiquitously useful. It always works. Except when you
have very large seeks, very large streams resulting from the thing. You
can't seek in a pipe you know, but you knew that.

However the "cat pipe" versus "bash redirect" syntax makes generally no
difference. It is insignificant. There is still going to be a process
receiving something on stdin, it's just that Bash will be on the other
end instead of Cat.

It is also more complicated for a beginner to use the syntax you advise.

You now ask this beginner to worry about insignificant performance
gains, when no one, and certainly no beginner, will ever notice that.

So I don't mind if you use that syntax, but please don't berate others
for using the other style when it is, in fact, better.


> In general, cat can almost always be avoided giving a more efficient
> (sometimes GREATLY more efficient depending on the situation, other
> times just a little bit more efficient) result.

I would actually be curious as to what those greatly more efficient
situations would be, if you don't mind.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Paul Smith-2
On Mon, 2017-03-06 at 18:35 +0100, Xen wrote:
> >> cat "file" | split -b 500M
> > 
> > can be more efficiently written:
> > 
> >   split -b 500M < "file"
>
> I disagree. The second expression is harder to write mentally and
> more prone to error.

Those are subjective assertions, and people who who have a lot more
experience than either of us have disagreed with you for over 20 years.
I have no interest in arguing about it in general.

> There is barely a functional difference apart from the extra cat
> process, but this doesn't really take any resources. It's either Bash
> doing it, or Cat, there is no other difference.

That's not correct.  In the "with cat" method you have this:

   1. shell creates a pipe
   2. shell forks a new process A
   3. shell forks a new process B
   4. In process A, duplicate the write side of the pipe to stdout
   5. In process A, exec /usr/bin/cat "file"
   6. In process B, duplicate the read side of the pipe to stdin
   7. In process B, exec split -b 500M
   8. The "cat" process reads from the file, writes to the pipe
   9. The "split" process reads from the pipe, writes to files.
  10. Wait for A and B to finish

In this method the entire file is read into memory by one process, then
copied back into the kernel via the pipe, then that data is read by the
second process (split) and worked on.  Essentially you have the
overhead of an entire extra read and write of the file.

In the "without cat" method you have this:

   1. Shell opens "file"
   2. shell forks a new process A
   3. In process A, duplicate the "file" descriptor to stdin
   4. In process A, exec split -b 500M
   5. The "split" process reads the "file" (stdin), writes to files.
   6. Wait for A to finish

Here the split process reads directly from the file; there's no extra
copying around and the shell doesn't ever read from the file and
neither does any other extra process.

Even better would be:

  split -b 500M "file"

with no redirection at all.  In this method you get this:

   1. Shell forks a new process A
   2. In process A, exec split -b 500M "file"
   3. The split process opens "file", reads from it, writes to files
   4. Wait for A to finish

Here the output filenames would be based on "file" which is nice; some
programs can behave better if they have an actual file they can stat(2)
to find the size, etc.

The advantage to "read from the file" methods rather than the first
"read from pipe" is that a pipe is uni-directional (you can't go
backward), and it contains a maximum of 4K (on Linux) bytes at a time.

A program like "tail", if it works on a file, can go to the end of the
file and then back up from there and not have to read the beginning.  A
program like "split" can read blocks much larger than 4k at a time and
gain efficiency.  There could even be special kernel support for bulk
file IO that can be taken advantage of, which clearly can't be used
with pipes.

> So I don't mind if you use that syntax, but please don't berate
> others for using the other style when it is, in fact, better.

Please don't state your personal subjective opinions as if they were
facts.

It's true that in many cases the processing difference is not
significant, because the amount of data involved is small.  But, I
didn't comment on the GENERAL case, I commented on THIS case.  In this
case, where we're dealing with such enormous files, there is absolutely
no question that the non-cat version is far superior and the cat
version should be avoided.

My view on using cat/pipes vs. simple redirection in general disagrees
with yours, but that's an opinion (held by many, but an opinion
nonetheless).

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
Paul Smith schreef op 06-03-2017 19:42:

> On Mon, 2017-03-06 at 18:35 +0100, Xen wrote:
>> >> cat "file" | split -b 500M
>> > 
>> > can be more efficiently written:
>> > 
>> >   split -b 500M < "file"
>>
>> I disagree. The second expression is harder to write mentally and
>> more prone to error.
>
> Those are subjective assertions, and people who who have a lot more
> experience than either of us have disagreed with you for over 20 years.
> I have no interest in arguing about it in general.

Well then don't argue it. And let everyone say their own thing.

I am being attacked not you. One cannot even mention a solution without
being talked over by other people because ones solution was not perfect
enough.

Might I also add that in those 20 years the shell has not become a more
user friendly environment in any way, so by that measure they have
failed.



>> There is barely a functional difference apart from the extra cat
>> process, but this doesn't really take any resources. It's either Bash
>> doing it, or Cat, there is no other difference.
>
> That's not correct.  In the "with cat" method you have this:
>
>    1. shell creates a pipe
>    2. shell forks a new process A
>    3. shell forks a new process B
>    4. In process A, duplicate the write side of the pipe to stdout
>    5. In process A, exec /usr/bin/cat "file"
>    6. In process B, duplicate the read side of the pipe to stdin
>    7. In process B, exec split -b 500M
>    8. The "cat" process reads from the file, writes to the pipe
>    9. The "split" process reads from the pipe, writes to files.
>   10. Wait for A and B to finish
>
> In this method the entire file is read into memory by one process, then
> copied back into the kernel via the pipe, then that data is read by the
> second process (split) and worked on.  Essentially you have the
> overhead of an entire extra read and write of the file.
>
> In the "without cat" method you have this:
>
>    1. Shell opens "file"
>    2. shell forks a new process A
>    3. In process A, duplicate the "file" descriptor to stdin
>    4. In process A, exec split -b 500M
>    5. The "split" process reads the "file" (stdin), writes to files.
>    6. Wait for A to finish

So you're saying split reads directly from disk. Well didn't know that,
thank you.

Regardless, that does of course agree with the idiom of cat | filter.
There is an extra step.


> Even better would be:
>
>   split -b 500M "file"
>
> with no redirection at all.

Yes yes, of course, that was the whole point of what I said.


> Here the output filenames would be based on "file" which is nice; some
> programs can behave better if they have an actual file they can stat(2)
> to find the size, etc.
>
> The advantage to "read from the file" methods rather than the first
> "read from pipe" is that a pipe is uni-directional (you can't go
> backward), and it contains a maximum of 4K (on Linux) bytes at a time.

Yes, that's a downside. However I thought that was bigger these times?
Something like 64k? Regardless that is not a deficiency of the idiom,
but of the implementation.

Sometimes pipes are incredibly slow. But that was mostly when I tried to
do it on MS Windows using GnuWin32 tools. I tried on Linux to use the
"buffer" command but it doesn't make a lot of difference.

I ran some tests on a rather slow device (a NAS) and I guess I was wrong
about the insignificance. The "superior method" (in terms of
performance) finished roughly twice as fast (or more) than the cat
version, feeding a binary file of some 1.8G through "wc -l".

For some reason I just don't expect CPU speeds to be a bottleneck in
these operations, but on this NAS the harddisk is actually faster than
the CPU in that sense.




>
> A program like "tail", if it works on a file, can go to the end of the
> file and then back up from there and not have to read the beginning.  A
> program like "split" can read blocks much larger than 4k at a time and
> gain efficiency.  There could even be special kernel support for bulk
> file IO that can be taken advantage of, which clearly can't be used
> with pipes.

Well I just think that is a deficiency of the pipe (at least the 4k
thing), not the method.

If you make it impossible to travel from your town to the next in a
direct line, but instead have to travel half the world because of
regulations, that doesn't suddenly mean travelling half the world is now
a "superior" measure. I remains, in that case, in that sense, a sad
situation.

I know buffer sizes can have enormous impact. Compare a dd with bs=512
vs a dd with bs=4M, the difference is huge.

> Please don't state your personal subjective opinions as if they were
> facts.

Then don't do it yourself.

> It's true that in many cases the processing difference is not
> significant, because the amount of data involved is small.  But, I
> didn't comment on the GENERAL case, I commented on THIS case.  In this
> case, where we're dealing with such enormous files, there is absolutely
> no question that the non-cat version is far superior and the cat
> version should be avoided.

The non-cat version should also be avoided in my view, and you should
use the direct file method instead. So it was irrelevant here, and as
such, not to the point.


> My view on using cat/pipes vs. simple redirection in general disagrees
> with yours, but that's an opinion (held by many, but an opinion
> nonetheless).

Sure I don't mind that. But my experience is that whenever one even
mentions the alternative view, one is chastised for it.

So, I would never go out of my way to berate people for using the "<
file" syntax, however people generally do go out of their way to berate
the "cat | process" syntax.

So, the situation is not equal.


See, the point is that people ARE arguing the general case.

But since you have provided many data on the benefit of more direct
methods, let's also introduce some analogies here.

Idiomatically, the cat method is met by at least:

zcat
ccat
bzcat

and others. In using anything else, such as gzip or gunzip one must
remember exactly how to ensure that the output is going to stdout, which
is often very hard, or hard to remember. So, from a programming
perspective, there is no simple equivalent for:

grep "pattern" < file.gz

You can't do that thing. However what you can do is:

zcat file.gz | grep "pattern".

Idiomatically this barely changes from:

cat file | grep "pattern" so we have more consistency here, which is
great from a programmer's perspective, everything becomes simpler.

In general high level languages are always less performant than low
level languages, and this is a perfect analogy. The "cat" method is less
performant, but easier to use.

(Compared to the stdin redirect method, not to the direct file method).

So I don't know who those "experts" are that have been arguing this
style for 20 years, but they don't know much about programmers. Or they
only program in C themselves, which is not exactly the most user
friendly language out there. So by that standard we should not expect
user friendly shell code either.

?

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
Xen schreef op 06-03-2017 20:30:

>> It's true that in many cases the processing difference is not
>> significant, because the amount of data involved is small.  But, I
>> didn't comment on the GENERAL case, I commented on THIS case.  In this
>> case, where we're dealing with such enormous files, there is
>> absolutely
>> no question that the non-cat version is far superior and the cat
>> version should be avoided.

Also, personally...

On this little NAS device my test took some 48 seconds on average
(varying wildly) for the cat method on that 1.8G file.

It took some 22 seconds (on average, not varying as much) for the "<
file" method.

Regardless for ME getting ready to issue the correct command in the
correct place often takes longer than those 22 seconds.

After I issue that command, I'm generally not waiting until it's done. I
go do something else, or sometimes I do wait. Sometimes it matters,
sometimes it doesn't.

So generally the execution of the command itself is not the thing that
takes the most time for me. It is getting ready to do it and then
dealing with the result, which sometimes requires some thing or some
arduous work and that is dwarfed by the amount of time the command runs
in the sense that I can go do something else in the meantime, and it
doesn't bother me all that much how long the command runs.

YMMV but I don't have a huge problem with these commands taking a bit
longer. This is a very slow NAS device and this file was processed in
one minute max, more or about and that is simply not a problem to me.

However to me, making errors in my command line syntax is more important
and that is why I prefer the slower, but more robust command line syntax
that I'm used to, thank you very much and I hope that can be the same
for everyone but everyone can choose their own path right.

I have just not had an occasion that I was not criticised for using the
cat version. Even when it matters to no one else, just me, I get
criticized for it.

When for me it is just far superior the other method by any means. On
*very* rare occasions do I care enough to use the < file direct pipe
syntax (redirect syntax) in opposition of the cat syntax. VERY RARE.

So that's all that is for me, I just want to share my vision on it but
experientally this is just for me the superior method, that's all.

Thank you, bye.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Ralf Mardorf-2
In reply to this post by Paul Smith-2
Indeed, "cat" usually is redundant, but what could we do, if the log
file should be a binary?

Regarding performance there's no big difference between dash and
bash, however "tail" could replace "cat", but it can't replace
"strings". To read systemd log files, the "journalctl" command could be
used, as long as it's accessible, if not, then we need to use "strings".

[root@archlinux log]# time strings journal/*/* | tail | grep 13
_SOURCE_REALTIME_TIMESTAMP=1402628901348508

real 0m30.255s
user 0m28.657s
sys 0m1.243s
[root@archlinux log]# time dash -c "strings journal/*/* | tail | grep 13"
_SOURCE_REALTIME_TIMESTAMP=1402628901348508

real 0m30.047s
user 0m28.510s
sys 0m1.347s
[root@archlinux log]# tail journal/*/* | grep 13
Binary file (standard input) matches



--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Ralf Mardorf-2
In reply to this post by Xen
On Mon, 06 Mar 2017 20:30:41 +0100, Xen wrote:
>So you're saying split reads directly from disk. Well didn't know
>that, thank you.

You don't run all shell commands for testing purpose with strace, to see
how they exactly work?

;)


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Bob
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Bob
In reply to this post by Xen
** Reply to message from Xen <[hidden email]> on Mon, 06 Mar 2017 07:53:00
+0100

> Bob schreef op 06-03-2017 7:33:
> > Using Ubuntu 16.10.  The system ran out of space on /root.  I found
> > that
> > /var/log/syslog was 27gb in size.  I deleted all the syslog.* files and
> > got the
> > system sort of working and copied /var/log/syslog to my home directory
> > (/home
> > is on a different partition than /root).  I then deleted the syslog
> > file and
> > rebooted.  Now things are working OK.  I would like to find out what
> > caused
> > this problem but can not look at the syslog file as it is too large.  
> > Any ideas
> > on how to display this file?
>
> In general commands such as:
>
> cat "file" | tail -n 500
>
> will get you the last 500 lines.
>
> You can make that 500 figure as large as you want.
>
> Maybe with a file so large a better command would be
>
> tail -n 500 "file" so that tail can do random seeks, but I'm not sure.
>
>
> The above command will read the entire file from disk before tail will
> filter out the last 500 lines.
>
> What you can easily do is to use the "split" command to first reduce it
> into smaller bits.
>
> The idiomatic expression:
>
>
> cat "file" | split -b 500M
>
> would do the trick, although again, you can better directly name a file:
>
>
> split -b 500M "filename".
>
> Both should work (tail and split with a direct filename). If you use
> split, you will need the same amount of space you are already using.

Thanks to all that replied, that was a very interesting discussion.



After reading the man page I did a

split -C 250M -d syslog

I thought 500M was too large and I was right.  Even at 250M the file took about
a minute to read and was very slow trying to page very far.  Now that I have
looked at the file I could have used "head -n 200" to get the start and "tail
-n 5000" to get the end.  The entire rest of the 27gb of syslog was

Mar   5 17:26:55 Jupiter nautilus-autostart.desktop[3036]:
H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033

etc, each line is a little over 3kb.  These time stamps start about 11am and
ends when the system runs out of space at about 5:30pm.

I do not know much about Ubuntu but looking at the syslog the first 133 lines
look like normal log entries.  The next 5 lines are garbage and is about 13kb
long.

So my guess is that the log entry that would show the problem has been
overlayed and I will never know what caused the problem.  I will just hope that
it does not happen again.

--
Robert Blair


The only difference between a tax man and a taxidermist is that the taxidermist leaves the skin.  -- Mark Twain

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
In reply to this post by Ralf Mardorf-2
Ralf Mardorf schreef op 06-03-2017 21:01:
> On Mon, 06 Mar 2017 20:30:41 +0100, Xen wrote:
>> So you're saying split reads directly from disk. Well didn't know
>> that, thank you.
>
> You don't run all shell commands for testing purpose with strace, to
> see
> how they exactly work?
>
> ;)

I only do that when I cannot find out why something doesn't work, and
then usually I still cannot find out ;-).

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Ralf Mardorf-2
On Mon, 06 Mar 2017 22:36:23 +0100, Xen wrote:

>Ralf Mardorf schreef op 06-03-2017 21:01:
>> On Mon, 06 Mar 2017 20:30:41 +0100, Xen wrote:  
>>> So you're saying split reads directly from disk. Well didn't know
>>> that, thank you.  
>>
>> You don't run all shell commands for testing purpose with strace, to
>> see
>> how they exactly work?
>>
>> ;)  
>
>I only do that when I cannot find out why something doesn't work, and
>then usually I still cannot find out ;-).


:D

Yes, the output usually isn't helpful, at least I usually don't
understand most of the output.



--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Xen
Ralf Mardorf schreef op 07-03-2017 6:54:

> On Mon, 06 Mar 2017 22:36:23 +0100, Xen wrote:
>> Ralf Mardorf schreef op 06-03-2017 21:01:
>>> On Mon, 06 Mar 2017 20:30:41 +0100, Xen wrote:
>>>> So you're saying split reads directly from disk. Well didn't know
>>>> that, thank you.
>>>
>>> You don't run all shell commands for testing purpose with strace, to
>>> see
>>> how they exactly work?
>>>
>>> ;)
>>
>> I only do that when I cannot find out why something doesn't work, and
>> then usually I still cannot find out ;-).
>
>
> :D
>
> Yes, the output usually isn't helpful, at least I usually don't
> understand most of the output.

I once wanted to know the parameters to some function call to see what
it actually was doing, but I just couldn't find a way to display them,
as they were displayed condensely.

So I saw the function call, but no way to see the actual parameters to
it. I hunted for like hours to find the solution to that, but to no
avail ;-).

Regards :p.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
SDA
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

SDA
In reply to this post by Xen
On Mon, Mar 06, 2017 at 12:17:11PM -0800, Bob wrote:

> I do not know much about Ubuntu but looking at the syslog the first 133 lines
> look like normal log entries.  The next 5 lines are garbage and is about 13kb
> long.
>
> So my guess is that the log entry that would show the problem has been
> overlayed and I will never know what caused the problem.  I will just hope that
> it does not happen again.

Back a couple of months, Debian Sid has this problem - I resorted to running bleachbit every couple of days to prune the log files. It eventually got sorted in an upgrade later. So, perhaps same bug, since Ubuntu is usually pulled from Sid.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of space on /root

Joel Rees
In reply to this post by Xen
Ah, yes, this conversation recalls me the days when cat was a shell built-in.

Now, I, personally, would tend to use less, and "g" and "G", or vim,
and "gg" and "G". And the cursor keys.

On Tue, Mar 7, 2017 at 5:17 AM, Bob <[hidden email]> wrote:

> ** Reply to message from Xen <[hidden email]> on Mon, 06 Mar 2017 07:53:00
> +0100
>
>> Bob schreef op 06-03-2017 7:33:
>> > Using Ubuntu 16.10.  The system ran out of space on /root.  I found
>> > that
>> > /var/log/syslog was 27gb in size.  I deleted all the syslog.* files and
>> > got the
>> > system sort of working and copied /var/log/syslog to my home directory
>> > (/home
>> > is on a different partition than /root).  I then deleted the syslog
>> > file and
>> > rebooted.  Now things are working OK.  I would like to find out what
>> > caused
>> > this problem but can not look at the syslog file as it is too large.
>> > Any ideas
>> > on how to display this file?
>>
>> In general commands such as:
>>
>> cat "file" | tail -n 500
>>
>> will get you the last 500 lines.
>>
>> You can make that 500 figure as large as you want.
>>
>> Maybe with a file so large a better command would be
>>
>> tail -n 500 "file" so that tail can do random seeks, but I'm not sure.
>>
>>
>> The above command will read the entire file from disk before tail will
>> filter out the last 500 lines.
>>
>> What you can easily do is to use the "split" command to first reduce it
>> into smaller bits.
>>
>> The idiomatic expression:
>>
>>
>> cat "file" | split -b 500M
>>
>> would do the trick, although again, you can better directly name a file:
>>
>>
>> split -b 500M "filename".
>>
>> Both should work (tail and split with a direct filename). If you use
>> split, you will need the same amount of space you are already using.
>
> Thanks to all that replied, that was a very interesting discussion.
>
>
>
> After reading the man page I did a
>
> split -C 250M -d syslog
>
> I thought 500M was too large and I was right.  Even at 250M the file took about
> a minute to read and was very slow trying to page very far.  Now that I have
> looked at the file I could have used "head -n 200" to get the start and "tail
> -n 5000" to get the end.  The entire rest of the 27gb of syslog was
>
> Mar   5 17:26:55 Jupiter nautilus-autostart.desktop[3036]:
> H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033[1:1H#033
>
> etc, each line is a little over 3kb.  These time stamps start about 11am and
> ends when the system runs out of space at about 5:30pm.
>
> I do not know much about Ubuntu but looking at the syslog the first 133 lines
> look like normal log entries.  The next 5 lines are garbage and is about 13kb
> long.
>
> So my guess is that the log entry that would show the problem has been
> overlayed and I will never know what caused the problem.  I will just hope that
> it does not happen again.
>
> --
> Robert Blair
>
>
> The only difference between a tax man and a taxidermist is that the taxidermist leaves the skin.  -- Mark Twain
>
> --
> ubuntu-users mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users



--
Joel Rees

I'm imagining I'm a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html
More of my delusions:
http://reiisi.blogspot.jp/p/novels-i-am-writing.html

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
12345
Loading...