Odd need: to track down NUL bytes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Odd need: to track down NUL bytes

Kevin O'Gorman
I've never run into this before, despite running Unix or Linux in my home continuously since about 1986.  I had a system freeze-up on Jan 31 from unknown causes.  The system affected is run as a computing server for a hobby project of mine.  I thought all was well, and that I was able to salvage the partial outputs of the jobs that were running.  I was wrong.

I now discover that some files that were under construction at that time are corrupted with extensive NUL bytes.  These are usually text files.  I want to find all such files and "cleanse" them to recover what information I can.  With just about anything _except_ a NUL byte this would be easy, but due to the special treatment of NUL in strings, it's not that clear to me how best to even search for those bytes.

Ideas?

--
Kevin O'Gorman
#define QUESTION ((bb) || (!bb))   /* Shakespeare */

Please consider the environment before printing this email.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Odd need: to track down NUL bytes

Nils Kassube-2
Kevin O'Gorman wrote:
> I now discover that some files that were under construction at that
> time are corrupted with extensive NUL bytes.  These are usually text
> files.  I want to find all such files and "cleanse" them to recover
> what information I can.  With just about anything _except_ a NUL byte
> this would be easy, but due to the special treatment of NUL in
> strings, it's not that clear to me how best to even search for those
> bytes.

Well, I also don't know how to search for those files, but you could at
least remove the NUL bytes with something like this:

cat infile | tr -d '\0' >outfile

Or perhaps you prefer to replace the NUL bytes with something else, like
space - that would be like this:

cat infile | tr '\0' ' ' >outfile

Now, you could just do this operation for all files, regardles if they
contain NUL bytes:

for f in *;do test -f "$f" && cat "$f" | tr '\0' ' ' >"outdir/$f";done

This would place a copy of each file in the existing folder "outdir"
with the NUL bytes replaced by spaces (if there are any).


Nils


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Odd need: to track down NUL bytes

Kevin O'Gorman
Thanks, I chose a slightly different way to go
1. I wrote a very small program to copy everything up to the first NUL byte, and report if anything after that was non-NUL (nothing was).
2. I ran it on all files, with output to a different directory
3. Checked that the last line was intact in each file (all but 2 were fine).
4. Computed what work was finished and what was not.
5. Resumed where it left off.

It had run for about 60 days before the crash.  Each line is a separate problem solved, and each takes at most 30 minutes.  Yes that means I have millions of problems and answers.  It's a big hobby.  I salvaged a lot of work this way.



On Sun, Feb 5, 2017 at 11:35 AM, Nils Kassube <[hidden email]> wrote:
Kevin O'Gorman wrote:
> I now discover that some files that were under construction at that
> time are corrupted with extensive NUL bytes.  These are usually text
> files.  I want to find all such files and "cleanse" them to recover
> what information I can.  With just about anything _except_ a NUL byte
> this would be easy, but due to the special treatment of NUL in
> strings, it's not that clear to me how best to even search for those
> bytes.

Well, I also don't know how to search for those files, but you could at
least remove the NUL bytes with something like this:

cat infile | tr -d '\0' >outfile

Or perhaps you prefer to replace the NUL bytes with something else, like
space - that would be like this:

cat infile | tr '\0' ' ' >outfile

Now, you could just do this operation for all files, regardles if they
contain NUL bytes:

for f in *;do test -f "$f" && cat "$f" | tr '\0' ' ' >"outdir/$f";done

This would place a copy of each file in the existing folder "outdir"
with the NUL bytes replaced by spaces (if there are any).


Nils


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users



--
Kevin O'Gorman
#define QUESTION ((bb) || (!bb))   /* Shakespeare */

Please consider the environment before printing this email.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users