Re: The Forest Gate raid and the alleged child porn

On Wed, 09 Aug 2006 08:19:11 GMT, Mike Scott
<usenet.11@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:

Yup. Watch out for those compressed images though! Up until a few
months ago I used to make a compressed image of my boot drive and burn
it to 3 DVD's. I had a HDD crash, and did a restore. Unfortunately
there was a read error on one of the DVDs, which meant that *none* of
the image could be restored. I had a much earlier backup, but that
was missing some applications and data that caused me a bit of grief.

I think we progress backwards sometimes. Way back when VAX/VMS was
/the/ thing, the VMS backup program would write to tape redundant blocks
(typically 1 in 10, but user-defined) derived by xor'ing groups of the
other blocks. I haven't noticed anything similar recently. You could cut
chunks out of the tape and still get your data back. Maybe we assume
too much reliability these days. Anyone for DVD-RAID? :-)

There are far better error recovery techniques than parity bits and
longitudinal redundacy checks that have since been invented. The
problem with parity bits (XORing data) is that you have to have a heck
of a lot of parity bytes to be able to recover from multiple
corruptions. Parity and LRC add a data overhead of well over 10%, and
can only recover a *single* corrupt bit in each protected block.

Error *detection* is pretty easy. Efficient error *correction* is far
more problematic.

The Solomon-Reed algorithm is an absolutely brilliant piece of
thinking that in essence enables any number of missing or corrupt data
bytes to be detected and recovered anywhere in the data, provided that
there are an approximately equal number of error correction bytes
available as there are corrupt bytes. So adding just over 5KB to the
end of a 500MB file (a 0.001% overhead) will enable the recovery of
*any* 10 random bad sectors anywhere in the file, for example.

So what I *should* have done (or what the programmer of the imaging
software should have done) is to have added an extra few percent of
error correction blocks to the end of each image. There is a freeware
utility called "Quickpar" that is able to both create such error
correction blocks, and use them to repair corrupt files.

The reason that such techniques are usually ignored is that the
reliability of storage media has improved so much since the magnetic
tape days that errors seldom occur.

--
Cynic

.

Relevant Pages

  • Re: repairing corrupt mp4 (and jpeg) video files
    ... and you can often squeeze a few extra pictures and some short video ... is *not* to write any further files on the card. ... successful file recovery is good, but not guaranteed, unfortunately, ... as some individual files may be corrupt even though the file system is ...
    (rec.video.desktop)
  • Re: repairing corrupt mp4 (and jpeg) video files
    ... and you can often squeeze a few extra pictures and some short video ... is *not* to write any further files on the card. ... successful file recovery is good, but not guaranteed, unfortunately, ... as some individual files may be corrupt even though the file system is ...
    (rec.video.desktop)
  • Re: RAR under linux: any alternative?
    ... > That's why you include a checksums file with the archive set and/or list ... > the sums at the point of origin. ... Added support of so called recovery volumes, ... Say the part of the archive that is corrupt is a file that isn't needed. ...
    (Debian-User)
  • Re: Word generates hidden files of the form ~$*.doc
    ... and have purchased third party Word doc recovery software to attempt ... > You should probably treat the document as corrupt. ... > Word MVP FAQ site: http://word.mvps.org ... >> correctly (replacing references with the message"Error! ...
    (microsoft.public.word.application.errors)
  • Re: Help with problem possibly caused by Norton SystemWorks
    ... Norton System Works Support: ... For the error message: ... Go to Recovery Console and type: ... > It is corrupt, absent, or not writable", so I couldn't do ...
    (microsoft.public.windowsxp.general)