Re: I know this won't work, I just want to know -why-.

Willem wrote:
Claudio wrote:
) I am wondering about this error correction thingy all the time. I just ) can't believe it is really able to correct anything. I hadn't yet got ) the time to learn details about it or to run the huge amount of ) necessary tests to see it clearly, so please excuse, that what follows ) is what I believe, not what I know - inspite of this I hope i can be ) helpful here:
)
) In my opinion the error correction implemented in many archiving ) programs works only in case the errors detected are errors with a very ) well known characteristics determined from experience with ) characteristics of failures found on storage media. In other words, if ) the kind of error is not as expected, the corrected version of the file ) will be wrong and there is no chance to see it when the original file ) version is no more available for comparison.

You are very much mistaken.  You may want to read up on ECC to relieve this
strange worldview.  Or, alternatively, just believe that it works.

For example, audio CDs and CDROMs contain error correcting codes.
These codes are roughly the same as the ones in archiving software.  They
can detect and recover from any kind of error, as long as there aren't too
many of them.  (As in: a mildly scratched CDROM can be read without errors,
a badly scratched CDROM will generate read errors, and on rare occasions,
give garbled data.)

) Sending files larger than 2 GByte over a 10 MBit network I have already ) seen many times damaged files on the receiving side without any notice ) about problems during transmission - theoretically as good as not ) possible, but seen by me multiple times (I was not able to reproduce the ) behaviour, since if I 'succeeded' in getting file damaged over the ) network again, the wrong data were not in the same area of it as ) before). With newer network (100 MBit and 1000 MBit) versions the ) problem with transmitting large files seem to go away, but maybe to ) provoke errors it need only larger files (e.g. 200 GByte)?
)
) Anyone here who has seen files transmitted over the network which got ) damaged due to the way network transmits them and not due to detected ) problems during transmission or failures on storage media?

Not for some time.  Ten years ago, there were a lot more problems.

Error detection works by adding some kind of hash.  If the has has,
for example, 32 bits, then there is a one-in-2^32 chance for each packet
that contains more than one error to go undetected.


SaSW, Willem 

You are very much mistaken.  You may want to read up on ECC to relieve this
strange worldview.  Or, alternatively, just believe that it works.

I would be glad to replace believing with knowledge, so I have put here an example of my reasoning behind 'this strange worldview':

#01: Let's archive a 100 bit long string along with 10 bits of error correction data. This makes a 110 bits large archive file i.e. 10% additional data for error correction.
#02: Let's assume, that only maximum of 5 bits in the 100 bit long string can get damaged (and none in the correction data area). This makes maximum of 5% of damaged data.
#03: With the 10% additional error correction data available, the error correction algorithm should be always able to recover the original data where maximal 5% of it is damaged , so:
#04a1: If one bit is damaged there are 100 possible differently damaged strings,
#04a2: If two bits are damaged there are 100*99/1*2 = 4950 possible differently damaged strings.
#04a3: If three bits are damaged, there are 100*99*98/1*2*3 = 161700
#04a4: If four bits are damaged, there are 100*99*98*97/1*2*3*4 =3921225
#04a5: If five bits are damaged, therea are 100*99*98*97*96/1*2*3*4*5 = 75287520 possible differently damaged strings.
#04b: It makes a total of 79375495 possible differently damaged strings
#04c: The error correction data bits must be able to distinguish all the differently damaged strings to be able to choose the right one as the corrected version.
#4d: 79375495 is as binary number 100101110110010110010000111 27 bits long, but there are only 10 bits error correction data available, so not all the differently damaged strings can be distinguished to choose the right original one among all the possible versions.

Conclusion: it is not possible to recover from all errors with up to 5 bit erronous bits in a 100 bit long bitstring having only 10 bits (10%) of additional data for correction purposes.

Is this wrong or right and if wrong which statement is wrong in the reasoning chain above?

For example, audio CDs and CDROMs contain error correcting codes.
As I have read somewhere it can't be neglected, that CDROMs provide silently wrong data, so the chance to get an error here is not so tiny, that it can be skipped from practical consideration.

Right or wrong?

Claudio
.

Relevant Pages

  • Re: I know this wont work, I just want to know -why-.
    ... I am wondering about this error correction thingy all the time. ... Sending files larger than 2 GByte over a 10 MBit network I have already ... about problems during transmission - theoretically as good as not ...
    (comp.compression)
  • Re: BBC considers reducing DTT picture quality
    ... carry out consumer research into picture quality on the DTT system ... is horribly low bitrate and ancient codec), the error correction is ... DVB-S (the standard used for satellite transmission) and DVB-T (the standard ... transmission where the signal strength varies from location to location - ...
    (uk.media.tv.misc)
  • Re: Finding useful functions- part 1
    ... >>Unsupervised learning means that the network does not receive ... which is employed as an error correction ... the supervised ans unsupervised NNs must have ...
    (sci.cognitive)
  • Re: So tired of waiting for a decent signal...
    ... transmission just for error correction ?? ... how does it compare with say GSM? ...
    (alt.radio.digital)