Re: QI and MQ Coder: First real-life experiences

On 2006-02-04, davvid_a_scott@xxxxxxxxx <davvid_a_scott@xxxxxxxxx> wrote:
Jasen Betts wrote:
On 2006-02-03, David A. Scott <daVvid_a_scott@xxxxxxxxx> wrote:
"nightlight" <nightlight@xxxxxxxxxxxxxx> wrote in
news:1138898612.601694.229860@xxxxxxxxxxxxxxxxxxxxxxxxxxxx:


all I would have liked to was being able to feel it's
"always better than arithmetic" compression.

For that, you don't even need the source or executable test. Just a
bit of math to work through [T3], especially p. 8, for QI and Volf's
or Tjalken's thesis for in depth AC formulation, plus Ryabko for
finite precision AC redundancy.

You know maybe if it wasn't so easy to write an arithmetic compressor
that actaully beats what nightlight claimed was the most fair test.
One might think there is some gems in his long posts. But from the fact
that it fails to do as claimed its no wonder nightligt is trying to
steer you away from scource code or executables where one can run ones
own tests on files and see the facts themselves. Nightlight does not
really want you to think for yourself. He is more than willing to
do that for you.

So the "always better than arithmetic" compression is like
a lot of his logic it's just plain wrong. He has quoted so many texts
over and over written before his QI I am surprised he hasn't quote
the Holy Books to support his postions I am waiting to see which
Holy Book he first uses to support his claims anyone have a guess?


Nightlights coder claims to be able to compress any 998358 of these
trinary symbols into 197795 bytes (it claims 1582359.99 bits)

That's about as close to ideal as anyone's likely to get.

has anyone confirmed that it's capable of this?

Jasen my newsreader died so going through google

How can anyone confirm anything nightlight says. The
code can't work with files and its hard to get a straight
anwser from him with out his quoting hundreds of lines
of text that usually has nothing to do with the question
asked.

I did a test for the 1,000,000 symbol case and based
on what little I could squeeze out of him his coder fails
to compress as well. The problem is the side infromation
he needs to use. He will wave his hands and who knows
what he will claim as the final anwser.

But I like the number you choose since
998358 * log3 = 1582359.99229497204300379190460185...

yeah, I did up a spread*** and searched for a big number that came to just
under a whole number of bytes, and confirmed that his buffer-based test code
also claimed a result that was clearly within that nuumber of bits.

I have not tested my code it really was a fast mod to arb255
however I suspect it would compress some strings to
197795 and some to 197796 bytes. Since this is close
to straddling the byte boundary. If I was writting to a bit
string instead of bytes. I think I could get it to code to
1582360 bits. But please check the string of 5 case.

Notice 5 * log3 = 7.92481250360578090726869471973908
I would like to see since you seem to be using his test code.
what does he spit out for this case. The reason I ask is
this 3 + 9 + 27 + 81 + 243 = 363 this is sum of the number
of possible strings of 3 symbols types up to a length of 5.

No, that's the number of strings of 5 symbols or fewer ( well, almost - you
missed the case of a zero-lenghth string), his code seems to be built
for the case where the length of the original is known beforehand rather
than being encoded in the data.

There are circumstances where this is the case, one that he mentioned was the
hash-bitmaps used to transmit queries in p2p networks, another could be video
compresion.

Notice the 363 is bigger thatn 256 the number of values for a
sinlge byte. It would be sweet if his code put out the number
7.92 when a reality check shows it has to exceed 8 bits.

No. the range 0..242 is sufficient to encode the content of strings of 5
symbols if length is not also being encoded.

Bye.
Jasen
.