Re: XDepth Raw: a jpeg-compatible Raw compressed format for digital cameras

Trellis Management wrote:
On 30 Set, 11:58, Thomas Richter <t...@xxxxxxxxxxxxxxxxx> wrote:
Trellis Management schrieb:
Uncompressed HDR images are generally 96bpp images. Especially if
computer-generated (CG, VFX etc.) you tend to have most of those bits
actually used.
Do you have sources to back this up? For example, OpenEXR uses
half-floats, 16bits per channel and pixel. Computer-rendered images are,
yet again, a different domain - those you can compress better by other
means than photographic images. Thus, I would be curious about your
sources (not that I doubt them, but it would be good to know *where*
such images are actually required).

I think the best way to find out is downloading .hdr light probes from
here:
http://www.debevec.org/Research/HDR/
and count bits on Matlab or similar software. AFAIK only Matlab 2008
supports HDR image loading.

Well, yes, but that's not quite what I'm asking for. I mean, do you have evidence that those bits carry *useful* information? And if so, *useful* for whom?

Although the .hdr Radiance format is a quasi-lossless format. It uses
a mantissa-exponent representation, so you might lose the very fine
precision for the purpose of counting effective bits used. Best of all
would be rendering out a frame from a software that supports HDR and
save in float Tiff or similar.
Most 3D software render out at 128bit/pixel (RGB) precision.

No doubt about it, but not quite the point I want to make.

http://turtle.as.arizona.edu/im/mdr16/
Thanks for the pointer - I browsed through it, and need to look into
this more carefully, but as I understand it, they take "16 bit
processing" as "pixels are encoded in 16 bits each", which is of course
for a today's CPU the most convenient representation. Not as such that
you need the full 16bpp range. As stated, I don't consider > 16bpp
unuseful, I just consider it unuseful for HDR photography. All you get
is noise in the lower bitplanes.

From a "digital photography" standpoint what you say about noise can
be true. As CCD's get better and better and will capture more data in
the upper range you will start to fall short even using 16bit/
component. As of now, in our experience, Reflex cameras (~$800) have a
white point of up to ~12000 in the range 0 - 65535. Top cameras like
the Hasselblad (~$25.000 - http://www.hasselblad.com ) are most
probably using the whole range already. Professional Camcorders like
"Red" (~$20.000 - http://www.red.com ) record a Raw image sequence and
they're also likely saturating the range at 16bit already.
My feeling is that the full 16bit range will soon arrive to a consumer
level, while at that point pro-level will get floating point.

Hmmm. Maybe, I'm not clear about it. My view is that with the current MPixel race, sensors are getting smaller, but not really better (at least in the consumer market). Maybe the "more precision" becomes an argument as soon as manufacturers change this strategy, I'm not sure. 16bpp seems to be overshooting for me, but anyhow, it can be covered with JPEG2000 and JPEG-XR (while I think it would be trivial to extend JPEG-1 for that, I don't see this to happen).


You mean they fixed this ?:
http://www.xdepth.com/images/XDepth_comparison.jpg
or this, from an indipendent beta tester:
http://www.hdrlabs.com/news/files/xdepth_compare3.png
No, and if you want a critical voice on JPEG-XR, you're talking to the
right guy anyhow, but I mean artifacts created on tone-mapping - that
has been fixed. The blurring & blocking you see in the above images can
be addressed, though, and is due to a rather naive implementation in the
DPK-Code provided by Microsoft - it can be avoided and image quality can
be considerably improved. Details will be published in the upcoming ICIP
conference this year, including some subjective testing results.

I'm sure they're "re-touching" it, but I also imagine they (MS) want
to hit the market ASAP...that is why I think it's going to be very
hard to remove that sorts of artifacts in the short term.

Ehem, sorry for not making myself clear, you can do *a lot* even today, meaning, there is software in existence that compresses an image to a valid JPEG-XR stream such that any conforming implementation will construct this stream to an image looking better than the same image compressed with the DPK-code. That is, a) only the compressor is altered, and b) all this happens within the limits of the standard, so nobody is stopping anyone from *not* using such code. Well, that's of course again the Pegasus code...

Also, that sorts of blockyness is a tricky one. It shows in HDR if you
tone map dark areas or if you rise the gamma and exposure of the
picture, but it actually reveals the overall precision of the encoding
- i.e. how much data is actually preserved across all the stops in
HDR, in dark and bright regions...which is relevant in this case,
rather than in the case of common 24bpp images.

Not so tricky as you might think - the problem is that the DPK compressor wastes a lot of rate into image data you cannot detect in first place, so the blockyness goes away as soon as you can invest that
rate into more useful data.

That is also why the quality metrics needs to address all the stops
and fundamental variations of the HDR data (gamma, exposure, range
compression mapping etc.)

That's of course true, and such things are addressed by "application metrics" (or how I call them). Yes, we're working on this kind of stuff.

Clearly, if MS' algorithm is tested using PSNR - as they intially
showed...they will always get good results, no question about it.

Right. Well, "good" is relative. Worse than JPEG2000 for example. The graph you find on Bill's HDPhoto blog is just the result of an apples to oranges comparison: JPEG2000 visually optimized against PSNR-optimized HDPhoto, plotting PSNR. What do you think will happen here? (-:

Furthermore, if you test images, you should also provide them "on
scale". The above images are magnified, and hence not observed under the
suggested viewing conditions - note that JPEG-1 (and good
implementations of JPEG2000 as well) will take care about the contrast
sensitivity of the human eye - that implies, however, that you know
something about the observer and the observation distance. A magnified
image is hence no longer optimal. Also note that the DPK code you seem
to have used for evaluation is not at all optimized for the CSF, this is
one of the easier things one can apply to improve image quality
considerable. Speaking of NDAs, I'm sure Pegasus Imaging would provide
such an implementation for you to come to a fair comparison - the same
goes for JPEG2000. (-:

Images are magnified *after* compression, in order to look better at
the details (in our case, of course :)). The original image is pretty
big anyway.

Yes, but that's an error in your methodology of testing. You know, there are standards for *how* to run subjective tests, e.g. ITU-BT.500 (IIRC), and for them, of course you need to judge the images displayed in native scale. Testing under magnification will make (due to the CSF) artifacts at higher frequency more visible than to the naked eye, and will make artifacts at lower scales less visible, so you're failing the goal of a subjective evaluation.

Also as you surely know, filtering won't do much for that kind of data
loss. Anyway, I will check improvements on Jpeg-XR as they will be
revealed.

I'm not talking about filtering or post-processing - we don't allow that in our testing (the same can be done for J2K,JPEG-1 and so on), but about encoder optimizations, all *within* the specs.

So, all that aside, what I'm saying is that you should be very careful
of *what* you are actually testing. You seem to have made an apples to
oranges comparison, namely comparing a non-optimal implementation with
an optimized one, and come to conclusions I can only agree partially:
While I'm personally not very happy about the JPEG-XR quality, one must
admit that one can improve on it, simply because the currently available
implementations are naive.

Our own implementation isn't optimal as well. Consider our software is
using LibJpeg and FreeImage with no modifications, and not much else -
compared to all the custom code MS's engineers wrote for 3 long years.
(we should probably buy the book "How to waste resources" from them -
joking :))

Ehem, certainly.

Our own implementation of our algorithms is in a pretty
coarse state...and even with all that, we're able to show much more
details and less artifacts with extreme ranges. So, I'd say that MS's
code is much more "optimized" and "looked after" than our own...

Certainly not. Look, there are many options in the specs the DPK code does not take advantage of. The specs are, in my impression, designed "the wrong way 'round", making it unnecessary hard to optimize its visual quality, but nevertheless, there *are* options.

that's
for sure...this means there's so much room for improvement on our own
system that you couldn't imagine. :) Not mentioning it can also employ
Jpeg2000 (not tested yet as a solution though)

Ditto for JPEG2000, but *within* the limits of the specs. Yet again, Pegasus offers such a "visually improved" codec.

If it that's really fixed as you claim then I must think the file size
has grown considerably than before the fix was in place.
Sorry, I don't quite understand. If I measure image quality, I always do
that against compression ratio (or rather, bits per pixel), so of course
we keep the filesize fixed to compare one codec to another.

I meant, if the current quantizer stuffs more data in order to get
finer details rather than the coarse blocks you have now... the size
on file will reflect this as well. So you will need to lower quality
to keep file size on par again, I imagine. That's to be seen when that
aspect will get better as you say.

Two comments: Yes, of course the changes we made in our in-house compressor result in having *smaller* files at *lower* quality for the same quantizer setting, when compared to the DPK code.
Second comment: Of course, you need to compare the rate-distortion curves of the two codecs to check whether that's an improvement, i.e. whether making the quantizer bucket size smaller in our code such that the two file sizes are identical yield to a better looking image. Please feel ensured that it does. So, no, it's "not to be seen", but rather "it has been seen". Literally, by human observers, not only by metrics. By that I mean, "yes, colleagues of mine did run subjective tests following ITU guidelines".

Of course, and I agree, one has to be very careful when making such statements, and I can only say that I tried my very best to ensure that we *do* have meaningful results. I'm not in marketing, you know. (-;

If MS is going to get its Jpeg-XR into most digital cameras we will
find a niche where quality is actually appreciated, and we'll be there
when most camera users will complain about MS' solution or will keep
shooting Jpeg because Jpeg-XR will only open on Vista photo viewer or
whatever it's called.
I doubt that JPEG-XR will be vista-only. Hey, I'm using that on Linux,
and the source is available, and will be available, so I doubt that very
much. I cannot speak for the committee, but it is the expressed will of
MS and of the JPEG-XR reference software developers (not me, not MS) to
have the code published and open, and I'm with that decision.

No question about openness of the format. I was doubting every single
software developer will hurry to implement it.

Well, I simply don't know. My suspicion is that you will get it, like it or not, in the same sense that you get windows, like it or not. While I regard image quality as a high goal, in real life other factors are more important, and one of them is that a strong company is behind the specs.

About the ISO, you know better than me that the most commonly used
implementation of Jpeg-1 - even today - is LibJpeg written by the IJG
- not affiliated with the ISO. Infact LibJpeg lacks lossless and
arithmetic coding support while 12bit is supported at compile-time,
and that's why these aren't common implementations nowadays.
That's certainly the case, but my point remains valid: If I have such an
image, let it 8bpp or 12bpp, I can certainly get the standard, make an
implementation and arrive at a reconstructed image. I cannot do that for
a proprietary solution. I *might* have some work to do, but I will
succeed at the end. The same cannot be said with your format.

That is certainly true. Although I was speaking from a user's point of
view. Jpeg is a "mass" usage format, nothing changes that...every
single user must get its data back, whether an absolute beginner or a
software engineer.
Would anyone dare to change that introducing a file format for mass
usage that doesn't share a single line of code with *any* of the many
formats around ?

Likely not, of course. And nobody will change JPEG anymore either.

Our format being "closed" today doesn't mean it will stay closed
forever. A free SDK along with a few executables and format
documentation will suffice the purpose of openness. Not that
complicated. Just a matter of choices, timing and goals.

Well, probably - likely even. It's an important aspect, at least. What I want to say is that - even this sounds rather stupid - "an ISO stamp helps for certain markets". Otherwise, I think MS wouldn't go that hard road with JPEG-XR.


Having Jpeg compatilbility, our 8bit fallback is simply Jpeg, or, in
the future, an enhanced version with in-loop deblocking. When we
started we faced the fact that Jpeg is, overall, still one of the best
image compression technology around - additionally to its universal
adoption. Apparently MS, instead, felt it was time to change
everyone's life and image format. :)

Well, I'm personally not against such a revolution, but for that it must be worth it. Sometimes it helps to throw old technology overboard, but only if something really convincing replaces it.

Citing Greg Ward, one of the HDR fathers, speaking about HDR but I
think that can be a valid statement in the most general MS's case:

'Backwards-compatible HDR formats are a win-win for manufacturers and
consumers alike. Easing the transition from 24-bit RGB to full dynamic-
range and gamut capture, editing, and viewing has the potential to
greatly accelerate market penetration. Early adopters would gain
immediate access to an HDR world, while others would not be
inconvenienced, and could even benefit from improved tone-mapping in
their legacy content'

from: http://www.anyhere.com/gward/papers/sid2007.pdf

Well, sure...

The file format we have adopted is called JFIF (see REFERENCES).
*This format
has been agreed to by a number of major commercial JPEG vendors, and
it has
become the de facto standard*. JFIF is a minimal or "low end"
representation."
Yes, but JFIF is *open*. While not being a standard, I find documents
about how it works. Do I find documents about your format, can I
re-implement it? Likely not. (-:

You will... that's my point. There's no reason why we should keep it
"closed" forever. Does this make sense ?

Much more than it did initially - thanks! (-;

We are aware of the 12bit capability of Jpeg-1 but it's not a de-facto
standard sub-set. Moreover that precision is still in the range 0 -
1.0 and does not address HDR "per-se" (same goes for a 16bit Jpeg2000,
for instance). See the argument on HDR precision above. Hence the need
for a custom solution. EXR is another good example as to why custom
algorithms are needed when dealing with HDR image compression.
I don't buy that, sorry. IMHO, 12bpp JPEG-1/JPEG2000 or JPEG-XR solve
this problem sufficiently. Do you have any data why it is *not* sufficient?
I know about OpenEXR of course, as one floating point format for images,
and we have been in discussion extending JPEG2000 for float in part-10,
but the problem is not so much solving it technically (we do have a
solution), but rather to identify interesting applications - and even
more - to define suitable metrics for those applications.

HDR *is* floating point. When cameras will capture >140dB of dynamic
range in a single shot you will have a very broad application for HDR.
If you (meaning the ISO, MS whoever) don't plan ahead such
enhancements, I guess, the format will be useless in a few years.

Hmm. First, I'm unclear whether >140dB dynamic range will be more than just a marketing gag, in the same sense that 12mMPixel on a consumer market camera are only a marketing gag. Second, 140dB is approximately 24bpp (per channel), and that's covered by JPEG-XR and JPEG2000, so there are formats to jump in.

Would the ISO and/or MS come out with yet another format to address
the issue later on ? That'd be smart. :)

No, we have those. (-;

This whole Jpeg-XR move has
probably been done too quickly and with little aftermath, IMHO, due to
obvious "private" interests, and this will probably "harm" users if
they succeed in getting it widely adopted.

Here a couple of my personal views on this: I agree that JPEG-XR is moving too quickly, and mostly due to private interest. I can hardly catch up with my testing, though I cannot speak for other people. I can only repeat that we try to get the best we ever can out of it, within the limitations set. JPEG-XR is, strangely enough, too early and too late at the same time. Too early, because we need more time to be more careful, and too late because "raw" photography is already in wide use, and I don't see professionals accepting *any* lossy compression in first place. MS seems to want to jump on this train, but it's gone already.


About the 12bit argument, download some light probes from the site
above (Debevec's) and measure the peak floating point value against 0
and get contrast ratio. See if it, theoretically, fits in 12bit/
component RGB without severe banding artifacts.
For instance, the HDR Radiance format (from Ward) is pretty smart with
its mantissa/exponent quasi-lossless representation at 32bit/pixel
RGBE (E stands for Exponent - 24bit mantissa (8*RGB) + 8bit exponent
per-pixel). Too bad you cannot really lossy-compress the exponent or
mantissa further since you introduce exponentially more artifacts than
on common images. (that's been done in the past by California
scientists if I remember correctly...results are problematic and need
extra data and processing to fix the most severe artifacts. Not
practical)

There are better ways to do lossy float compression, indeed, and we do have ideas for that - just no useful testing methodology. Measuring PSNR is definitely not.

Also considering the industry is *almost* there with consumer HDR:
http://www.fujifilm.com/photokina2008/pdf/release/super_ccd_exr_e.pdf
...what should we expect in, say, 2 years time ?... enough said.

Thanks for discussing.

Definitely. The best I had in this group for months.

So long,
Thomas
.