Re: ISO 200000 ?

In article <43D2025F.4090203@xxxxxxxxx>, "Roger N. Clark (change username to rnclark)" <username@xxxxxxxxx> writes 

Hmm.  I just got back from JPL...so much to read.

Hmm, small world indeed - I just had two guys out there at the back end of last week as well. I no longer hold them in as much awe as I once did, having had to send a couple of engineers out last year to teach them that their claim that it was impossible to measure the MTF of a CMOS chip beyond its Nyquist frequency. Hard to believe, but their guys assured us that what most people testing cameras in this group do on regular basis was impossible to achieve.

Statements here begin to clear the fog of noise.  Kennedy has alluded
to less than 1 photon per pixel per frame and implied that 1 photon
was useful.  What he failed to clarify is that in none of the references,
do people actually claim a detection with one photon.

For good reason - I never claimed detection of an object with a single photon either! 

The Poisson noise limit is NOT "Roger's limit."  It is fundamental
math and physics.

The derivation of the Poisson noise level is fundamental maths and physics. The threshold of the detectable noise is Roger's limit. That is neither maths nor physics.

 Kennedy is tricking you with giving part of the
answer.

That is pretty rich, Roger - the reason you don't understand the problem was because I am tricking you all! :-(

I am not tricking anyone and have made it perfectly clear throughout that the issue is that your detection threshold of 1 photon/pixel/frame is not always correct. In fact there is only one condition where it is correct, and that is where the detection is achieved in 1 pixel and 1 frame! In almost all cases the object being detected covers several, even many pixels in the frame whilst in some cases many frames can also be used. Both spatial and temporal integration lower the threshold of SNR per pixel per frame required for detection.

 If you really had 1 photon, and a perfect
system with no noise, you would have a 50% chance of correctly
saying when you detected that photon.

Yes, assuming you mean "no excess noise", only photon noise, not "no noise" at all.

 No scientific journal will accept
a paper where you say you detected something but it was 50-50 chance.
I would hope no police/military decision to shoot was decided
on 50-50 chance.  Oops-better not go there....

That is precisely the point - 1 photon/pixel/frame is not a 50:50 chance, not even when the sensor excess noise itself is the same level. Often if not always, excluding a few exceptions, the object being detected subtends many pixels and can be viewed over many frames. Both criteria stack the probabilities much better than 50:50.

Detecting a signal less than the noise is done all the time.  We are currently
doing just that on the Cassini mission orbiting Saturn: detection of the
faint E-ring takes hours of 1-second integrations with the VIMS (thermal
noise saturates the instrument with longer integrations).  In any one
frame, the signal is a small fraction of one A-to-D bit.   But the noise
is digitized so by averaging many frames, you can average the noise
and measure the signal.

You may well be stacking many frames to obtain images of the rings, but I hope that JPL are doing better than just integrating frames. You know the general shape of the rings, their spatial extent, and should therefore be able to use that to aid the detection process. I suspect that the need for integration is that they want to see the structure of the rings, not merely detect their position or presence.

It is fairly well known that the human eye/brain has a detection capability with closely matches the SNR of the image convolved with a matched filter of the object being observed - at least up to spatial extents of several milliradians. How this is achieved is less well understood and nobody is suggesting that the brain sets up spatial filters to extract the signal from the noise, only that the probability of detection closely follows the matched filter output. This is the core of most extended object detection models, including those of the US Army's Night Vision & Electronic Sensors Directorate (NVESD), since Jim Ratches used this approach in their first relatively accurate model, back in the early 70's.

If you know what you are looking for, it is trivial to design a detection process which uses the spatial extent and form of the object to achieve detection in a single frame even when the signal is <1photon/pixel. The real skill for ATD and ATR systems is to be able to consistently match or exceed the performance of the human observer when you don't know the precise geometry of what you are looking for. Not impossible though.

Years ago it was thought only long exposures could do the job.  This included
cooling CCDs and doing hour long exposures so the signal would be
larger than the read noise.  The old school of engineering developed the
intensified devices to deal with the problem, and pretty effectively,
but with lots of side effects.

Somewhat misleading. Intensified devices were not developed to deal with long exposures - these were and still are two completely different concepts. Intensified devices existed years before the 1971 invention of the CCD - and plenty were in use in Vietnam. Intensifiers were developed to amplify the available light, not to deal with the problem of long exposures.

 But a number of years ago (5-ish?)
a revolution was begun by amateur astronomers using less than perfect
sensors that they wanted to push beyond normal limits.  This included
using $20 web cams with video feeds digitized 5, 10 even 20 minutes
of video, that is then analyzed with software, individual pieces of an
images extracted and added to make planetary images that rivaled major
observatories with million dollar research grade adaptive optics
systems.  E.G. see (amateur astronomer images):

HIGH RESOLUTION CCD IMAGING: http://legault.club.fr/index.html
(go down the page and see "Saturn with a web cam" its stunning!)
(It is a stack of 1,100 images).

And this was being done while professionals were spending millions on
adaptive optics.  Not that adaptive optics don't work well, just that
here is another solution that costs $20 and some free software (astrostack).

So then the amateur astronomers started stacking deep sky images and found
they could beat down the read noise.  And guess what?   The cooled CCD imagers
have realized that they too can take shorter exposures and average the
read noise, and now for many, even with top of the line cooled
CCDs, multiple stacked shorter exposures (minutes instead of hours per frame
when going for the faintest stuff) is common.

Yes, but you are always chasing the read noise. The advantage of the E2V and TI sensors is that the read noise is already extremely low, so you aren't chasing so much. Consequently you get the same result in less time, or you can be more selective in the frames you integrate, throwing those which fail to meet defined quality thresholds away. At the end of the day, you do far better with a lower read noise. 

With lower read noise cameras, like the 20d and 20Da having about 3 times
less read noise, and working in darker skies, are getting much much fainter,
and thus working near and below the 1 photon/pixel/frame level.

No, they are working near and below 1 phot/pixel/final image - you have already stated that the noise is 3phot/pixel/frame.

Kennedy said the intensified devices are limited to about 1 megapixel.

They currently are, however I remind you of the issue which kicked this discussion off - a press release from a Korean company claiming to have developed a technology which would enable camera sensors with much higher ISO levels than is currently achieved. Since we are now apparently agreed that this claim is indeed feasible, having established that the E2V & TI already show the way, I would be very surprised if they chose to develop the technology for their stated application with a single megapixel size.

I think I would rather have stacked 5D or 1D Mark II images (12 to
16 megapixels).  With read noise of 3 and 16 megapixels, one photon
per pixel, one could average 16 pixels and get 16 photons with 4x
lower averaged read noise (so less than 1 electron), thus have a 1
megapixel image with a signal to noise >16.

If that is what you think then you should be able to demonstrate a positive result to the challenge I gave earlier. With an exposure of 1/50th sec or similar and any post processing you like, show an image of similar noise characteristics to any of the E2V images in 10^-4 lux (overcast starlight) illumination. Try it - it doesn't work, and the reason is pretty obvious. It isn't just a matter of noise, it is limited ISO - precisely what the Koreans are claiming.

So, in summary, 1 photon per pixel per frame IS useful, if you
sum many pixels to get many photons.  Whether you do that with
pixel summing and one frame or multiple frames is irrelevant, but
you need multiple photons, and in all cases, your maximum signal-to-noise
ratio is the square root of the number of pixels you counted.

Yes, that is Poisson statistics, but it isn't the limit that you stated in previous posts.
--
Kennedy
Yes, Socrates himself is particularly missed;
A lovely little thinker, but a bugger when he's pissed.
Python Philosophers (replace 'nospam' with 'kennedym' when replying)
.

Relevant Pages

  • Re: ISO 200000 ?
    ... Indeed, however, certain applications require these photons to be counted up over many frames and to do so whilst achieving genuine photon counting performance requires a readout noise of much less than one photon per pixel per frame - otherwise you are simply chasing the RO noise, to say nothing of the 1/f noise becoming significant due to the long exposures necessary to capture sufficient photons to have a photon noise which exceeds the RO noise by a reasonably useful margin. ...
    (rec.photo.digital)
  • Re: ISO 200000 ?
    ... noise and collect multiple frames to average the noise. ... but the final image isn't a single frame. ... total photons for detection ... What is the noise in a 10 photon signal in a single pixel in a single frame. ...
    (rec.photo.digital)
  • Re: Einsteins energy formula E=mc^2 is WRONG!
    ... can dance on the head of a pin -- neither one is science. ... the known 'laws' of physics to prevent a photon have a frequency of 1Hz, ... and detection of a very low frequency photon, below any proposed limit, ... determination by changing frames to a frame in which the light has a much ...
    (sci.physics.relativity)
  • Re: Atomic Rockets and stealth
    ... in order to get a signal about 5 times the fluctuations in the noise. ... telluride infrared detector with a detection band from 0.8 micron to ... radiating temperature - going to a 1 MW source, for example, gives a ... The zodiacal IRB matches 270 K black body spectrum, ...
    (rec.arts.sf.science)
  • Re: depth charge avoidance strategy
    ... With Sonar, no. ... up the noise of your propellers; ... To avoid detection the general rule ... You wanted to get him to drop his charges. ...
    (sci.military.naval)