Re: Olive and Toole
The full .pdf files are available either from Harman's website,
or from Sean Olive
Or I can e-mail the full files to you.
2.7 Blind versus Sighted Listening Tests From a Harman white paper
It is generally accepted among scientists that psychometric experiments
must be performed double blind. For audio tests, this means the identities
of the components under test cannot be made known to the listener, and
the experimenter cannot not directly control or administer the actual test.
In 1996 Toole and Olive in  conducted some blind versus sighted
loudspeaker tests that showed both experienced and inexperienced
listeners' judgments were significantly influenced by factors such as
price, brand name, size and cosmetics. In fact, the effect of these
biases in the sighted tests were larger than any other significant factors
found in the blind tests, including loudspeaker, position and program
interactions. These experiments clearly show that an accurate and unbiased
measurement of sound quality requires that the tests be done blind.
To remove these biases from listening tests in the MLL an acoustically
transparent curtain that is visually opaque is placed between the products
and the listeners so that they do not know the identities of the products
under test. All other associated equipment in the signal path is also
out-of-sight and locked in an equipment rack, since the performance and
paranoia of some listeners can be affected by simply having knowledge that a
certain brand of interconnect or CD player is in the signal path.
The front screen consists of a black open knit polyester knit cloth chosen
for its acoustic transparency and used as grille clothe in many of our
loudspeakers. The material is attached to a large automated curtain roller
so it can be easily lifted down and up with an infrared remote control.
Weights are attached to a seam in the bottom so the cloth retains
its tautness when in use. Retractable curtains made of the same material
surround the listeners to hide the identities of loudspeakers located at the
sides and rear of the listening room. Figures 4 and 5 show the front, side
and rear curtains fully retracted when not in use, and Figure 8 shows the
curtains in place during an actual listening test.
BLIND vs. SIGHTED TESTS - SEEING IS BELIEVING From a paper by Floyd Toole
Knowledge of the products that are being evaluated is generally understood
to be a powerful source of psychological bias. In scientific tests of many
kinds, and even in wine tasting, considerable effort is expended to ensure
the anonymity of the devices or substances being subjectively evaluated. In
audio, though, things are more relaxed, and otherwise serious people persist
in the belief that they can ignore such factors as price, size,
brand, etc. In some of the "great debate" issues, like amplifiers, wires,
and the like, there are assertions that disguising the product identity
prevents listeners from hearing differences that are in the range of
extremely small to inaudible. That debate shows no signs of slowing down. In
the category of loudspeakers and rooms, however, there is no doubt that
differences exist and are clearly audible. To satisfy ourselves that the
The results are very clear, and strongly supportive of the scientific view.
Figure 4 shows that, in subjective ratings of four loudspeakers, the
differences in ratings caused by knowledge of the products is as large or
larger than those attributable to the differences in sound alone. The two
left-hand striped bars are scores for loudspeakers that were large,
expensive and impressive looking, the third bar is the score for a well
designed, small, inexpensive, plastic three-piece system. The right-hand bar
represents a moderately expensive product from a competitor that
had been highly rated by respected reviewers.
When listeners entered the room for the sighted tests, their positive verbal
reactions to the big beautiful speakers and the jeers for the tiny sub/sat
system foreshadowed dramatic ratings shifts - in opposite
directions. The handsome competitor's system got a higher rating; so much
for employee loyalty.
Other variables were also tested, and the results indicated that, in the
sighted tests, listeners substantially ignored large differences in sound
quality attributable to position in the listening room and to program
material. In other words, knowledge of the product identity was at least as
important a factor in the tests as the principal acoustical factors.
Incidentally, many of these listeners were very experienced and, some of
them thought, able to ignore the visually-stimulated biases .
At this point, it is correct to say that, with adequate experimental
controls, we are no longer conducting "listening tests", we are performing
SUBJECTIVISM vs. OBJECTIVISM - IN CONCLUSION
In this lengthy summary we have covered a lot of topics. Much of it was
matter-of-factly technical, driven by data and the need to measure, and
much of it was subjective, driven by the desire to understand what
we can hear. All of it was oriented towards creating loudspeakers that sound
The literature of audio continues to be sprinkled with letters and articles
debating the merits of science in audio. The subjectivist stance is that
"to hear is to believe", and that is all that matters. Some of the arguments
conjure images of white-coated engineers with putty in their ears, designing
audio equipment, and not caring how it sounds, only how it measures.
I have never met such a person in my 30 years in audio science and
The simple fact is that, without science, there would be no audio as we know
it. Without extensive and meticulous subjective evaluation, there would be no
audio science as we know it. Without audio science, audio engineering
reverts to trial and error. So, where does this leave us? Clearly, to be
successful in this business, one must be actively involved with both of the
objective and subjective sides.
A faith in the scientific method is not a blind faith. It is a faith built
on a growing trust that measurements can guide us to produce better
sounding products at every price level, for every application. The proof, as
always, is in the listening, and one MUST listen.
The Harman International loudspeaker companies, JBL, Infinity, and Revel
have invested heavily in measurement facilities that allow them to take the
fullest advantage of existing audio science. They have invested in talented
engineers who understand and respect the scientific method, good sound
and great music.
They have invested in elaborate listening rooms where they can enjoy and
criticize the fruits of their labors. There are people on staff with many
years of experience in successfully probing the frontiers of knowledge in
product design and audio science, and they are equipped to continue those
investigations, to push those frontiers.
The arrival of multichannel audio for films required some adjustments in the
performance objectives of speakers, certainly at the high end. Multichannel
music is another, as yet ill-defined, challenge. More speakers in rooms,
means less consumer tolerance for large boxes. Merging loudspeakers
with rooms is not easy, and it is the one remaining large challenge for our
industry. We are working on all of these fronts. Stay tuned.
- Olive and Toole
... Blind versus Sighted Listening Tests From a Harman white paper ... For audio tests, this means the identities of the ... that showed both experienced and inexperienced listeners' judgments were ... transparency and used as grille clothe in many of our loudspeakers. ...
- Re: SACD High quality audio
... >ABX didn't force the invention of the listening room, ... or even the audio signal relay box. ... >years before the first ABX test was even conceived of. ... >the operation of the relay box truely blind. ...
- Re: what to do with DSD ?
... THOSE SPECS HAVE NEVER BEEN CORRELATED WITH LISTENING ... the sound quality of audio chains tend to ... is mostly a blind evaluation. ... unknowns" and the "unknown knowns"). ...
- Re: Maybe Feddie had it right........
... his boss what Arny was like. ... Look at most audio domains. ... Seems like analog took that path for a bit as many high end ... Because of that, more and more of my listening is at my desk, where I ...
- Re: How safe operating an amp with no fuse?
... Care wrote: ... and also claims related to audio. ... If you don't like my standard of proof, I have no problem with that. ... proof is good old fashioned LISTENING, as God intended it (or as audio ...