AB
TESTING, A MISAPPLICATION OF VISUAL CRITERIA IN AUDIO
©2006 Mark B. Anstendig
(AB testing
is a form of testing designed to compare different qualities of sound. In audio,
it is used to compare and evaluate the differences in sound between components.
The most prevalent form is to switch back and forth between components while a
single sound source, usually a recording of music, is playing. Similar methods
are used in aural research.)
For
decades, controversy has raged in the audio world over the validity of AB
testing. While the controversy primarily concerns the AB comparison of audio
components, AB testing is also used extensively in scientific research into
human hearing and in the evaluation of recorded sound quality. In fact,
scientific investigation of human hearing in more than a rudimentary fashion and
the investigation of complex sounds, containing both timbral distinction and
details of nuance as the sounds flow in time, first became possible with the
advent of recorded sound. Before sound recordings, it was impossible to repeat
any such sounds exactly the same way, especially humanly produced sounds such as
the expressive nuances of music.
The senses
differ markedly in their characteristics. In order to investigate any of the
senses, it is necessary to differentiate the characteristics of each of the
senses, recognize how each works, and apply only procedures pertinent to the
sense being investigated. Above all, while testing one of the senses, it is
important to not misapply procedures that only apply to another of the senses.
Unfortunately, most methods of testing hearing do just that by trying to
duplicate the visual procedure of direct comparison. Direct visual comparison
has been accepted for centuries as scientifically accurate. But direct
comparison is possible only with sight and impossible with all the other senses.
That fact is probably the most pertinent scientifically established fact about
all sensory perception.
It
has long been known that the only capacity of any of the five senses that meets
scientific standards for accuracy and dependability belongs to sight. That
capacity, known in the optical world as direct comparison, is the visual
comparison of objects lying directly next to each other (not an inch or a
centimeter apart, but absolutely next to each other. In color testing, the one
color is laid directly on top of the other color). This direct comparison of
immediately adjacent visual images reaches its highest level of precision in the
comparison of shades of color and gray tone (scientifically accurate color
charts, determined by direct visual comparison, have existed for
centuries).
That direct
visual comparison is the only scientifically accurate capacity of any of the
senses was well known in the first half of our century, when it was acknowledged
that there was a need to devise a focal-point-exact method of focusing optical
lenses in cameras. (The original method of focusing in cameras, the ground
glass, is highly inaccurate, mainly because it does not utilize the highly
accurate sensory capacity of direct comparison.) But, in the second half of the
century, mention of direct comparison has been pointedly avoided in the
optical-photographic fields, because the only device that succeeded in utilizing
direct comparison and, thereby, achieving absolutely exact (focal-point-exact)
focusing, the Messraster, was not owned by the leaders of the industry. From
1939, when that patent was first introduced, until the inventor's death, the
large firms that controlled the German optical industry fought to keep that
patent off the market. All of the manual focusing devices that have been
available to the public fail to achieve focusing accuracy because they do not
utilize the only accurate capability of sight, direct
comparison.
In
the experience of the author, who was born in 1936 and reached maturity in the
second half of this century, the ascendancy of direct visual comparison over all
other sensory comparison has been mentioned only once. That was in 1960, in
West
Berlin, by Joseph
Dahl, the inventor of the Messraster focusing device. Mr. Dahl, from whom the
author bought a number of Messrasters in the early 1960's, took great pains over
quite a number of months, to explain and demonstrate the problems of sensory
comparison to the author in order to show why the Messraster, by using direct
comparison, is the only device in photography that can focus
accurately.
Only the
visual comparison of unmoving objects directly next to each other, with no space
between them, can claim scientific accuracy. All other forms of comparison using
any of the senses, including other forms of visual comparison, do not begin to
meet scientific standards of accuracy and dependability. A clear understanding
of why that is so, as well as an understanding of why direct comparison is
impossible with the other senses, is essential to valid comparison-testing
in audio and, in fact, all comparison of sensory
impressions.
Why is
direct visual comparison the only accurate form of comparison in all five
senses? Because it is the only form of sensory comparison that places no demands
upon our memory. All other forms of comparison, including visual comparison of
objects not directly adjacent to each other, depend on our memory for sensory
impressions. And our immediate memory for sensory impressions is notoriously
undependable. Place minutely varying shades of color next to each other and we
have absolutely no difficulty telling which ones are lighter, shinier,
warmer-toned, cooler-toned, etc. But showing them to us one after the other
introduces a profound degree of uncertainty and doubt and we will often guess
wrong as to their differences. (Mr. Dahl demonstrated this by showing me two
pieces of paper, one after the other, and then asking me which was the lighter
in tone. I remembered wrong.) That uncertainty can only be definitively resolved
by again placing them next to each other, i.e., by direct comparison. This truth
is the reason that the first, most basic through-the-lens focusing device, the
ground glass, proved inaccurate.
Understanding
why the basic ground glass fails to achieve accuracy is fundamental to defining
and understanding the problems of comparison, not only in sight, but in all
sensory perception. With a ground glass, it is necessary to focus
back and forth over the apparently sharpest setting, remembering how far one can
go in each direction before the image becomes obscured. Not only is our memory
for the images at the various stages of focus undependable, but the eye quickly
loses acuity and begins to see longer stretches as sharp the longer one tries to
focus. Experienced photographers using a ground glass know that focusing should
be done quickly, going back and forth over the point of apparent focus as few
times as possible. Otherwise, whatever little bit of accuracy the ground glass
can deliver will suffer, as the memory vacillates more and more the longer the
process continues. Absolute accuracy through direct comparison was achieved with
a ground glass in the Messraster, which is simply a divided ground glass that
eliminates the use of memory by allowing the viewer to compare directly the too
far and too close settings, right next to each other. The main reason this
little known, but very important, device achieves its accuracy is that it
utilizes the only accurate sensory capacity, direct visual comparison, and
eliminates the need to use the memory.
The other
senses have the same problem: memory of sensory impressions is undependable and,
with even slightly extended non-direct comparison, the characteristics of the
different sensory stimuli blend into each other and the differences become
blurred. With smell, the longer one compares different scents, without long
waits in between, the more the difference blurs. And the longer one sniffs a
scent, the less strongly one can smell it, to the point that one eventually
stops smelling it. With taste, flavors quickly weaken and our palate also
quickly blends the flavors, losing its ability to differentiate them. For
example, salt lovers know that the more salt they use the more they have to add,
because, like being subjected to a particular smell for a long period of time,
the palate quickly stops tasting the salt until more is
added.
Evaluations
of delicate differences in tea and coffee flavors, perfume scents, and other
similar sensory products, have to be performed by highly sensitive, specially
trained experts under specially controlled circumstances. Even the slightest
distraction can ruin their work, because of the great demands these activities
place on the memories even of those trained individuals who are intimately
familiar with the various pitfalls of their work. And great demands are made on
these people in regard to physical discipline, poise, personal delicacy and
refinement in order to preserve their physical
sensitivity.
The body of
the listener is another variable to which differences perceived in AB testing
can be attributed. The body changes throughout the day. Disciplined people are
usually not as sensitive when they wake up as later in their progress through
the day. Physically undisciplined people's bodies also vary throughout the day,
though not necessarily in the sense of becoming more sensitive over the course
of the day. All sensory perception is conveyed to us through our bodies. There
are no abstract sensations. It is well-known that various states of tension and
relaxation bring with them differing amounts of sensitivity. Disciplines like
Yoga, Zen, etc. can heighten sensitivity through manipulation of the
body.
The point
that must be especially emphasized in regard to activities that demand great
sensitivity is that, no matter how naturally gifted the person, a high degree of
physical sensitivity is a cultivated thing that has to be purposely achieved and
sustained. (Even Mozart, probably the most naturally gifted human being with
regards to sensitivity, had to go through long training, had to be subjected to
the finest examples of art in Europe, and had
to mingle with the most refined, cultivated people of great artistic
discrimination and personal discipline in order to develop that gift. Yet most
people think they can simply sit down at any time, in any physical state, and
discriminate between subtle differences in the nuances of musical performance
and the small but often crucial differences in sound qualities between sound
components.) The need to cultivate physical delicacy and discipline during
activities involving sensory perception is well known in the fields of touch,
taste, and smell. In those fields, not only are controlled circumstances
considered necessary for all critical perception, but the people doing the
perceiving are expected to preserve the physical delicacy and discipline
necessary for such perception. On the days they work, tea tasters, perfume
testers, and wine-tasters follow strict physical and dietary regimens designed
to keep them in the most sensitive physical state. And their surroundings are
carefully controlled to provide an ideally calm and non-distracting environment
for extremely delicate perceptive work.
But similar
conditions regarding the listener's physical discipline, refinement, and
surroundings etc., are seldom, if ever, insisted upon in attempts at audio
comparison, even though hearing is the most complex, most variable, most easily
disturbed, least dependable and most difficult to monitor of all the senses.
That is partly because hearing is also the most taken-for-granted of all the
senses, and the least often tested.
There is
also an enormous range of differences in hearing acuity. There are people barely
able to hear a loud sound and those who hear that sound so loudly that it is
almost painful. There are those who can concentrate on a sonic event intensely,
for long periods of time, and those who cannot sustain their concentration for
more than a second or two and allow any little thing to distract them. There are
those who can keep their mind firmly on what is happening in the exact juncture
of the present, and those who are either anticipating what is coming or, having
missed some detail or lost their concentration, lose themselves in reviewing
what they have heard, while the music or other sonic event continues, thus, in
effect, missing everything. In truth, most normal people who have not had
specific training, exhibit some form of these aberrations in their manner, i.e.,
habits, of hearing. In normal life, without utilizing complicated testing that
is not completely dependable, it is extremely difficult, if not impossible, even
to notice let alone differentiate differences in the way we hear. However,
correct habits of listening can be trained, and with the help of basic yoga-type
disciplines, both Eastern and Western, concentration, the ability to resist
distractions, and the ability to keep the mind empty and concentrated solely on
the (sonic) events of the moment (of the present) can be
developed.
But the
usual assumption in society is that people who do not need a hearing aid (i.e.,
do not have a medically proven hearing disability) all hear essentially alike.
Because most of our hearing is used to receive dispassionate information, which
is conveyed in the meaning of words and does not depend on the nuance of how it
is conveyed, we do not think about all the different ways a sound can be
produced or all the different ways we can hear it. Because most of us can make
out the basic information in the sounds we hear, i.e., words and their meanings,
and most daily communication is mainly to convey such information, we generally
do not make demands of sensitivity, especially sensitivity to nuance, upon our
hearing and we ignore the differences in hearing that our different physical
states (moods) will produce. Yet AB testing deals mostly with the perception of
differences in sound qualities and nuances, and not at all with the conveying of
information.
It
has been necessary to establish the role of physical refinement in sensory
perception because it is an important factor in attempts at AB testing in sound,
and, for that matter, in all comparison of sonic impressions. There is always
that distinct possibility that any differences in the way sounds were heard
could be just as much because the listener moved, became upset, tensed, or
otherwise changed his/her physical state as because the sounds actually differed
in the manner they were produced at the source.
There is
also the distinct probability with AB testing when one performance or component
is less delicate than another to which it is compared, that the listener will
still be vibrating in the vibration of the more coarse example when the finer
one is played. Since we actually hear the vibrating of our own bodies, the
delicacies of the finer example will be filtered through, i.e., produced by, the
listener's own more coarsely vibrating body, and, therefore, changed or not
heard at all.
In
fact, except for an extremely few people with the natural talent of the true
orchestra conductor, who can hear with great acuity even when physically and
mentally active, the only time people--any people--are actually able to hear and
experience the nuances of finely-performed, high-quality music, is when they are
absolutely calm, quiet, fully concentrated, and perfectly still. Without
specific training, few people are able to place themselves in such a state at
will and, therefore, have to wait for the moments when it happens by itself,
i.e., when they just happen to gravitate into the right mood. Most of us have
certain recordings that can make us cry, or uplift us, or cause such piercingly
exquisite experiences that we feel like our heart has jumped into our throat, to
utilize a particularly apt colloquial description. But we also know that we
cannot just sit down and have those experiences happen at will. We have to wait
until we are in the right "place" to be able to experience
them.
Since fine
music is seldom available at the same moment that most people are physically
able to be receptive to it, few people ever hear and enjoy the felicities of
fine musical performance, and those that do are not able to do so very often or
for very long periods of time. Therefore, few people have even the slightest
preparation for any kind of sonic comparisons. They lack the necessary acuity,
awareness of the need for physical discipline, practiced concentration over long
periods of time, etc. to be dependable subjects. In AB, or other relatively
quick forms of comparison, there always has to remain the suspicion that
differences in how the sound was heard were as much due to physical instability
in the listener as to differences in the sound.
I
have made the point that, because direct visual comparison is generally easy to
perform and the most often utilized method of differentiation in our lives, we
tend to take it for granted that we can accomplish the same thing with the other
senses. I have also shown that direct comparison is simply not possible with the
other senses because no sensory comparisons with sound, smell, touch, or taste
can utilize direct, simultaneous comparison and must, therefore, use the memory
(touch would seem to come closest to visual comparison because most things being
touched do not change appreciably over the short periods of needed to attempt
comparison, and we can simultaneously touch two different things with our two
hands. But no two hands or fingers are exactly the same. Alternately touching
two objects with the same body part again makes demands on our undependable
memory). I have also made the point that, in sound, there must be even greater
uncertainty than with other senses, because sound is the most fleeting of
sensory stimuli. Sound cannot linger, as in taste or smell, and cannot remain
still, as in touch and direct visual comparison. Only carefully engineered
mechanical sounds can be absolutely steady and unwavering. All other sounds,
even seemingly sustained ones, are constantly changing, i.e. fluctuating, in
time.
Am
I saying that sonic comparisons are impossible? Not at all. I am saying that
they must be accomplished in a completely different, unrelated manner from
visual comparisons and with even more care than the extraordinary care taken in
serious comparisons of taste, touch, and smell.
But how?
The answer lies in understanding that quick, immediate comparisons do not work.
The way to make dependable comparisons is through great familiarity with the
audio components, sounds, performances, etc. that are to be compared. They must
be listened to enough times for the persons doing the comparing to be sure they
have really heard and experienced all of the subtle content of the sounds. And
once they are sure they have accurately heard the content of the sounds, they
must become familiar with it. That usually means living with the sounds over an
extended enough period of time to allow the listener to be fresh and attentive
during listening periods. The whole process can take hours, days, or even weeks.
With familiarity, memory becomes dependable, as long as proper precautions have
been taken to maintain the same sound-quality, the same room conditions, a
refined state of body, etc., during all listening.
Furthermore,
to be truly accurate, all listening comparisons, including those in medical
testing of hearing, should be made under circumstances in which the listeners
feel completely comfortable, as they would in their own homes. Listening periods
should not exceed the listener's comfortable span of attention, and the sonic
programs to be compared must be repeated often enough for the listeners to be
absolutely sure they are familiar with the programs. Above all, during these
periods, there should be no interruptions or physical exertions on the part of
the listener that might disturb his/her physical equilibrium, which means that
the programs have to be turned on and off by someone other than the
listener.
But these
preconditions should not be misconstrued as possible means of better conducting
AB comparisons. AB testing has absolutely no validity in audio
comparisons. Far from being a means of bringing scientific accuracy to
audio evaluations, as believed by many audio practitioners, AB testing is based
on human capacities that are undependable and do not at all fulfill the
requisites of scientific accuracy. There are no exceptions. But the invalidity
of AB testing is particularly true when music is used for the comparison,
especially when a comparator device is used to switch back and forth between
audio components while the music is playing. For that process to be at all
logical, the exact same portion of the music would have to be heard each time
the switch is operated. But the repetition of exactly the same short sequence of
music (or any other sonic program) would bring with it its own irritations that
would disturb the listener and negate the test.
The
Anstendig Institute strongly recommends that all people professionally involved
in AB testing and other comparison of sensory impressions thoroughly study and
understand, through first-hand experience and demonstration, the principles
involved in the various available photographic focusing devices. It is important
to an understanding of all sensory perception to know why these devices that use
the human eye are inaccurate. It is also of the greatest importance to
understand the truth about depth of field, in the photographic sense: that it
really pertains to unsharpness, not sharpness; that depth of field does not
exist in the sense of depth of sharpness, but is, rather, a description of the
extent to which increasing unsharpness can be tolerated before it disturbs the
viewer, a parameter that is entirely subjective and, therefore, undependable
because it is determined by and changes with the sensitivity and mood of the
individual viewer.
This
understanding of photographic images and the effect of sharpness is so crucial
because it is with visual comparisons that the human being usually begins
conscious, purposely initiated sensory comparisons. With visual comparisons, we
first and most dependably develop our sense of discrimination, i.e., our ability
to differentiate and evaluate subtle differences in all things. But there are
important shortcomings and misunderstandings in photographic imagery that carry
over into all visual imagery and, unless can we are aware of them, ultimately
affect our powers of discrimination. Along with sounds, photographic imagery is
probably the most omnipresent element and influence in our modern life. It
pervades everything we do, especially when we mistakenly attempt to utilize
processes pertinent only to sight in our work with the other senses.2 The misuse
of visual criteria in hearing would by itself be bad enough. But the fact that
our understanding of visual images is based on wrong assumptions makes the use
of visual criteria all the worse.
Unfortunately,
a large part of the audio research that has already been published has utilized
AB or similar testing that is simply a misapplication of visual criteria in the
realm of sound. All of that research has, therefore, to be considered invalid.
If any valid conclusions have been reached by these methods, their acceptance
will have to wait until they can be confirmed by means that are scientifically
accurate. It is difficult to comprehend the enormity of this situation. Whole
edifices of scientific thought, methods, and practice have been built upon this
scientifically invalid procedure. No matter how the procedure is refined (as in
double-blind AB testing, using two or more blindfolded subjects and comparing
components, etc., in such an order that the subjects could not guess their
identity), there is no possibility of dependably recognizing subtle differences.
In AB testing, any differences being recognized and compared have to be so large
that they should be apparent to the same people in any kind of
listening.
An
argument has blazed for years between those in the audio community who swear
they hear subtle differences between components they have lived with and those
in the AB testing community who insist that AB testing has proved those people
wrong -- that those people must be imagining the differences, because carefully
controlled AB testing has shown that the differences do not exist. There are
many pitfalls in any kind of listening. But we have seen that those who live
with their components before evaluating them could very well be correct in their
evaluations. At least they are using a valid procedure.
What is
clear is that those using AB testing have not been using a valid procedure.
Unless the misconceptions of sight and sound in the scientific world are quickly
cleared up, when current or future generations finally realize the truth, they
will have to throw out most previous research and, therefore, almost their whole
fund of knowledge, because it will all have been based on invalid premises and
carried out under invalid conditions.
1
See
Messraster patents of 1939 and 1966, in
USA and all
Germanies. This patent, well known in the optical and photographic fields at the
time, is the only focusing device that utilizes direct comparison as the actual
focusing method and is the only patent that claims focal-point-exact focusing of
lenses in cameras. The correctness of its assumptions was attested by the
leading expert witnesses in the field of optics of the time, the optical
institutes of the technical universities of Germany (The Anstendig Institute has
copies of the affidavits from the Optical Institute of the Berlin Technical
University, which, at the time, was the leading optical institute of the world).
From the time it appeared, the Messraster was fought against by the industry and
kept off the market. It still remains the only possibility of achieving
dependable focal-point-exact focusing in all photography.
The Anstendig Institute is a
non-profit research and educational institute that studies the vibrational
influences in our environment, particularly those of sight and sound, and how
they affect sensory perception. Its papers on sound reproduction, problems of
focusing in photography, psychology of hearing and seeing, and erratic
vibrational influences that affect our lives are widely distributed throughout
the world. All are available free of charge.
