Existential Technology of Synthetic Synesthesia
for the Visually Challenged
University of Toronto
10 King's College Road
Existential technology (existech) is an attempt at defining a
technological framework for self-determination and mastery over one's
own destiny. The physical embodiment of existech is called ``WearComp''
and was described in
GO TO THE FEATURE ARTICLE of FEBRUARY 1997................................
IEEE Computer, Vol 30, No. 2].
WearComp originated as a new synergy between humans and computers
for photographic ``lightpainting'' experiments in the 1970s
and early 1980s [WearComp2, completed in 1981, is pictured in
and a close-up of the "keyboard" input device is pictured in
WearComp has evolved into an assertiveness tool for self-determination.
Humanistic Intelligence [Ars Electronica Symposium, plenary lecture,
Sept. 10, 1997 = http://wearcam.org/previous_experiences/ars_electronica/]
is an alternative to environmental intelligence
Some recent performance pieces based on WearComp include
SafetyGlasses, and Live.
ShootingBack was based on a covert version of the WearComp
apparatus [WearComp7 and WearComp8, pictured in
In ShootingBack, a personal documentary video was made in
establishments where photography is strictly prohibited, yet
surveillance is used extensively (e.g. department stores, etc.).
Representatives of these "totalitarian surveillance"
establishments were questioned as to why they capture images of
people without their permission.
SafetyGlasses (performed in Linz, Austria, Sep. 8-12,
1997) was an attempt to hold a "mirror" up to society, in particular
to the surveillance superhighway (SS).
By allowing representatives of the SS to see their own
rhetoric (total surveillance as utopian safety)
in a "mirror", and to be "photographed" "for their own protection",
a symmetry was created. The rhetoric of "public safety" was
reflected through the construction of "personal safety" devices,
such as in SafetyGlasses and Live.
"Live" is a performance in which Wearable Wireless Webcam is
turned into a life-support system of sorts [FIGURE 5].
In this performance, the personal safety device is configured
so that its removal would cause the death of the wearer.
A large capacitor
bank is connected through a silicon-controlled rectifier (SCR) who's
gate is grounded through heavy cabling sewn into the clothing.
Cutting through the cabling would unground the gate, allowing the
pullup resistor to anode to turn on the SCR, connecting the wearer
to a lethal dose of 480 volts DC. Because of the latching nature
of the SCR, death is assured once the SCR is turned on.
The performance is called "Live" partly because it produces
"live video", (transmitted "Live"). But in addition to the fact that
Wearable Wireless Webcam is a life camera on the internet, this
very camera suggests a possible lifeline to friends and relatives
who might look out for the safety of the individual to make sure
that that individual is properly treated (e.g. as a means of
preventing human rights violations).
Lastly, Live WearComp = Live Human.
Live asserts the right to self-surveillance (by friends and relatives)
for personal safety.
The removal of the apparatus, which is something on might expect from
an entity intending to commit human rights violations (such as torture
or other misconduct), is something that would result in death.
Thus murder-as-misconduct is a deterrent to other lesser forms of
misconduct (e.g. torture, improper treatment, etc.).
These performances were influenced by conceptual art
(which challenged the notion of art as a commodity). In
the spirit of the situationist movement, public space (what's
left of it, as well as some of the private space that has replaced it)
is used as the performance space, to further challenge the notion of
art as commodity, for then there is no cost of admission to
attend the performance. In the spirit of the situationist movement,
the work uses the principle of detournement; it re-situates ordinary
everyday objects in a disturbing and disorienting fashion in order
to challenge our pre-conceived biases. Furthermore, in addition
to appropriating the tools of the oppressor, it follows the
reflectionist ideal [Reflectionism and Diffusionism, Leonardo, 1998
of turning those same tools against the oppressor.
What these performances and experiences have in common is their extended
duration, and their incorporation into day-to-day living, in the spirit
of Linda Montano's "Living Art", e.g., three years of "Wearable Wireless
Webcam" is somewhat related to Montano's "three day blindfold",
in the sense that vision was somewhat degraded/mediated by WearCam
over this time period. Some variations of this piece (e.g. My Manager,
Painful Disconnect, etc., where I would experience a painful electric
shock for images not sent/received to the WWW,
or loss of Internet connectivity)
had some similarities to her year long performance tied to Hsieh,
although in my case, the "rope" was invisible --- a virtual tether.
Many of my performances/experiments/experiences deal
with forms of computer-mediated reality, and this allows me to
experience the world differently. My "WearCam" invention,
for example, has provided me with an enhanced sense of awareness
of light and shade. In addition to altering or enhancing existing
sensory modalities, I also experiment with new senses that we
ordinary don't have. These extra sensory capabilities further
contribute to the "living art" and reflectionist performance pieces.
SYNTHETIC SYNESTHESIA FOR A SIXTH OR SEVENTH SENSE
But slow, what light through yonder window breaks.
Tail lights are red, headlamps are blue
Time seems to fly when I spend it with you
Let me have peace, nature serene
Lie still in the grass, the trees ever so green
--(c) Steve Mann, 1987
In just eight hours your thesis is due
I am the clock and I'm watching you
The hurrier you go the aheader I get
But when you wait in the rain you're gonna get wet
'Cause when you're waiting my hands stand still
Time is a concept you just can't kill
--(c) Steve Mann, 1990
Mediated Reality (MR) has been presented as a framework for
computationally augmenting, diminishing, or otherwise altering
our perception of reality [http://wearcam.org/mr.html].
Mediated reality may include, in addition to video, an
audio reality mediator, or, more generally, a `perceptual
reality mediator'. This generalized mediated perception system
may include deliberately induced
Synesthesia is manifest as the crossing of sensory modalities,
as, for example, the ability (or as some might call
a disability) to taste shapes, see sound, etc.
Examples I have explored include seeing sounds, hearing light,
etc., but some of the more interesting ones pertain to the addition
of a new sense (depending on who's opinion one takes, we already have five
or six senses, so the new one would be the sixth or seventh).
One such new sense that I have created and explored is that of radar.
In particular, I developed a number of vibrotactile wearable radar systems
in the 1980s, of which there were four primary variations:
One of the problems with this work was the processing, which, in the early
1980s embodiments was done using a wearable computer of much lesser
capability than Today's wearable computers.
Today's rig, capable of computing the
chirplet transform [IEEE Trans. Sig. Proc. Vol.43, No.11, November 1995]
in real time, and processors of the future which will compute
it at much higher resolution, suggest a fully digital VibraVest
- `CorporealEnvelope': baseband
output from the radar system was envelope-detected to provide
a vibrotactile sensation which was essentially
proportional to the overall energy of the return.
This provided the sensation of an extended `envelope'
around the body, in which I could feel objects at a distance.
In later (late 1980s) embodiments of `CorporealEnvelope',
envelope detection was done after splitting the signal into
three or four separate frequency bands, each driving a
separate vibrotactile device, so that each would convey a
portion of the Doppler spectrum (e.g. each corresponding to
a range of velocities of approach).
In another late 1980s embodiment, variously colored
lamps were used, attached to the wearer's eyeglasses to
provide a visual synesthesia of the radar sense. In one particular
embodiment, red, green, and blue lamps were used, such that
objects moving toward the wearer illuminated the blue
lamp, while objects moving away illuminated the red lamp.
Objects not moving relative to the wearer,
but located near the wearer appeared green.
This work was inspired by
using the metaphor of the natural Doppler shift colors one
might experience while approaching the speed of light, or equivalently,
what one might experience if light were slowed down to the
speeds that we encounter in our day-to-day life.
(See "slowlight" quote entitled "But slow, what light through
yonder window breaks" above.)
In a much more recent (1996) version of `corporeal envelope',
I used seven vibrotactile elements, each conveying a portion of
the Doppler spectrum, together with one of my early 24.360GHz
wearable radars. It is also not difficult to imagine a
continuum of vibrotactile elements that would convey a
continuous Doppler spectrum.
- `VibroTach' (vibrotactile tachometer): the speed of objects
moving toward or away from the wearer was conveyed, but not
the magnitude of the Doppler return (e.g. it was not possible
to distinguish between objects of small radar cross section and
those of large radar cross section). This was done by
having a Doppler return drive a motor, so that the faster an
object moved toward or away from the wearer, the faster the
motor would spin. The first VibroTach was built as an art
installation, to drive a clock (see ``TimeWarp'' quote above).
Upon holding onto the clock in ``TimeWarp'', it was noticed
that one could ``feel'' the motion of objects at a distance.
Holding onto the clock and moving back and forth created a
surreal sensation as though strings were attached to objects
in the room, and that these strings were passing through the
clock's gear trains in a way that I could feel the vibration
through my hands. The spinning motor could be felt as a
vibration having frequency proportional to that of the
dominant Doppler return. In wearable versions of `VibroTach',
I could feel very small increments of motion (e.g. someone
sneaking up behind me). I also explored the use of multiple
vibrotactile transducers (typically permanent-magnet
loudspeakers) to simulate motion without movement in the same way
that a light-chaser simulates a visual motion percept
by turning lights on and off in the proper sequence.
The initial reason for using multiple units around the body was
to be able to perceive the difference between clockwise
(e.g. toward the body) and counter-clockwise (away from the
body) motion of radar targets. Early 1980s versions
of VibroTach used synchronous AC motors (driven directly from
the plates of a backpack-based audio amplifier,
e.g., with output transformer removed), while later
versions tended to be characterized mostly by the use of DC
motors (driven by solid-state amplifiers and the like).
- A fly-swatter has holes in it so that the fly will not
feel the draft from the swatter and fly away before it is
hit. Humans have a much weaker ability to sense air currents
of possible predators. Accordingly, a variation of VibroTach
used a fan instead of relying on just the vibrations of a motor.
In one such variation, called "BackDraft", the fan was situated
in a backpack, so that it would blow air on my
back when objects approached from some distance
away. A person sneaking up behind the wearer
creates an air current that is perceptible from a great distance
away. By the time the predator is within stabbing distance, the
fan is spinning quite rapidly, and the wearer is certainly aware
of the presence of someone sneaking up from behind.
Early embodiments of BackDraft used fans plugged into the
backpack-based audio amplifier [FIGURE 6, top]. The "accessory"
AC outlet on the back of the amplifier was connected directly to
the plates, bypassing the output transformer, so that various
everyday objects (fans, alarm clocks, could be plugged in
easily for quick prototyping. Later a solid-state version was
made for driving DC devices [FIGURE 6, bottom],
smaller and lighter wearable computer systems and miniature
- `Electric Feel Sensing': the entire doppler signal
(not just a single dominant speed or amplitude) was conveyed
to my body. Thus if there were two objects approaching me
at different speeds, I could discern them separately from a
single vibrotactile sensor (e.g. both at the same point on my
body). Various embodiments of `electric feel sensing' included
direct electrical stimulation of the body using an automobile
spark coil, and later a trigger coil from an electronic flashlamp,
as well as the use of a single broadband
vibrotactile device. An example of the latter included a
backpack-based 6~by~9 inch elliptical loudspeaker mounted in
a wooden cabinet, driven by an audio amplifier made from
transistors salvaged from a surplus electronic cash register.
The above are wearable examples of what Hiroshe Ishii refers to
as "tangible user interfaces". In this sense, WearComp functions as
a tangible user interface to the real world, and functions as a true
extension of the mind and body. It is hoped that such a device will,
through its ability to enhance situational awareness, assist the
visually challenged as well as
function as a personal safety device.
The ability to feel objects at a distance would provide some navigational
aid to the visually challenged, in addition to the personal safety that
they might wish to have owing to their otherwise being easy targets
I would like to thank my former PhD thesis advisor, Rosalind Picard,
my master's thesis advisor, Dr. Simon Haykin, as well as many others,
including Hiroshe Ishii, Krzysztof Wodiczko, and Julia Scher, who
have contributed much in the way of thoughts, ideas, etc..
I would also like to acknowledge those involved with the recent Linz
performance, Gerfried Stocker, Jutta Schmiederer, Patrizia Maier,
Hans Soukup, Andy Kleen, etc., as well as those at the List Visual
Arts Center, Jennifer Riddell, Katherine Kline, Jonathan Roll, etc.,
for much in the way of creative input during the organization of my
upcoming show dealing with surveillance in society.