VibraVest/ThinkTank:

Existential Technology of Synthetic Synesthesia for the Visually Challenged

Steve Mann
University of Toronto
10 King's College Road
Toronto, Ontario,
M5S 1A4

INTRODUCTION

FIGURE 1

WearComp has evolved into an assertiveness tool for self-determination. Humanistic Intelligence [Ars Electronica Symposium, plenary lecture, Sept. 10, 1997 = http://wearcam.org/previous_experiences/ars_electronica/] is an alternative to environmental intelligence [FIGURE 3].

Some recent performance pieces based on WearComp include My Manager, ShootingBack, SafetyGlasses, and Live. ShootingBack was based on a covert version of the WearComp apparatus [WearComp7 and WearComp8, pictured in FIGURE 4]. In ShootingBack, a personal documentary video was made in establishments where photography is strictly prohibited, yet surveillance is used extensively (e.g. department stores, etc.). Representatives of these "totalitarian surveillance" establishments were questioned as to why they capture images of people without their permission.

SafetyGlasses (performed in Linz, Austria, Sep. 8-12, 1997) was an attempt to hold a "mirror" up to society, in particular to the surveillance superhighway (SS). [http://sicherheitsglaeser.aec.at, http://n1nlf-11.mit.edu/safetyglasses, http://wearcam.org/safetyglasses] By allowing representatives of the SS to see their own rhetoric (total surveillance as utopian safety) in a "mirror", and to be "photographed" "for their own protection", a symmetry was created. The rhetoric of "public safety" was reflected through the construction of "personal safety" devices, such as in SafetyGlasses and Live.

"Live" is a performance in which Wearable Wireless Webcam is turned into a life-support system of sorts [FIGURE 5]. In this performance, the personal safety device is configured so that its removal would cause the death of the wearer. A large capacitor bank is connected through a silicon-controlled rectifier (SCR) who's gate is grounded through heavy cabling sewn into the clothing. Cutting through the cabling would unground the gate, allowing the pullup resistor to anode to turn on the SCR, connecting the wearer to a lethal dose of 480 volts DC. Because of the latching nature of the SCR, death is assured once the SCR is turned on.

The performance is called "Live" partly because it produces "live video", (transmitted "Live"). But in addition to the fact that Wearable Wireless Webcam is a life camera on the internet, this very camera suggests a possible lifeline to friends and relatives who might look out for the safety of the individual to make sure that that individual is properly treated (e.g. as a means of preventing human rights violations). Lastly, Live WearComp = Live Human.

Live asserts the right to self-surveillance (by friends and relatives) for personal safety. The removal of the apparatus, which is something on might expect from an entity intending to commit human rights violations (such as torture or other misconduct), is something that would result in death. Thus murder-as-misconduct is a deterrent to other lesser forms of misconduct (e.g. torture, improper treatment, etc.).

These performances were influenced by conceptual art (which challenged the notion of art as a commodity). In the spirit of the situationist movement, public space (what's left of it, as well as some of the private space that has replaced it) is used as the performance space, to further challenge the notion of art as commodity, for then there is no cost of admission to attend the performance. In the spirit of the situationist movement, the work uses the principle of detournement; it re-situates ordinary everyday objects in a disturbing and disorienting fashion in order to challenge our pre-conceived biases. Furthermore, in addition to appropriating the tools of the oppressor, it follows the reflectionist ideal [Reflectionism and Diffusionism, Leonardo, 1998 http://wearcam.org/reflectionism.html] of turning those same tools against the oppressor.

What these performances and experiences have in common is their extended duration, and their incorporation into day-to-day living, in the spirit of Linda Montano's "Living Art", e.g., three years of "Wearable Wireless Webcam" is somewhat related to Montano's "three day blindfold", in the sense that vision was somewhat degraded/mediated by WearCam over this time period. Some variations of this piece (e.g. My Manager, Painful Disconnect, etc., where I would experience a painful electric shock for images not sent/received to the WWW, or loss of Internet connectivity) had some similarities to her year long performance tied to Hsieh, although in my case, the "rope" was invisible --- a virtual tether.

Many of my performances/experiments/experiences deal with forms of computer-mediated reality, and this allows me to experience the world differently. My "WearCam" invention, for example, has provided me with an enhanced sense of awareness of light and shade. In addition to altering or enhancing existing sensory modalities, I also experiment with new senses that we ordinary don't have. These extra sensory capabilities further contribute to the "living art" and reflectionist performance pieces.

SYNTHETIC SYNESTHESIA FOR A SIXTH OR SEVENTH SENSE

\begin{quote}
But slow, what light through yonder window breaks.
-------------------------------------------------
Tail lights are red, headlamps are blue
Time seems to fly when I spend it with you
Let me have peace, nature serene
Lie still in the grass, the trees ever so green

--(c) Steve Mann, 1987
\end{quote}

\begin{quote}
TimeWarp
--------
In just eight hours your thesis is due
I am the clock and I'm watching you

The hurrier you go the aheader I get
But when you wait in the rain you're gonna get wet

'Cause when you're waiting my hands stand still
Time is a concept you just can't kill

--(c) Steve Mann, 1990
\end{quote}

Mediated Reality (MR) has been presented as a framework for computationally augmenting, diminishing, or otherwise altering our perception of reality [http://wearcam.org/mr.html]. Mediated reality may include, in addition to video, an audio reality mediator, or, more generally, a `perceptual reality mediator'. This generalized mediated perception system may include deliberately induced synesthesia.

Synesthesia is manifest as the crossing of sensory modalities, as, for example, the ability (or as some might call a disability) to taste shapes, see sound, etc.

Examples I have explored include seeing sounds, hearing light, etc., but some of the more interesting ones pertain to the addition of a new sense (depending on who's opinion one takes, we already have five or six senses, so the new one would be the sixth or seventh).

One such new sense that I have created and explored is that of radar. In particular, I developed a number of vibrotactile wearable radar systems in the 1980s, of which there were four primary variations:

1. `CorporealEnvelope': baseband output from the radar system was envelope-detected to provide a vibrotactile sensation which was essentially proportional to the overall energy of the return. This provided the sensation of an extended `envelope' around the body, in which I could feel objects at a distance. In later (late 1980s) embodiments of `CorporealEnvelope', envelope detection was done after splitting the signal into three or four separate frequency bands, each driving a separate vibrotactile device, so that each would convey a portion of the Doppler spectrum (e.g. each corresponding to a range of velocities of approach). In another late 1980s embodiment, variously colored lamps were used, attached to the wearer's eyeglasses to provide a visual synesthesia of the radar sense. In one particular embodiment, red, green, and blue lamps were used, such that objects moving toward the wearer illuminated the blue lamp, while objects moving away illuminated the red lamp. Objects not moving relative to the wearer, but located near the wearer appeared green. This work was inspired by using the metaphor of the natural Doppler shift colors one might experience while approaching the speed of light, or equivalently, what one might experience if light were slowed down to the speeds that we encounter in our day-to-day life. (See "slowlight" quote entitled "But slow, what light through yonder window breaks" above.) In a much more recent (1996) version of `corporeal envelope', I used seven vibrotactile elements, each conveying a portion of the Doppler spectrum, together with one of my early 24.360GHz wearable radars. It is also not difficult to imagine a continuum of vibrotactile elements that would convey a continuous Doppler spectrum.
2. `VibroTach' (vibrotactile tachometer): the speed of objects moving toward or away from the wearer was conveyed, but not the magnitude of the Doppler return (e.g. it was not possible to distinguish between objects of small radar cross section and those of large radar cross section). This was done by having a Doppler return drive a motor, so that the faster an object moved toward or away from the wearer, the faster the motor would spin. The first VibroTach was built as an art installation, to drive a clock (see ``TimeWarp'' quote above). Upon holding onto the clock in ``TimeWarp'', it was noticed that one could ``feel'' the motion of objects at a distance. Holding onto the clock and moving back and forth created a surreal sensation as though strings were attached to objects in the room, and that these strings were passing through the clock's gear trains in a way that I could feel the vibration through my hands. The spinning motor could be felt as a vibration having frequency proportional to that of the dominant Doppler return. In wearable versions of `VibroTach', I could feel very small increments of motion (e.g. someone sneaking up behind me). I also explored the use of multiple vibrotactile transducers (typically permanent-magnet loudspeakers) to simulate motion without movement in the same way that a light-chaser simulates a visual motion percept by turning lights on and off in the proper sequence. The initial reason for using multiple units around the body was to be able to perceive the difference between clockwise (e.g. toward the body) and counter-clockwise (away from the body) motion of radar targets. Early 1980s versions of VibroTach used synchronous AC motors (driven directly from the plates of a backpack-based audio amplifier, e.g., with output transformer removed), while later versions tended to be characterized mostly by the use of DC motors (driven by solid-state amplifiers and the like).
3. A fly-swatter has holes in it so that the fly will not feel the draft from the swatter and fly away before it is hit. Humans have a much weaker ability to sense air currents of possible predators. Accordingly, a variation of VibroTach used a fan instead of relying on just the vibrations of a motor. In one such variation, called "BackDraft", the fan was situated in a backpack, so that it would blow air on my back when objects approached from some distance away. A person sneaking up behind the wearer creates an air current that is perceptible from a great distance away. By the time the predator is within stabbing distance, the fan is spinning quite rapidly, and the wearer is certainly aware of the presence of someone sneaking up from behind. Early embodiments of BackDraft used fans plugged into the backpack-based audio amplifier [FIGURE 6, top]. The "accessory" AC outlet on the back of the amplifier was connected directly to the plates, bypassing the output transformer, so that various everyday objects (fans, alarm clocks, could be plugged in easily for quick prototyping. Later a solid-state version was made for driving DC devices [FIGURE 6, bottom], used with smaller and lighter wearable computer systems and miniature wearable radar.
4. `Electric Feel Sensing': the entire doppler signal (not just a single dominant speed or amplitude) was conveyed to my body. Thus if there were two objects approaching me at different speeds, I could discern them separately from a single vibrotactile sensor (e.g. both at the same point on my body). Various embodiments of `electric feel sensing' included direct electrical stimulation of the body using an automobile spark coil, and later a trigger coil from an electronic flashlamp, as well as the use of a single broadband vibrotactile device. An example of the latter included a backpack-based 6~by~9 inch elliptical loudspeaker mounted in a wooden cabinet, driven by an audio amplifier made from transistors salvaged from a surplus electronic cash register.

The above are wearable examples of what Hiroshe Ishii refers to as "tangible user interfaces". In this sense, WearComp functions as a tangible user interface to the real world, and functions as a true extension of the mind and body. It is hoped that such a device will, through its ability to enhance situational awareness, assist the visually challenged as well as function as a personal safety device. The ability to feel objects at a distance would provide some navigational aid to the visually challenged, in addition to the personal safety that they might wish to have owing to their otherwise being easy targets for predators.

ACKNOWLEDGEMENTS