# Smart Clothing: The " Wearable Computer" and WearCam...

Steve Mann
(steve@media.mit.edu)
MIT Building E15-389,
20 Ames Street, Cambridge, MA02139
Now with University of Toronto, 10 King's College Road, Toronto,
Ontario, Canada, M5S 3G4, (416) 946-3387

appears in:
Personal Technologies, Vol. 1, No. 1, March 1997.

@ARTICLE{mannpt,
author = "Steve Mann",
title = "Smart Clothing: The Wearable Computer and WearCam",
journal = "Personal Technologies",
publisher = "Springer",
month = "March",
note = "Volume 1, Issue 1",
year = "1997"
}


## INTRODUCTION

Due to recent advances in miniaturization, the author's "wearable computer" invention of the late 1970s and early 1980s, comprising a backpack-based, tetherless computer system together with wireless communications, has been transformed from an awkward and cumbersome burden into a completely unobtrusive internet-connected multimedia computer built within an ordinary pair of eyeglasses and ordinary clothing. This transformation allows it to be worn constantly, with the goal of becoming a seamless extension of the body and mind. In addition to replacing one's cellular telephone, personal music system, dictating machine, pager, camcorder, laptop computer, PDA, and the like, with a single well-integrated unit, the apparatus can perform new and previously unexpected useful functions.

Equipped with various sensors which measure heart rate (and waveform), respiration, footstep rate (and waveform), etc., the apparatus can function as a personal safety device for reducing crime, as well as a personal health monitor for improving health care by encouraging individuals to take an active role in diagnosis and body-maintenance.

Presently, conductive cloth in various forms is available, due in part to certain people's desire to shield themselves from radiation effects of cellular telephones and the like, as well as for decorative purposes (the aesthetics of metallic fibers).

The "wearable computer" apparatus is imbedded within nontransparent clothing which provides shielding, (radio-frequency privacy, and solitude). Conductive fibers are also been used for the antenna structure, as well as for connectivity.

Smart clothing has been proposed as a form of existential media [Mann97]. Existential media defines new forms of social interaction through enhanced abilities for self expression and self-actualization, as well as through self-determination. The aspects of existentialism pertaining to existential media are best understood through a reading of [Frankl94].

Examples of smart clothing include internet-connected shoes (Figure 1) which allow one to run with a jogging partner located in some distant place, but connected via the network. Viewpoints might also be shared using the "eye-to-eye" glasses (where a portion of each runner's visual field comes from the other runner [Mann94].

[FIGURE 1]

[CAPTION] One of author's shoes, equipped with sensors, measure the impact of various parts of the shoe with the ground. This gives the wearer's computer a sense of the pace, so that, for example, joggers might pace themselves with a distant partner connected through wireless communications. Such technologies enhance one's capabilities of self-expression.

The SONY "Walkman" is another example of "existential media". Its ability to reclaim a portion of this sort of personal space that has largely been lost to "muzak" (a use of technology that has stolen much of our solitude) affords the user a good deal of self-determination. The proposed "wearable computing" apparatus allows this functionality (playing music) to be "disappear" into clothing and be integrated in a more natural manner.

Smart clothing contributes to the wearer's eudaemonic space [Mann97] which defines a boundary around the user, providing both privacy and self-expression.

## CRITERIA OF THE EXISTENTIAL COMPUTER

Towards the goal of self-determination and enhanced creative potential, a new form of computation has been proposed [Mann96], which is:
1. eudaemonic: the apparatus is subsumed into the "eudaemonic space" [Mann97] of the wearer (e.g. it may be worn, or otherwise situated in a manner that makes it part of what the user considers "himsef" or "herself", and in a manner that others also regard it as part of the user, e.g. not a separate object being carried by the user). It is sufficient that the interface be eudaemonic (e.g. some of the compute power can be remote);
2. existential: the apparatus is controlled by the wearer. This control need not require conscious thought, but the locus of control must be such that it is entirely within the wearer's domain (e.g. that it behave as an extension of the body). The functionality of the apparatus must also be potentially known to the user (e.g. although the user may not have the time or ability to completely understand its inner workings, the apparatus must not be built to deliberately obscure or hide its functionality). Furthermore, the apparatus provides the wearer with the ability to make its operation completely private and secure when desired. In addition to the obvious privacy afforded by its eudaemonic nature (e.g. securely attached to the body so that it might be less likely to be stolen when working in close quarters with adversaries), the output can be made private when desired by, for example, using a screen that cannot be read by others looking over the wearer's shoulder;
3. constant: (constancy of both operation and interaction):
• Operational constancy: it is always active while worn. It may have sleep modes, but it always has the ability to turn itself on and, for example, take notice of its environment by virtue of some sensing apparatus (e.g. it may "sleep", but it should never "die").
• Interactional constancy: one or more output channels (e.g. screen/viewfinder) are known (e.g. visible) to the user at all times, not just when the computer is being interacted with in a deliberate manner. In this way, context switching time to mentally engage with the apparatus is near zero;
The above criteria define what the author means by "wearable computer" or "existential computer".

The existential computer, which exists entirely within the eudaemonic space of the user is owned, operated, and controlled by the wearer, in order to prevent it from becoming a means of controlling the wearer. This principle is best understood through the distinction between "smart clothing" and "smart uniforms" provided in [Mann96a]. (An extreme example of external control is given in [Hoshen95].)

Furthermore, in contrast to either the desktop "on-the-job" computer, or the "smart uniform" [Mann96a], the existential computer is used for all facets of living (not just at a place of work), and there is an increased familiarity with the interface apparatus through its continuous use.

Although the proposed apparatus will replace much of the environmental technology, a goal is also to work in harmony with some remaining degree of environmental technology. Furthermore, not all environmental technology is counter to the principles of existential media. Most notably, is Ishii's so-called {\em ambient media} [Ishii]. Ambient media provide a natural and unobtrusive environmental awareness to the user. Natalie Jeremijenko's beautiful instrument called "Live Wire" [Jeremijenko] (called "dangling string" by many), which she developed as an artist in residence at Xerox PARC [Ishii][Weiser] is one example of tangible media; the string vibrates in response to network traffic and provides clues for peripheral attunement. It violates neither solitude (by virtue of its ambient nature, it remains in the perceptual periphery), nor privacy (for it has no capability to measure or record user-specific information).

## INTERACTION BETWEEN SMART CLOTHING AND ENVIRONMENTAL INTELLIGENCE

Even when we take off our "smart clothing" at night, we might still choose to keep on our "smart underwear" which controls the heater or air conditioner in the room (Figure 2). [FIGURE 2]

[CAPTION] Biosensors used in the author's "smart clothing" apparatus include ProComp ECG, EEG, respiration, and sweat sensor built into a Jantzen bathing suit. Upon arriving home, late at night, one is generally too hot from just climbing the stairs, etc., so when first going to sleep, the underwear tells the heater to turn off, but after a couple of hours sleeping, when one's metabolism slows down, the underwear senses the resulting changes in one's body temperature/conductivity, and turns up the heat. Our clothing of the future may some day be interoperable and interconnected, so that it keeps track of our physical condition and allows us to decrypt this information for evaluation by a doctor or other professional of our choosing. Further description of the "smart underwear" prototype, and anecdotes on the author's experience designing, building, and using it is appears in [Mann96b].

Having a patient wear his or her entire medical history would solve much of the medical records privacy problems we face today. With various biosensors, the most current and up-to-date information would be readily available within the very clothing that's taking the measurements. Of course, one would want to have one's medical records replicated (backed up) in the clothing of selected friends and relatives, to prevent data loss in the event of clothing failure.

This approach would eliminate the need for, and the possible abuses that can arise with, a central database of medical records, and would eliminate the need to for a person to venture through bureaucratic procedures to access his or her own medical information (e.g. when an individual wishes to obtain a second opinion from a different doctor). It would also eliminate the problems associated with smart cards, as clothing is almost always worn, while cards may be misplaced and inaccessible in times of emergency care. Epidemilogical research would still be possible with the patient's data --- participating patients could make the data accessible to organizations doing the research, but this would be done through a query to each participating patient's online "smart clothing" each time the data were needed, so that the patient's clothing would be kept "in the loop", that is, access logs would be automatically generated in the "smart clothing", so that patients could trace the history/usage of their data at a later date if desired.

## EXISTENTIAL TECHNOLOGY

The above examples illustrate even a small amount of "intelligence", owned operated and controlled by an individual, may strike a balance, and reclaim some of the eudaemonic space that has been eroded or lost by modern technology.

The first "existential computers", designed and built in the late 1970s and early 1980s (Figure 3) were originally developed for applications in the visual arts, in particular, still-life and landscape imaging [Mann96] --- combining multiple exposures of a static scene to a variety of different light sources, using principles of linearity and superposition of light [Ryals].

[FIGURE 3]

[CAPTION] Evolution of author's "existential computer": Large head-mounted CRT and separate inbound and outbound communications antennas of the late 70s were awkward. Author's waist-mounted television of the mid 80s was somewhat more comfortable but not constantly visible. Small viewfinders from consumer video cameras of the late 80s made possible an eyeglass-based system which later evolved toward author's current embodiment built into ordinary eyeglasses with the advent of newer solid-state display technology.

The current state of this simple invention exists as a miniaturized multimedia computer built unobtrusively into a pair of ordinary sunglasses (Figure 4).

[FIGURE 4]

[CAPTION] Current state of author's "existential computer" invention comprises a complete multimedia computer, with cameras, microphones, and earphones, all built into an ordinary pair if eyeglasses (except for some of the electronics items that fit into a small box into a shirt pocket, which will soon "disappear" altogether). This rig is currently running the Linux 2.0 operating system, with XFree86 (variant of X-windows).

with a small box that clips on the author's belt or fits into a shirt pocket. The input device is fashioned into an ordinary belt, using microswitches, so that one can type or issue commands unobtrusively. Other variations of the input device include microswitches sewn into clothing and the like.

The small size of the apparatus (negative size if one considers what it replaces --- cellphone, pager, walkman, camcorder, Personal Data Assistant (PDA), etc.), provides for a single unified point of contact --- all of the personal electronics we normally carry become subsumed into this one unit. Clearly the unit is capable of email, "i-phone" (internet telephony), and other forms of multimedia communication, so that it eliminates (or will eliminate, in time, once other people have such units) the need to carry a cellular phone or pager.

The system also has sound capability, so that it can function as a personal sound system, thus subsuming that form of existential media which provides the wearer musical self-determination. However, new forms of interaction --- interaction that goes beyond the aggregate functionality of all of the consumer electronics we normally carry --- is possible because they are all subsumed into a single unit.

For example, while one is jogging, one's shoes (Figure 1) can provide the system with an awareness of the running pace, so that music can be adjusted appropriately to maintain an even pace under program control. This allows the runner to use music as a "clocking" mechanism. Furthermore, since heart-rate is known to the system, it could establish a training schedule where the music serves a clock, in the feedback loop of a process that maintains constant heart rate, so that the pace is decreased or increased in accordance with the wearer's training goals. Such a system would also screen calls appropriately to delay them into the rest periods of the training schedule, so, if the wearer were doing interval training, and nearing the end of a max-level set, and an incoming call were detected, the apparatus might silence the first two or three "rings" of the "phone", rather than disturb the wearer during an instant when maximum concentration is desired.

## "WEARCAM" AS MECHANISM OF SELF-DETERMINATION AND PERSONAL SAFETY

"WearCam" (Figure 4) is a simple apparatus for effortless (yet controlled) capture, display, processing, and transmission of visual imagery. The WearCam invention is a particular variant of the "existential computer" invention which comprises the following entities: (1) one or more cameras that is/are attached to the body in some manner that permits both hands to be free (2) means of recording, processing, and transmitting images from the camera(s); processing means may be remote if facilitated through communications means (3) a display means that has the capability of presenting an image or stream of images from the camera, as well as other images (e.g. from the processor, or received from an external signal) and is also worn in a manner that permits both hands to be free

Clearly, today's digital cameras fail to meet the hands-free operation and processing/transmission capabilities. Another, perhaps less obvious yet important missing element from today's digital cameras is the electronic viewfinder. Again, it is quite obvious to anyone skilled in the photographic arts, that not only should there be a means of previewing previously captured images, but the importance of having the human in the loop of the creative process should be regarded as absolutely essential. Just as in {\em Deconstructing the automatic copier}, the benefit of realtime display of imagery should be obvious.

However, the WearCam viewfinder goes beyond merely setting the camera correctly. Since the apparatus is worn over an extended period of time, one adapts to it, and it begins to function as a true extension of the body. In this way, the viewfinder transcends being just a compositional tool, toward allowing the camera to "become" the eye of the wearer.

Long term use of WearCam gave rise to a heightened sense of awareness of light and shade; by living in a "mediated reality" environment [Mann94]. WearCam was an electronic/computational realization of the same sort of adaptation processes/experiments that have been previously done optically [Stratton1896][Kohler64][Dolezal82]

## HOMOGRAPHIC MODELING

Homographic modeling has also been used in the context of the new algorithm. In particular, the flowfield of a rigid planar patch in the scene is tracked, and virtual objects are inserted. In this way, a virtual "Post-It" note may be left on any flat surface (Figure 5) and will be seen only by the recipient through his/her WearCam. [FIGURE 5]

[CAPTION] Virtual message left on the wall under the entrance to the grocery store. When the recipient of the message approaches the store wearing his/her WearCam, he/she will see the message that has been left there. As the wearer moves his/her head, the message appears to be attached to the wall, because the homography of the plane is tracked and a projective coordinate transformation is performed on the message before it is inserted into the wearer's reality stream. (TOP) Lens distortion in WearCam results in poor registration. (BOTTOM) After correcting for lens distortion using the Campbell method [Campbell] the sub-pixel registration is possible.

Combining this algorithm with the "wearable face recognizer" proposed in [Mann96c], a virtual name tag, that appears together with other information, is possible. The "existential computer" recognizes the face, and then the flowfield of the entire room is used to give an approximate illusion of a ridid planar patch (only exact for if the wearer and named individual remain in the same place while the wearer looks around).

## PERSONAL DOCUMENTARY

A creative/expressive application of WearCam is in personal documentary. Wearing the apparatus in day-to-day life has resulted in the ability to spontaneously caputure events of interest, but in a more natural way than in other forms of personal documentary.

One such personal documentary, "ShootingBack", was a meta-documentary of sorts --- it was a documentary about making a documentary. In "ShootingBack" the author carries a standard camcorder into organizations characterized by totalitarian surveillance (e.g. department stores where photography is prohibited). Typically a conversation with the representative of the organization, where the representative states (in response to a querry as to why surveillance cameras are being used) something to the effect that only criminals would be afraid of cameras, or that the author must be trying to steal something if he were afraid of cameras, or that cameras should be of no concern. Then, pulling the camcorder out of a sachel, the viewer sees the eyecup of the camcorder moving up toward the "eye" (which is actually the eyeglass-based camera), and the resulting response from a representative (Figure 6).

[FIGURE 6]

[CAPTION] "ShootingBack" is a meta-documentary (a documentary about making a documentary. Here the viewer sees the process of shooting with a conventional camcorder, as recorded by WearCam. Because, after time, the wearer forgets that he is wearing the WearCam apparatus, the process of using the camcorder is captured in a natural manner giving the viewer the impression that he/she is living the life of the documentary artist.

Because there is also the output from the camcorder, in "ShootingBack", a split-screen effect, reminiscent of the movies "The Boston Strangler" and "The Thomas Crown Affair" is used. However, keeping to the integrity and reality-metaphor of the documentary, the natural window sizes are used at all times (e.g. nothing is cropped as is often done in other uses of split-screen). (See Figure 7) The effect is also reminiscent of the split-screen spatial division multiplexers used for "4-up" recordings of surveillance cameras.

[FIGURE 7]

[CAPTION] Split-screen format shows both the view from the viewer's "eye", as well as the output of the camcorder. Here a representative of an organization using totalitarian video surveillance responds to the author's camcorder.

Most of ShootingBack is shot in "rot90" (90 degree rotated) format, as this brings the viewer in closest proximity to the person being shot. The aspect ratio of the human face fits well this 3:4 format (as opposed to the usual 4:3 ratio of TV). The adaptation to "rot90" took place over a number of years of having worn the apparatus in that configuration, and it was found that a dual-adaptation space formed (e.g. where the author could adapt to seeing the world rotated 90 degrees, and than adapt back to seeing normally without the WearCam apparatus). The adaptation was quite pronounced, and confirmed the author's hypothesis that mappings near the identity have a longer-lasting per-unit effect (e.g. when WearCam was used in a non-rotated format, it was harder to switch back and forth).

## PRIVACY ISSUES

The question of privacy is often raised with respect to WearCam (and, in fact, part of the reason for its construction was to raise awareness of these issues). The apparatus suggests that shopkeepers and customers alike, police and ordinary citizens alike, etc., must respect the possiblity that they could be caught on camera. Examples of police brutality have often been brought to light because of small-size consumer camcorders. With WearCam, nobody will know whether or not a particular person is wearing a camera, as the present (and future) units are so small that they cannot be detected.

This creates an interesting form of balance. In the past, there have often been hidden cameras in the architecture (e.g. there is a large market for smoke detectors with built in cameras, clocks with hidden cameras inside, etc., and many of the companies selling these cameras offer government discounts). But with WearCam there is balance, for now a visitor to a new space can attain the same kind of respect that the owner of the space might have. WearCam gives the nomad a form of personal protection once available only to establishments.

In some sense, WearCam is also less of an invasion of privacy than the cameras of the establishment, for at least one still knows that when one is alone, one has privacy from WearCams. There is already in place a natural tendency to behave in a dignified manner in the presence of others, so it is really the cameras of the establishment that pose the gravest threat to privacy. Furthermore, a community of WearCams gives rise to a more distribued form of crime reduction than a network of cameras wired into the police station or the like.

## SAFETY NET

A WearCam transmitting and recording images at remote locations [Mann96] was explored as a personal safety device (e.g. to construct a continuous and indestructable personal documentary, functioning much like the "black box" flight recorders in airplanes).

A further improvement to the personal safety device includes the use of biosensors where the quotient of heart rate divided by footstep rate (as measured using the apparatus of Figure 1 provides a visual saliency index. Suppose that someone were to draw a gun and demand cash from the wearer. The likely scenario is that the wearer's heart rate would increase without an increase in footstep rate to explain it (in fact footstep rate would no doubt fall to zero at the request of the assailant). Such an occurrence would be programmed to trigger a "maybe I'm in distress" message to other members of the personal safety network ("SafetyNet"). These members, comrpised of friends and relatives, would then look at the video signal and might conclude merely that the wire has come loose from the shoe, or might find, in fact, that the distress is genuine. Such a system provides an alternative to Orwellian surveillance cities like Liverpool or Baltimore. Instead of relying on some distant and unaccountable force to keep watch over the society, individuals form small social networks that function much like small towns in a global neighbourhood watch.

The SafetyNet might be comprised of some members linked in with their home computers, as well as various members who have WearCams or just ordinary wearable computers (Figure 8)

[FIGURE 8]

[CAPTION] Personal Safety Network ("SafetyNet"). In the future, groups of individuals connected together wirelessly might create a virtual small-town global neighbourhood watch to reduce crime.

With technolgies like automatic face recognition, an individual can quickly verify that the person knocking on their door is really a police officer, or contractor, and not, for example, the Boston Strangler. This of course would require that police organizations and the like put images of their employees online for all to see. Mutual observability might make face capture more acceptable than the current totalitarian forms of face databases (e.g. that only law enforcement officials can have access to). Mutual observability need not even be assured, merely that there be the possiblity that the WearCam apparatus function. Such possibility serves as a reverse of the unseen guard [Foucault77].

## ACKNOWLEDGEMENTS

The author wishes to thank advisors Roz Picard and Simon Haykin for much encouragement and inspiration, as well as Hiroshi Ishii, Krzysztof Wodiczko, Julie Scher, Obed E. Torres, Jeffrey Levine, Brian Bradley, Thad Starner, Flavia Sparacino, Freedom Baird, Christine Southworth, Kris Popat, and Erik Trimble for many helpful suggestions and insights. The work was funded, in part, by the Council for the Arts at MIT; HP Labs, VirtualVision, Bran Ferren of Disney, ProComp, Compaq, Kopin, Colorlink, and Miyota are also to be thanked for equipment donations to this project.

## REFERENCES

[Campbell] Lee Campbell and Aaron Bobick. Correcting for radial lens distortion: A simple implementation. TR 322, M.I.T. Media Lab Perceptual Computing Section, Cambridge, Ma, Apr 1995.

[Dolezal82] Hubert Dolezal. Living in a world transformed. Academic press series in cognition and perception. Academic press, Chicago, Illinois, 1982.

[Foucault77] Michel Foucault. Discipline and Punish. 1977. Translated from "Surveiller et punir".

[Frankl94] VIKTOR E. FRANKL. MAN'S SEARCH FOR MEANING. WASHINGTON SQUARE PRESS, 1230 Avenue of the Americas, New York, N.Y. 10020 ISBN: 0-671-83465-7 1994.

[Hoshen95] Joseph Hoshen, Jim Sennott, and Max Winkler. Keeping tabs on criminals. IEEE SPECTRUM, pages 26--32, February 1995.

[Jeremijenko] Natalie Jeremijenko. Bureau of inverse technology, fivaonline. http://www.fivaonline.com or http://cdr.stanford.edu/

[Kohler64] Ivo Kohler. The formation and transformation of the perceptual world, volume 3 of Pshchological issues. International university press, 227 West 13 Street, 1964. monograph 12.

[Mann94] S. Mann "mediated reality". TR 260, M.I.T. Media Lab Perceptual Computing Section, Cambridge, Ma, 1994.

[Mann96] Steve Mann. Wearable computing: A first step toward personal imaging. IEEE Computer, Feb 1997.

[Mann96a] Steve Mann. Smart clothing: The shift to wearable computing. Communications of the ACM, pages 23--24, August 1996.

[Mann96b] S. Mann. "smart clothing". TR 366, MIT Media Lab Perceptual Computing Section, Cambridge, Ma, Feb. 2 1996

[Mann96c] Steve Mann. Wearable, tetherless computer--mediated reality: Wearcam as a wearable face--recognizer, and other applications for the disabled. TR 361, MIT Media Lab Perceptual Computing Section, Cambridge, Ma, Feb. 2 1996. Also appears in AAAI Fall Symposium on Developing Assistive Technology for People with Disabilities, 9-11 November 1996, MIT.

[Ryals] Cynthia Ryals. Lightspace: A new language of imaging. PHOTO Electronic Imaging, 38(2):14--16, 1995. http://www.novalink.com/pei/mann2.html.

[Stratton1896] George M. Stratton. Some preliminary experiements on vision. Psychological Review, 1896.

[Weiser] Mark Weiser and John Seely Brown. Designing calm technology, 1996. http://powergrid.electriciti.com/1.01/calmtech-wp.html.