Archive | Information RSS for this section

The Sonic Roots of Surveillance Society: Intimacy, Mobility, and Radio

Radio for Backup - 92/365 - 2 April 2013

Sound and Surveilance4

It’s an all too familiar movie trope. A bug hidden in a flower jar. A figure in shadows crouched listening at a door. The tape recording that no one knew existed, revealed at the most decisive of moments. Even the abrupt disconnection of a phone call manages to arouse the suspicion that we are never as alone as we may think. And although surveillance derives its meaning the latin “vigilare” (to watch) and French “sur-“ (over), its deep connotations of listening have all but obliterated that distinction.

Rounding out our series on surveillance, Kathleen Battles offers a historical perspective that shows how early twentieth century crime drama naturalized practices of citizen surveillance. A million eyes were activated as millions of listeners learned that the immediacy of radio and telephone allowed for an unprecedented level of participation in law enforcement. Calling all cars…Calling all cars… -AT

Police Headquarters, a 1932 radio crime drama, was produced in the infancy of narrative radio. Containing barely 12 minutes of narrative content, the program opened each episode with a repeating segment of call, connection, and dispatch to quickly establish both the crime committed and how the police responded to it. For example, in the “Payroll Robbery” episode it takes just over a minute and a half to hear a phone call to the titular headquarters, its connection to the proper unit, a radio call to a specific police car, and the responding officers arriving at the assigned location. Compared with the graphically and visually intense images of modern surveillance in contemporary popular culture, this brief exchange no doubt sounds quaint, simplistic, and even banal. After all, radios, cars, and telephones have served as the routine backdrop of most police dramas for some 70 years. But in 1932 the interlinking of these technologies was factually, as well as imaginatively, novel. This essay shows how radio, as “new media,” was central to imagining surveillance in sonic terms, prefiguring many features of contemporary surveillance practices.

"Kindergarden of the Air," by the Australian Broadcasting Corporation CC BY-NC-ND.

“Kindergarden of the Air,” by the Australian Broadcasting Corporation CC BY-NC-ND.

The introduction of radio and cars into police work took place in the first decades of the twentieth century, especially during the years between the two world wars that Richard Popp (2011) calls “the machine age”. He argues that this period witnessed vast transformations brought on by cars and radios, which, when combined with existing technologies like the telephone, forged new communication networks that transformed both work and leisure. These changes were central to the narratives of criminality and policing that emerged during the interwar years. Police were the focus of radio dramas, including Police Headquarters and Calling All Cars. These dramas played with the intermingling of automobility, telephony, and radio in ways that spoke to the main problems police forces saw themselves facing: organized, professional, mobile, machine-age criminals. Cars, telephones, and automobiles were not just tools to criminals, but they were also the building blocks for a machine age surveillance made possible by the sonic power of radio.

Recently, Robin James (2014) has suggested that the acousmatic is a useful metaphor for understanding the emerging practices of data based surveillance. Acousmatic surveillance listens for patterns in “ambient data environments” instead of profiling individuals in the panoptic sense. At the turn of the twentieth century, radio allowed for a panacoustic mastery of spaces that bridged both panoptic and acousmatic surveillance. Radio also speaks to another key feature of information age surveillance, what Mark Andrejevic (2011) describes as the “redoubling of tools for communication and leisure as technologies for surveillance and security.” (165-66) The technical capabilities and imaginative potentials of radio help us to consider it both as a police technology and entertainment medium. The sonic power of radio was often figured as an “Invisible Man Hunter,” whose realignment of spatial and temporal arrangements rendered criminal escape impossible. As an entertainment medium, radio’s aurality was key to understanding its imaginative potential as highly intimate and mobile: invasive and expansive.

In Police Headquarters we hear how radio’s sonic and aural qualities come together. Radio acts as the link between the telephone and car, allowing for a swift response to a citizen request. The tactical use of sound effects and narrative compression in the broadcast situate the listener inside a machine like apparatus that presents the police as always available. At their broadest level, radio crime dramas aurally situate communication and transportation technologies, like radio, as key to both the narrative organization of the story and as a plot element. In the opening to the “Stop That Car” episode of Calling All Cars, a dispatcher advises for cars to be on the lookout for a specific car involved in a hit and run, including the address of the crime and possible location of the vehicle. Overlaid with sound effects made to signify a car, these openings situate listeners as riders eavesdropping on the adventures of mobile police officers. As the program’s title suggests, each episode opened with a police radio call, often voiced by real life LAPD dispatcher, Jessie Rosenquist. The program’s sponsor was the gasoline company that supplied the fuel for LAPD patrol cars – further linking cars and radio as a key theme. In the opening to the “Two Man Crime Wave Episode,” the very ad for the product is performed as a police radio call.

The conceit of eavesdropping on a police adventure did much to link private life and the police. This theme runs tandem to radio’s sonic immediacy, which allowed listeners to imagine a seemingly instantaneous response to citizen phone calls.. For example, the “July 4th in a Radio Car” episode of Calling All Cars situates radio listeners as sonic participants, able to ride along with police from the comfort of their own living rooms. Here, cars respond to a number of calls made by private citizens that bring the policing function into daily life. There is even one call that involves domestic violence, in particular. Throughout the episode, police are situated as an available force – thanks to the telephone, radio, and automobile – to adjudicate all manner of private disputes. In these particular instances the intimacy of radio as a machine age technology is “redoubled” with radio as a police technology. Radio’s intimate address allowed the voices of police officers to enter the private space of the home. At the same time, the machinery of crime fighting required citizen participation, most often figured through the phone call from within the space of the home to the police.

"Radio Police Automation 1924," by Robert Wade, CC BY-NC-SA.

“Radio Police Automation 1924,” by Robert Wade, CC BY-NC-SA.

If the intimacy of radio served to cross the divide between public and private, the spatial-temporal collapse achieved by radio made it ideal for sonically monitoring great swaths of space. Intimately linked with cars, radio was understood as especially mobile. Radio’s ability to compress the relationship between space and time is frequently dramatized. For example, in the “Crime vs. Time” episode of Calling All Cars, the host explains how radio has rendered the average response time to a police call for help only two minutes and forty seconds. The episode then proceeds to show how radio was used by the police to track and apprehend two men who robbed a movie theater. Representing phone and radio calls, while specifically referencing geographic locations, radio dramatists used radio’s aural dimensions to render radio’s sonic power of surveillance. Capable of reaching everywhere, police radio, when linked with the telephone and automobile, could be used to sonically pinpoint any somewhere that a criminal might try to escape to.

In our era of high-tech and sophisticated technologies, visually rich narratives, and algorithmically driven methods of tracking, there is certainly something simple, comforting, and even nostalgic in these tales of low-tech machine age criminal apprehension. Depression era true crime dramas certainly do not offer the kind of sophistication as information age narratives, such as The Wire. The medium’s sonic qualities were key to linking together its use as both a police instrument and as a domestic entertainment technology. With sonic forces invisibly and silently crossing the into intimate domestic spaces and covering large swaths of territory, radio became key to imagining many features that we take for granted as “new” about the information age: the control of movement across space, the constant availability of communicative connection, the promise of perfect coordination across a field of institutional actors, the marshaling of citizen participation in surveillance efforts, and the construction of increasingly intimate links between domestic life and law enforcement procedures. In serving as central node in refiguring shifting notions of space and time that existing institutions were not prepared to handle, radio’s sonic qualities remapped the meaning of police work and helped to establish a relationship between the police and citizen body that still resonates today. While not as technologically sophisticated, machine age policing and police narratives took advantage of radio’s double function as both a machine of coordination and medium of entertainment to extend the policing function into more areas of life. This reflects a sonic mode of power that allows neither the interior space of the home nor the exterior world of the road relief from police presence. This moment of technological interconnection, however, evoked the excitement and anxiety that made sonic surveillance at once thrilling and calming; a salve to soothe the woes of a world that now seemed intensely close and impossibly far flung. Is it any wonder that while some fret over the power of corporate and state dataveillance today, that some continue to find comfort in the possibility of being recognized as someone in a world of ever more intense interconnection?

Featured image “Radio for Backup” by Jonathan Flinchbaugh CC BY-NC-SA.

Kathleen Battles is Associate Professor and Graduate Director in the Department of Communication and Journalism at Oakland University. Her research focuses on radio history, especially as it relates to issues of policing, sound and surveillance, questions concerning technology and culture, and sexuality and the media. She is the author of Calling All Cars: Radio Dragnets and the Technology of Policing (University of Minnesota Press, 2010); co-editor (with Joy Hayes and Wendy Hilton-Morrow) of War of the Worlds to Social Media: Mediated Communication in Times of Crisis (Peter Lang, 2013); and co-author (with Wendy Hilton-Morrow) of Sexual Identities and the Media: An Introduction (Routledge, 2015). In addition, her work has appeared in Critical Studies in Media Communication, The Radio Journal, and the Journal of Homosexuality.

tape reelREWIND!…If you liked this post, check out:

Acousmatic Surveillance and Big Data-Robin James

The Dark Side of Game Audio: The Sounds of Mimetic Control and Affective ConditioningAaron Trammell

À qui la rue?: On Mégaphone and Montreal’s Noisy Public Sphere-Lilian Radovac


Acousmatic Surveillance and Big Data

11928222826_d311dabe2a_o

Sound and Surveilance4

It’s an all too familiar movie trope. A bug hidden in a flower jar. A figure in shadows crouched listening at a door. The tape recording that no one knew existed, revealed at the most decisive of moments. Even the abrupt disconnection of a phone call manages to arouse the suspicion that we are never as alone as we may think. And although surveillance derives its meaning the latin “vigilare” (to watch) and French “sur-“ (over), its deep connotations of listening have all but obliterated that distinction.

Moving on from cybernetic games to modes of surveillance that work through composition and patterns. Here, Robin James challenges us to consider the unfamiliar resonances produced by our IP addresses, search histories, credit trails, and Facebook posts. How does the NSA transform our data footprints into the sweet, sweet, music of surveillance? Shhhhhhhh! Let’s listen in. . . -AT

Kate Crawford has argued that there’s a “big metaphor gap in how we describe algorithmic filtering.” Specifically, its “emergent qualities” are particularly difficult to capture. This process, algorithmic dataveillance, finds and tracks dynamic patterns of relationships amongst otherwise unrelated material. I think that acoustics can fill the metaphor gap Crawford identifies. Because of its focus on identifying emergent patterns within a structure of data, rather than its cause or source, algorithmic dataveillance isn’t panoptic, but acousmatic. Algorithmic dataveillance is acousmatic because it does not observe identifiable subjects, but ambient data environments, and it “listens” for harmonics to emerge as variously-combined data points fall into and out of phase/statistical correlation.

Dataveillance defines the form of surveillance that saturates our consumer information society. As this promotional Intel video explains, big data transcends the limits of human perception and cognition – it sees connections we cannot. And, as is the case with all superpowers, this is both a blessing and a curse. Although I appreciate emails from my local supermarket that remind me when my favorite bottle of wine is on sale, data profiling can have much more drastic and far-reaching effects. As Frank Pasquale has argued, big data can determine access to important resources like jobs and housing, often in ways that reinforce and deepen social inequities. Dataveillance is an increasingly prominent and powerful tool that determines many of our social relationships.

The term dataveillance was coined in 1988 by Roger Clarke, and refers to “the systematic use of personal data systems in the investigation or monitoring of the actions or communications of one or more persons.” In this context, the person is the object of surveillance and data is the medium through which that surveillance occurs. Writing 20 years later, Michael Zimmer identifies a phase-shift in dataveillance that coincides with the increased popularity and dominance of “user-generated and user-driven Web technologies” (2008). These technologies, found today in big social media, “represent a new and powerful ‘infrastructure of dataveillance,’ which brings about a new kind of panoptic gaze of both users’ online and even their offline activities” (Zimmer 2007). Metadataveillance and algorithmic filtering, however, are not variations on panopticism, but practices modeled—both historically/technologically and metaphorically—on acoustics.

In 2013, Edward Snowden’s infamous leaks revealed the nuts and bolts of the National Security Administration’s massive dataveillance program. They were collecting data records that, according to the Washington Post, included “e-mails, attachments, address books, calendars, files stored in the cloud, text or audio or video chats and ‘metadata’ that identify the locations, devices used and other information about a target.” The most enduringly controversial aspect of NSA dataveillance programs has been the bulk collection of Americans’ data and metadata—in other words, the “big data”-veillance programs.

 

Borrowed fro thierry ehrmann @Flickr CC BY.

Borrowed from thierry ehrmann @Flickr CC BY.

Instead of intercepting only the communications of known suspects, this big dataveillance collects everything from everyone and mines that data for patterns of suspicious behavior; patterns that are consistent with what algorithms have identified as, say, “terrorism.” As Cory Doctorow writes in BoingBoing, “Since the start of the Snowden story in 2013, the NSA has stressed that while it may intercept nearly every Internet user’s communications, it only ‘targets’ a small fraction of those, whose traffic patterns reveal some basis for suspicion.” “Suspicion,” here, is an emergent property of the dataset, a pattern or signal that becomes legible when you filter communication (meta)data through algorithms designed to hear that signal amidst all the noise.

Hearing a signal from amidst the noise, however, is not sufficient to consider surveillance acousmatic. “Panoptic” modes of listening and hearing, though epitomized by the universal and internalized gaze of the guards in the tower, might also be understood as the universal and internalized ear of the confessor. This is the ear that, for example, listens for conformity between bodily and vocal gender presentation. It is also the ear of audio scrobbling, which, as Calum Marsh has argued, is a confessional, panoptic music listening practice.

Therefore, when President Obama argued that “nobody is listening to your telephone calls,” he was correct. But only insofar as nobody (human or AI) is “listening” in the panoptic sense. The NSA does not listen for the “confessions” of already-identified subjects. For example, this court order to Verizon doesn’t demand recordings of the audio content of the calls, just the metadata. Again, the Washington Post explains:

The data doesn’t include the speech in a phone call or words in an email, but includes almost everything else, including the model of the phone and the “to” and “from” lines in emails. By tracing metadata, investigators can pinpoint a suspect’s location to specific floors of buildings. They can electronically map a person’s contacts, and their contacts’ contacts.

NSA dataveillance listens acousmatically because it hears the patterns of relationships that emerge from various combinations of data—e.g., which people talk and/or meet where and with what regularity. Instead of listening to identifiable subjects, the NSA identifies and tracks emergent properties that are statistically similar to already-identified patterns of “suspicious” behavior. Legally, the NSA is not required to identify a specific subject to surveil; instead they listen for patterns in the ambience. This type of observation is “acousmatic” in the sound studies sense because the sounds/patterns don’t come from one identifiable cause; they are the emergent properties of an aggregate.

Borrowed from david @Flickr CC BY-NC.

Borrowed from david @Flickr CC BY-NC.

Acousmatic listening is a particularly appropriate metaphor for NSA-style dataveillance because the emergent properties (or patterns) of metadata are comparable to harmonics or partials of sound, the resonant frequencies that emerge from a specific combination of primary tones and overtones. If data is like a sound’s primary tone, metadata is its overtones. When two or more tones sound simultaneously, harmonics emerge whhen overtones vibrate with and against one another. In Western music theory, something sounds dissonant and/or out of tune when the harmonics don’t vibrate synchronously or proportionally. Similarly, tones that are perfectly in tune sometimes create a consonant harmonic. The NSA is listening for harmonics. They seek metadata that statistically correlates to a pattern (such as “terrorism”), or is suspiciously out of correlation with a pattern (such as US “citizenship”). Instead of listening to identifiable sources of data, the NSA listens for correlations among data.

Both panopticism and acousmaticism are technologies that incite behavior and compel people to act in certain ways. However, they both use different methods, which, in turn, incite different behavioral outcomes. Panopticism maximizes efficiency and productivity by compelling conformity to a standard or norm. According to Michel Foucault, the outcome of panoptic surveillance is a society where everyone synchs to an “obligatory rhythm imposed from the outside” (151-2), such as the rhythmic divisions of the clock (150). In other words, panopticism transforms people into interchangeable cogs in an industrial machine.  Methodologically, panopticism demands self-monitoring. Foucault emphasizes that panopticism functions most efficiently when the gaze is internalized, when one “assumes responsibility for the constraints of power” and “makes them play…upon himself” (202). Panopticism requires individuals to synchronize themselves with established compulsory patterns.

Acousmaticism, on the other hand, aims for dynamic attunement between subjects and institutions, an attunement that is monitored and maintained by a third party (in this example, the algorithm). For example, Facebook’s News Feed algorithm facilitates the mutual adaptation of norms to subjects and subjects to norms. Facebook doesn’t care what you like; instead it seeks to transform your online behavior into a form of efficient digital labor. In order to do this, Facebook must adjust, in part, to you. Methodologically, this dynamic attunement is not a practice of internalization, but unlike Foucault’s panopticon, big dataveillance leverages outsourcing and distribution. There is so much data that no one individual—indeed, no one computer—can process it efficiently and intelligibly. The work of dataveillance is distributed across populations, networks, and institutions, and the surveilled “subject” emerges from that work (for example, Rob Horning’s concept of the “data self”). Acousmaticism tunes into the rhythmic patterns that synch up with and amplify its cycles of social, political, and economic reproduction.

Sonic Boom! Borrowed from NASA's Goddard Space Flight Center @Flickr CC BY.

Sonic Boom! Borrowed from NASA’s Goddard Space Flight Center @Flickr CC BY.

Unlike panopticism, which uses disciplinary techniques to eliminate noise, acousmaticism uses biopolitical techniques to allow profitable signals to emerge as clearly and frictionlessly as possible amid all the noise (for more on the relation between sound and biopolitics, see my previous SO! essay). Acousmaticism and panopticism are analytically discrete, yet applied in concert. For example, certain tiers of the North Carolina state employee’s health plan require so-called “obese” and tobacco-using members to commit to weight-loss and smoking-cessation programs. If these members are to remain eligible for their selected level of coverage, they must track and report their program-related activities (such as exercise). People who exhibit patterns of behavior that are statistically risky and unprofitable for the insurance company are subject to extra layers of surveillance and discipline. Here, acousmatic techniques regulate the distribution and intensity of panoptic surveillance. To use Nathan Jurgenson’s turn of phrase, acousmaticism determines “for whom” the panoptic gaze matters. To be clear, acousmaticism does not replace panopticism; my claim is more modest. Acousmaticism is an accurate and productive metaphor for theorizing both the aims and methods of big dataveillance, which is, itself, one instrument in today’s broader surveillance ensemble.

-

Featured image “Big Brother 13/365″ by Dennis Skley CC BY-ND.

-

Robin James is Associate Professor of Philosophy at UNC Charlotte. She is author of two books: Resilience & Melancholy: pop music, feminism, and neoliberalism will be published by Zer0 books this fall, and The Conjectural Body: gender, race and the philosophy of music was published by Lexington Books in 2010. Her work on feminism, race, contemporary continental philosophy, pop music, and sound studies has appeared in The New Inquiry, Hypatia, differences, Contemporary Aesthetics, and the Journal of Popular Music Studies. She is also a digital sound artist and musician. She blogs at its-her-factory.com and is a regular contributor to Cyborgology.

tape reelREWIND!…If you liked this post, check out:

“Cremation of the senses in friendly fire”: on sound and biopolitics (via KMFDM & World War Z)–Robin James

The Dark Side of Game Audio: The Sounds of Mimetic Control and Affective ConditioningAaron Trammell

Listening to Whisperers: Performance, ASMR Community, and Fetish on YouTube–Joshua Hudelson

Sounds of Science: The Mystique of Sonification

NYU_full

Hearing the Unheard IIWelcome to the final installment of Hearing the UnHeardSounding Out!s series on what we don’t hear and how this unheard world affects us. The series started out with my post on hearing, large and small, continued with a piece by China Blue on the sounds of catastrophic impacts, and Milton Garcés piece on the infrasonic world of volcanoes. To cap it all off, we introduce The Sounds of Science by professor, cellist and interactive media expert, Margaret Schedel.

Dr. Schedel is an Associate Professor of Composition and Computer Music at Stony Brook University. Through her work, she explores the relatively new field of Data Sonification, generating new ways to perceive and interact with information through the use of sound. While everyone is familiar with informatics, graphs and images used to convey complex information, her work explores how we can expand our understanding of even complex scientific information by using our fastest and most emotionally compelling sense, hearing.

– Guest Editor Seth Horowitz

With the invention of digital sound, the number of scientific experiments using sound has skyrocketed in the 21st century, and as Sounding Out! readers know, sonification has started to enter the public consciousness as a new and refreshing alternative modality for exploring and understanding many kinds of datasets emerging from research into everything from deep space to the underground. We seem to be in a moment in which “science that sounds” has a special magic, a mystique that relies to some extent on misunderstandings in popular awareness about the processes and potentials of that alternative modality.

For one thing, using sound to understand scientific phenomena is not actually new. Diarist Samuel Pepys wrote about meeting scientist Robert Hooke in 1666 that “he is able to tell how many strokes a fly makes with her wings (those flies that hum in their flying) by the note that it answers to in musique during their flying.” Unfortunately Hooke never published his findings, leading researchers to speculate on his methods. One popular theory is that he tied strings of varying lengths between a fly and an ear trumpet, recognizing that sympathetic resonance would cause the correct length string to vibrate, thus allowing him to calculate the frequency. Even Galileo used sound, showing the constant acceleration of a ball due to gravity by using an inclined plane with thin moveable frets. By moving the placement of the frets until the clicks created an even tempo he was able to come up with a mathematical equation to describe how time and distance relate when an object falls.

Illustration from Robert Hooke's Micrographia (1665)

Illustration from Robert Hooke’s Micrographia (1665)

There have also been other scientific advances using sound in the more recent past. The stethoscope was invented in 1816 for auscultation, listening to the sounds of the body. It was later applied to machines—listening for the operation of the technological gear. Underwater sonar was patented in 1913 and is still used to navigate and communicate using hydroacoustic phenomenon. The Geiger Counter was developed in 1928 using principles discovered in 1908; it is unclear exactly when the distinctive sound was added. These are all examples of auditory display [AD]; sonification-generating or manipulating sound by using data is a subset of AD. As the forward to the The Sonification Handbook states, “[Since 1992] Technologies that support AD have matured. AD has been integrated into significant (read “funded” and “respectable”) research initiatives. Some forward thinking universities and research centers have established ongoing AD programs. And the great need to involve the entire human perceptual system in understanding complex data, monitoring processes, and providing effective interfaces has persisted and increased” (Thomas Hermann, Andy Hunt, John G. Neuhoff, Sonification Handbook, iii)

Sonification clearly enables scientists, musicians and the public to interact with data in a very different way, particularly compared to the more numerous techniques involving vision. Indeed, because hearing functions quite differently than vision, sonification offers an alternative kind of understanding of data (sometimes more accurate), which would not be possible using eyes alone. Hearing is multi-directional—our ears don’t have to be pointing at a sound source in order to sense it. Furthermore, the frequency response of our hearing is thousands of times more accurate than our vision. In order to reproduce a moving image the sampling rate (called frame-rate) for film is 24 frames per second, while audio has to be sampled at 44,100 frames per second in order to accurately reproduce sound. In addition, aural perception works on simultaneous time scales—we can take in multiple streams of audio data at once at many different dynamics, while our pupils dilate and contract, limiting how much visual data we can absorb at a single time. Our ears are also amazing at detecting regular patterns over time in data; we hear these patterns as frequency, harmonic relationships, and timbre.

Image credit: Dr. Kevin Yager, data measured at X9 beamline, Brookhaven National Lab.

Image credit: Dr. Kevin Yager, Brookhaven National Lab.

But hearing isn’t simple, either. In the current fascination with sonification, the fact that aesthetic decisions must be made in order to translate data into the auditory domain can be obscured. Headlines such as “Here’s What the Higgs Boson Sounds Like” are much sexier than headlines such as “Here is What One Possible Mapping of Some of the Data We Have Collected from a Scientific Measuring Instrument (which itself has inaccuracies) Into Sound.” To illustrate the complexity of these aesthetic decisions, which are always interior to the sonification process, I focus here on how my collaborators and I have been using sound to understand many kinds of scientific data.

My husband, Kevin Yager, a staff scientist at Brookhaven National Laboratory, works at the Center for Functional Nanomaterials using scattering data from x-rays to probe the structure of matter. One night I asked him how exactly the science of x-ray scattering works. He explained that X-rays “scatter” off of all the atoms/particles in the sample and the intensity is measured by a detector. He can then calculate the structure of the material, using the Fast Fourier Transform (FFT) algorithm. He started to explain FFT to me, but I interrupted him because I use FFT all the time in computer music. The same algorithm he uses to determine the structure of matter, musicians use to separate frequency content from time. When I was researching this post, I found a site for computer music which actually discusses x-ray scattering as a precursor for FFT used in sonic applications.

To date, most sonifications have used data which changes over time – a fly’s wings flapping, a heartbeat, a radiation signature. Except in special cases Kevin’s data does not exist in time – it is a single snapshot. But because data from x-ray scattering is a Fourier Transform of the real-space density distribution, we could use additive synthesis, using multiple simultaneous sine waves, to represent different spatial modes. Using this method, we swept through his data radially, like a clock hand, making timbre-based sonifications from the data by synthesizing sine waves using with the loudness based on the intensity of the scattering data and frequency based on the position.

We played a lot with the settings of the additive synthesis, including the length of the sound, the highest frequency and even the number of frequency bins (going back to the clock metaphor – pretend the clock hand is a ruler – the number of frequency bins would be the number of demarcations on the ruler) arriving eventually at set of optimized variables.

Here is one version of the track we created using 10 frequency bins:

.

Here is one we created using 2000:

.

And here is one we created using 50 frequency bins, which we settled on:

.

On a software synthesizer this would be like the default setting. In the future we hope to have an interactive graphic user interface where sliders control these variables, just like a musician tweaks the sound of a synth, so scientists can bring out, or mask aspects of the data.

To hear what that would be like, here are a few tracks that vary length:

.

.

.

Finally, here is a track we created using different mappings of frequency and intensity:

.

Having these sliders would reinforce to the scientists that we are not creating “the sound of a metallic alloy,” we are creating one sonic representation of the data from the metallic alloy.

It is interesting that such a representation can be vital to scientists. At first, my husband went along with this sonification project as more of a thought experiment rather than something that he thought would actually be useful in the lab, until he heard something distinct about one of those sounds, suggesting that there was a misaligned sample. Once Kevin heard that glitched sound (you can hear it in the video above), he was convinced that sonification was a useful tool for his lab. He and his colleagues are dealing with measurements 1/25,000th the width of a human hair, aiming an X-ray through twenty pieces of equipment to get the beam focused just right. If any piece of equipment is out of kilter, the data can’t be collected. This is where our ears’ non-directionality is useful. The scientist can be working on his/her computer and, using ambient sound, know when a sample is misaligned.

procedure

It remains to be seen/heard if the sonifications will be useful to actually understand the material structures. We are currently running an experiment using Mechanical Turk to determine this kind of multi-modal display (using vision and audio) is actually helpful. Basically we are training people on just the images of the scattering data, and testing how well they do, and training another group of people on the images plus the sonification and testing how well they do.

I’m also working with collaborators at Stony Brook University on sonification of data. In one experiment we are using ambisonic (3-dimensional) sound to create a sonic map of the brain to understand drug addiction. Standing in the middle of the ambisonic cube, we hope to find relationships between voxels, a cube of brain tissue—analogous to pixels. When neurons fire in areas of the brain simultaneously there is most likely a causal relationship which can help scientists decode the brain activity of addiction. Computer vision researchers have been searching for these relationships unsuccessfully; we hope that our sonification will allow us to hear associations in distinct parts of the brain which are not easily recognized with sight. We are hoping to leverage the temporal pattern recognition of our auditory system, but we have been running into problems doing the sonification; each slice of data from the FMRI has about 300,000 data points. We have it working with 3,000 data points, but either our programming needs to get more efficient, or we have to get a much more powerful computer in order to work with all of the data.

On another project we are hoping to sonify gait data using smartphones. I’m working with some of my music students and a professor of Physical Therapy, Lisa Muratori, who works on understanding the underlying mechanisms of mobility problems in Parkinsons’ Disease (PD). The physical therapy lab has a digital motion-capture system and a split-belt treadmill for asymmetric stepping—the patients are supported by a harness so they don’t fall. PD is a progressive nervous system disorder characterized by slow movement, rigidity, tremor, and postural instability. Because of degeneration of specific areas of the brain, individuals with PD have difficulty using internally driven cues to initiate and drive movement. However, many studies have demonstrated an almost normal movement pattern when persons with PD are provided external cues, including significant improvements in gait with rhythmic auditory cueing. So far the research with PD and sound has be unidirectional – the patients listen to sound and try to match their gait to the external rhythms from the auditory cues.In our system we will use bio-feedback to sonify data from sensors the patients will wear and feed error messages back to the patient through music. Eventually we hope that patients will be able to adjust their gait by listening to self-generated musical distortions on a smartphone.

As sonification becomes more prevalent, it is important to understand that aesthetic decisions are inevitable and even essential in every kind of data representation. We are so accustomed to looking at visual representations of information—from maps to pie charts—that we may forget that these are also arbitrary transcodings. Even a photograph is not an unambiguous record of reality; the mechanics of the camera and artistic choices of the photographer control the representation. So too, in sonification, do we have considerable latitude. Rather than view these ambiguities as a nuisance, we should embrace them as a freedom that allows us to highlight salient features, or uncover previously invisible patterns.

__

Margaret Anne Schedel is a composer and cellist specializing in the creation and performance of ferociously interactive media. She holds a certificate in Deep Listening with Pauline Oliveros and has studied composition with Mara Helmuth, Cort Lippe and McGregor Boyle. She sits on the boards of 60×60 Dance, the BEAM Foundation, Devotion Gallery, the International Computer Music Association, and Organised Sound. She contributed a chapter to the Cambridge Companion to Electronic Music, and is a joint author of Electronic Music published by Cambridge University Press. She recently edited an issue of Organised Sound on sonification. Her research focuses on gesture in music, and the sustainability of technology in art. She ran SUNY’s first Coursera Massive Open Online Course (MOOC) in 2013. As an Associate Professor of Music at Stony Brook University, she serves as Co-Director of Computer Music and is a core faculty member of cDACT, the consortium for digital art, culture and technology.

Featured Image: Dr. Kevin Yager, data measured at X9 beamline, Brookhaven National Lab.

Research carried out at the Center for Functional Nanomaterials, Brookhaven National Laboratory, is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, under Contract No. DE-AC02-98CH10886.

tape reelREWIND! ….. If you liked this post, you might also like:

The Noises of Finance–Nicholas Knouf

Revising the Future of Music Technology–Aaron Trammell

A Brief History of Auto-Tune–Owen Marshall

That Infernal Racket: Sound, Anxiety, and the IBM Computer in AMC’s Mad Men

IBM_System360_Mainframe

[Warning: Spoilers Ahead for Folks Not Caught Up with Season 7, Episode 5!]

In one of the more memorable – and squirm-inducing – scenes of this season of AMC’s Mad Men, brilliant but eccentric copywriter Michael Ginsberg (Ben Feldman) presents his colleague, agency copy chief Peggy Olsen (Elisabeth Moss) with his own severed nipple, placed carefully in a gift box. Ginsberg explains to the understandably horrified Peggy that the gift is both a token of his affection and a means of relieving pressure caused by the arrival of Sterling, Cooper & Partners’ (SC&P) newest acquisition: a humming, room-sized IBM System/360 mainframe computer. Explaining his enmity for the machine and his increasingly erratic behavior, Ginsberg tells Peggy that the “waves of data” emanating from the computer were filling him up, and that the only solution was to “remove the pressure” by slicing off his “valve.”

The arrival of the IBM 360 in the idealized 1960s office space inhabited by Mad Men is obviously an unsettling presence – and not only for Ginsberg. Since its debut in Episode 4, commentators (e.g. WaPo’s Andrea Peterson, Slate’s Seth Stevenson) have meditated on the heavy-handed symbolism surrounding the machine – both in terms of its historical significance and its implications for plot and character development. Typically cued through noise (or lack thereof), it is worth reflecting upon the role of sound in establishing the computer as a source of disruption. Between the pounding and screeching of installation and the drone of the completed machine’s air conditioner and tape reels, the sonic motifs accompanying the computer underline tensions between (and roiling within) SC&P staffers grappling with the incipient digital age. Likewise, the infernal racket produced by the installation and operation of the IBM 360 adds an important dimension to the tensions resulting from its presence, which can be read as allegories for the complexities and contradictions of our relationship with technology.

.

The tone of the conflict is set even before we meet the IBM 360 toward the end of Episode 4: The Monolith – a reference to Kubrick’s 1968 classic 2001: A Space Odyssey (Slate’s Forrest Wickman ably discusses the references). Like the unnerving silence used with such great effect in that film, the absence of sound frames our first encounter with the computer – or at least its promise. Early in the episode, Don Draper (Jon Hamm), newly rehabilitated from his forced exile from the agency, arrives one morning at SC&P to find the office deserted. The ghostly sequence is clearly meant to symbolize Draper’s detachment from the firm. But as the episode progresses and tensions mount over the possibility that the IBM 360 will render jobs obsolete, the desolate office suggests a more ominous meaning – a once lively space muted by cold, impersonal automation.

In following scenes, successive stages of mainframe installation are marked by convergences of conflict and cacophony. First, there is the din of the creative team as they evacuate their beloved lounge – now earmarked as computer space – and during which a distraught Ginsberg projects his indignation onto art director Stan Rizzo, who appears more accepting. “They’re trying to erase us!” Ginsberg exclaims bitterly. Later, Draper lounges on his office couch as a clop clopping of hammers outside signifies tangible change. As if this weren’t enough of a distraction, two men in the corridor begin to chat loudly over the noise. Going out to investigate, Draper strikes up a conversation with one of the men, Lloyd Hawley, installation supervisor and founder of a small technology company competing with IBM. “Who’s winning?” Draper asks innocently, “who’s replacing more people?” Clearly irritated by Draper’s tone, Harry Crane – SC&P media director and the computer’s lead cheerleader – offers Draper a condescending apology for the loss of his “lunchroom,” assures him the change was “not symbolic.” “No, it’s quite literal,” Draper retorts. Unabated, the pounding and screeching of construction work emphasizes his point.

For the remainder of the episode, the raucous noise of construction acts as a leitmotif underscoring tensions between characters – between Peggy and Lou Avery (Draper’s priggish replacement at creative director), and between Draper and the interloper Lloyd. Finally, the end of construction is punctuated by a return to silence, as Peggy arrives one morning to see workers glide mainframe components noiselessly into the office.

Mad Men Logo. Used under the auspices of fair use for identification and critical commentary.

Mad Men Logo. Used under the auspices of fair use for identification and critical commentary.

With this emphasis on technology as a source of symbolic, physical, and sonic disruption, Matthew Weiner and the creators of Mad Men draw upon a rich literary tradition. A relevant example contemporaneous with the show’s “present,” is literary critic Leo Marx’s 1964 text The Machine in the Garden, which examines the complicated relationships between a “pastoral ideal” and technological progress within American literature and popular imagination. Marx’s analysis reveals that sound is often used to convey the disruptive presence of technology within the bucolic landscape of the American continent. In Hawthorne’s Sleepy Hollow for example, it is the interrupting shriek of a locomotive whistle that breaks the author’s harmonious reverie: “Now tension replaces repose: the noise arouses a sense of dislocation, conflict, and anxiety” (15). In the decidedly un-pastoral modern office space, the noise of the computer installation nevertheless signifies a momentous social change and irrevocable loss. Picking out these tensions has always been one of the show’s strengths – whether it is the computer, Draper’s double identity, or the quiet endurance of women to the misogyny of midcentury work and domestic life.

Change, however, has significant consequences for Ginsberg, the young copywriter and Holocaust survivor who, as CBS’s Jessica Firger observes, has been deteriorating psychologically for some time. The proximity of the IBM 360, and the incessant drone of its mind-controlling waves eventually puts him over the edge. As Draper and Peggy enter the office early in Episode 5, Ginsberg glowers into the room housing the IBM 360. “Stop humming, you’re not happy!” he explodes. As Peggy attempts to soothe her colleague, our perspective shifts to look out at them from inside the glass-encased computer room. From here, the mainframe’s ambient noise muffles Peggy’s words, suggesting isolation between human and non-human. This play of speech and silence reoccurs later in the episode as Ginsberg, working alone on a Saturday with tissues wedged in his ears, spies Lou Avery and SC&P partner Jim Cutler inside the computer room, their voices made inaudible by the droning computer in a delicious homage to 2001 (see Vulture’s amusing gif). But the noise is clearly affecting Ginsberg. “It’s that hum at the office! It’s getting to me!” he tells Peggy later that evening. He even claims the computer has affected his sexuality.

Ginsberg’s noise complaints would have resonated in 1969 New York. In November of that year, the New York Times ran a feature on the city’s nerve-shattering noise pollution, calling it a “slow agent of death.” In addition to the myriad construction projects, subways, car horns, jet planes, and standing machinery populating the city soundscape, office workers found scant respite indoors where phones, air conditioners, “computers and typewriters and tabulators” whirred, whined, and clacked throughout the day. The article went on to report that scientists studying the impact of prolonged noise exposure on the human body had concluded a variety of ill effects on the heart and nervous system. Though no connection was made between computers and sexuality (as Ginsberg claimed), the article reported that laboratory rats under prolonged noise exposure had indeed “turned homosexual,” an opinion that underlined deterministic associations between sexuality, psychological disorder, and external stimuli.

An advertisement for the IBM 360. Borrowed from Wikimedia Commons.

An advertisement for the IBM 360. Borrowed from Wikimedia Commons.

As SO! editor Jennifer Stoever-Ackerman has argued, noise in midcentury New York also signified a sonic-racial politics, in which the mainstream “listening ear” recoiled at the “noise” created by Black and Puerto Rican others. In terms of Mad Men’s computer however, it is technology, economic anxiety, and mental illness, rather than ethnicity that frames sonic disruption. The basis of these tensions are similar however, and various interactions with SC&P’s IBM 360 demonstrate, as Stoever-Ackerman writes in SO!, “the ways in which Americans have been disciplined to consider some sounds as natural, normal, and desirable, while deeming alternate ways of listening and sounding as aberrant [and] dangerous.” Though similar, the conflict with technology on Mad Men does not suggest a clear us/them, or us/”it” binary. The banging of construction may be at first antagonistic, but it’s finite – eventually the computer is normalized within the SC&P office space to the extent that Peggy chides Ginsberg’s exasperation in Episode 5 by insisting “it’s just a computer!” Ginsberg’s reaction is more complex however, implicating a contradictory relationship with technology: once fully installed, has the droning computer become “natural, normal, and desirable” despite previous ambivalence? Is the keen awareness and anxiety towards technology symbolized through Ginsberg (albeit in a extreme form) suggested as the “aberrant” listening practice, or could it be Peggy’s apparent acceptance?

Like most cultural texts set in the past, it is possible to read Mad Men allegorically, as suggesting a certain ordering of meaning and values. From the perspective of those who have long since domesticated computers, the controversies and tropes activated by SC&P’s IBM 360 might strike us as familiar, even quaint. As the sociologist Bruno Latour has argued however, we would be wise to consider how technology exerts a kind of social agency that structures and impacts our daily lives. As historical symbolism, the sounds and noises of the IBM 360 on Mad Men should remind us that technological progress is not teleological, but a struggle over meaning in which anxieties (about jobs, mind-control, surveillance, subjectivity, etc.) may be variously accommodated, suppressed, or dismissed as irrational.

Featured image: An IBM 360 Mainframe. Borrowed from Wikimedia Commons CC 2.0

Andrew J. Salvati is a Media Studies Ph.D. candidate at Rutgers University. His interests include the history of television and media technologies, theory and philosophy of history, and representations of history in media contexts. Additional interests include play, authenticity, the sublime, and the absurd. Andrew has co-authored a book chapter with colleague Jonathan Bullinger titled “Selective Authenticity and the Playable Past” in the recent edited volume Playing With the Past (2013)and has written a recent blog post for Play the Past titled The Play of History.”

tape reelREWIND!…If you liked this post, you may also dig:

“DIY Histories: Podcasting the Past” -Andrew J. Salvati

“The Noise of SB 1070: Or Do I Sound Illegal to You?”- Jennifer Stoever-Ackerman

“DIANE… The Personal Voice Recorder in Twin Peaks” -Tom McEnaney

%d bloggers like this: