Archive | May 2014

Sounding Out! Podcast #29: Game Audio Notes I: Growing Sounds for Sim Cell

Sound and Pleasure2A pair of firsts! This is both the lead post in our summer Sound and Pleasure series, and the first podcast in a three part series by Leonard J. Paul. What is the connection between sound and enjoyment, and how are pleasing sounds designed? Pleasure is, after all, what brings y’all back to Sounding Out! weekly, is it not?

In today’s installment Leonard peels back the curtain of game audio design and reveals his creative process. For anyone curious as to what creative decisions lead to the bloops, bleeps, and ambient soundscapes of video games, this is essential listening. Stay tuned for next Monday’s installment on the process of designing sound for Retro City Rampage, and next month’s episode which focuses on the game Vessel. Today, Leonard begins by picking apart his design process at a cellular level. Literally! -AT, Multimedia Editor

CLICK HERE TO DOWNLOADGame Audio Notes I: Growing Sounds for Sim Cell

SUBSCRIBE TO THE SERIES VIA ITUNES

ADD OUR PODCASTS TO YOUR STITCHER FAVORITES PLAYLIST

Game Audio Notes I: Growing Sounds for Sim Cell

Sim Cell is an educational game released in Spring 2014 by Strange Loop Games. Published by Amplify, a branch of News Corp, it teaches students how the human cell works. Players take control of a small vessel that is shrunk down to the size of a cell and solve the tasks set by the game while also learning about the human cell. This essay unpacks the design decisions behind the simulation of a variety of natural phenomena (motion, impact, and voice) in Sim Cell.

For the design of this game I decided to focus on the elements of life itself and attempted to “grow” the music and sound design from synthetic sounds. I used the visual scripting language Pure Data (PD) to program both the sounds and the music. The music is generated from a set of rules and patterns that cause each playback of a song to be slightly different each time. Each of the sound effects are crafted from small programs that are based on the design of analogue modular synthesizers. Basic synthetic audio elements such as filtered noise, sawtooth waves and sine waves were all used in the game’s sound design.

An image of the game's space-like world.

A screenshot of Sim Cell’s microscopic space-like world. Used with permission (c) 2014 Amplify.

The visuals of the game give a feeling of being in an “inner space” that mirrors outer space. I took my inspiration for Sim Cell‘s sound effects from Louis and Bebe Baron’s score to Forbidden Planet. I used simple patches in PD to assemble all of the sounds for the game from synthesis. There aren’t any recorded samples in the sound design – all of the sound effects are generated from mathematics.

The digital effects used in the sound design were built to emulate the effects available in vintage studios such as plate reverb and analog delay. I found a simulation of a plate reverb that from a modified open source patch from the RjDj project, and constructed an analogue delay by using a low-pass filter on a delayed signal.

https://www.youtube.com/watch?v=3m86ftny1uY&feature=youtu.be

In keeping with the vintage theme, I used elements of sound design from the early days of video games as well. Early arcade games such as Space Invaders used custom audio synthesis microchips for each of their sounds. In order to emulate this, I gave every sound its own patch when doing the synthesis for Sim Cell. I learned to appreciate this ethic of design when playing Combat on the Atari 2600 while growing up. The Atari 2600 could only output two sounds at once thus had a very limited palette of tones. All of the source code for the sounds and music for Sim Cell are less than 1 megabyte, which shows how powerful the efficient coding of mathematics can be.

Vector image on the left, raster graphics on the right.

Vector image on the left, raster graphics on the right. Photo used with permission by the author.

Another cool thing about generating the sounds from mathematics is that users can “zoom in” on sounds in the same way that one can zoom in to a vector drawing. In vector drawings the lines are smooth when you zoom in, as opposed to rasterized pictures (such as a JPEG) which reveal a blurry set of pixels upon zooming. When code can change sounds in real-time, it makes them come alive and lends a sense of flexibility to the composition.

For a human feeling I often filter the sounds using formant frequencies which simulate the resonant qualities of vowel sounds, thus offering a vocal quality to the sample. For alarm sounds in Sim Cell I used minor second intervals. This lent a sense of dissonance which informed players that they needed to adjust their gameplay in order to navigate treacherous areas. Motion was captured through a whoosh sound that used filtered and modulated noise. Some sounds were pitched up to make things seem like they were travelling towards the player and likewise they exploited a sense of doppler shift when pitched down, giving the feel that a sound was traveling away from the player. Together, these techniques produced a sense of immersion for the player while simultaneously building a realistic soundscape.

Our decision to use libPD for this synthesis turned problematic and its processing requirements were too high. In order to remedy this, we decided to convert our audio into samples. Consider a photograph of a sculpture. Our sounds, like sculpture in the photograph, could now only be viewed from one direction. This meant that the music now only had a single version and that sound effects would repeat as well. A small fix was exporting the file with a set of five different intensities from 0 to 1.0. Like taking a photograph from several angles, this meant that the game could play a sample at intensity levels of 20%, 40%, 60%, 80% and 100%. Although a truly random sense of variation was lost, this method still conveyed the intensity of impacts (and other similar events) generated by the physics engine of the game.

An image of the inexpensive open-source computer: The Rasberry PI

An image of the inexpensive open-source computer: The Rasberry PI. Photo used with permission by the author.

PD is a great way to learn and play with digital audio since you can change the patch while it is running, just like you might do with a real analogue synthesizer. There’s plenty of other neat stuff that PD can do, like being able to be run on the Raspberry Pi, so you could code your own effects pedals and make your own synths using PD for around $50 or so. For video games, you can use libPD to integrate PD patches into your Android or iOS apps as well. I hope this essay has offered some insight as to my process when using PD. I’ve included some links below for those interested in learning more.

Additional resources:

Leonard J. Paul attained his Honours degree in Computer Science at Simon Fraser University in BC, Canada with an Extended Minor in Music concentrating in Electroacoustics. He began his work in video games on the Sega Genesis and Super Nintendo Entertainment System and has a twenty year history in composing, sound design and coding for games. He has worked on over twenty major game titles totalling over 6.4 million units sold since 1994, including award-winning AAA titles such as EA’s NBA Jam 2010NHL11Need for Speed: Hot Pursuit 2NBA Live ’95 as well as the indie award-winning title Retro City Rampage.

He is the co-founder of the School of Video Game Audio and has taught game audio students from over thirty different countries online since 2012. His new media works has been exhibited in cities including Surrey, Banff, Victoria, São Paulo, Zürich and San Jose. As a documentary film composer, he had the good fortune of scoring the original music for multi-awarding winning documentary The Corporation which remains the highest-grossing Canadian documentary in history to date. He has performed live electronic music in cities such as Osaka, Berlin, San Francisco, Brooklyn and Amsterdam under the name Freaky DNA.

He is an internationally renowned speaker on the topic of video game audio and has been invited to speak in Vancouver, Lyon, Berlin, Bogotá, London, Banff, San Francisco, San Jose, Porto, Angoulême and other locations around the world.

His writings and presentations are available at http://VideoGameAudio.com

Featured image: Concept art for Sim Cell. Used with permission (c) 2014 Amplify.

tape reelREWIND! . . .If you liked this post, you may also dig:

Sounding Out! Podcast #10: Interview with Theremin Master Eric Ross- Aaron Trammell

Papa Sangre and the Construction of Immersion in Audio Games- Enongo Lumumba-Kasongo

Playing with Bits, Pieces, and Lightning Bolts: An Interview with Sound Artist Andrea Parkins

That Infernal Racket: Sound, Anxiety, and the IBM Computer in AMC’s Mad Men

[Warning: Spoilers Ahead for Folks Not Caught Up with Season 7, Episode 5!]

In one of the more memorable – and squirm-inducing – scenes of this season of AMC’s Mad Men, brilliant but eccentric copywriter Michael Ginsberg (Ben Feldman) presents his colleague, agency copy chief Peggy Olsen (Elisabeth Moss) with his own severed nipple, placed carefully in a gift box. Ginsberg explains to the understandably horrified Peggy that the gift is both a token of his affection and a means of relieving pressure caused by the arrival of Sterling, Cooper & Partners’ (SC&P) newest acquisition: a humming, room-sized IBM System/360 mainframe computer. Explaining his enmity for the machine and his increasingly erratic behavior, Ginsberg tells Peggy that the “waves of data” emanating from the computer were filling him up, and that the only solution was to “remove the pressure” by slicing off his “valve.”

The arrival of the IBM 360 in the idealized 1960s office space inhabited by Mad Men is obviously an unsettling presence – and not only for Ginsberg. Since its debut in Episode 4, commentators (e.g. WaPo’s Andrea Peterson, Slate’s Seth Stevenson) have meditated on the heavy-handed symbolism surrounding the machine – both in terms of its historical significance and its implications for plot and character development. Typically cued through noise (or lack thereof), it is worth reflecting upon the role of sound in establishing the computer as a source of disruption. Between the pounding and screeching of installation and the drone of the completed machine’s air conditioner and tape reels, the sonic motifs accompanying the computer underline tensions between (and roiling within) SC&P staffers grappling with the incipient digital age. Likewise, the infernal racket produced by the installation and operation of the IBM 360 adds an important dimension to the tensions resulting from its presence, which can be read as allegories for the complexities and contradictions of our relationship with technology.

.

The tone of the conflict is set even before we meet the IBM 360 toward the end of Episode 4: The Monolith – a reference to Kubrick’s 1968 classic 2001: A Space Odyssey (Slate’s Forrest Wickman ably discusses the references). Like the unnerving silence used with such great effect in that film, the absence of sound frames our first encounter with the computer – or at least its promise. Early in the episode, Don Draper (Jon Hamm), newly rehabilitated from his forced exile from the agency, arrives one morning at SC&P to find the office deserted. The ghostly sequence is clearly meant to symbolize Draper’s detachment from the firm. But as the episode progresses and tensions mount over the possibility that the IBM 360 will render jobs obsolete, the desolate office suggests a more ominous meaning – a once lively space muted by cold, impersonal automation.

In following scenes, successive stages of mainframe installation are marked by convergences of conflict and cacophony. First, there is the din of the creative team as they evacuate their beloved lounge – now earmarked as computer space – and during which a distraught Ginsberg projects his indignation onto art director Stan Rizzo, who appears more accepting. “They’re trying to erase us!” Ginsberg exclaims bitterly. Later, Draper lounges on his office couch as a clop clopping of hammers outside signifies tangible change. As if this weren’t enough of a distraction, two men in the corridor begin to chat loudly over the noise. Going out to investigate, Draper strikes up a conversation with one of the men, Lloyd Hawley, installation supervisor and founder of a small technology company competing with IBM. “Who’s winning?” Draper asks innocently, “who’s replacing more people?” Clearly irritated by Draper’s tone, Harry Crane – SC&P media director and the computer’s lead cheerleader – offers Draper a condescending apology for the loss of his “lunchroom,” assures him the change was “not symbolic.” “No, it’s quite literal,” Draper retorts. Unabated, the pounding and screeching of construction work emphasizes his point.

For the remainder of the episode, the raucous noise of construction acts as a leitmotif underscoring tensions between characters – between Peggy and Lou Avery (Draper’s priggish replacement at creative director), and between Draper and the interloper Lloyd. Finally, the end of construction is punctuated by a return to silence, as Peggy arrives one morning to see workers glide mainframe components noiselessly into the office.

Mad Men Logo. Used under the auspices of fair use for identification and critical commentary.

Mad Men Logo. Used under the auspices of fair use for identification and critical commentary.

With this emphasis on technology as a source of symbolic, physical, and sonic disruption, Matthew Weiner and the creators of Mad Men draw upon a rich literary tradition. A relevant example contemporaneous with the show’s “present,” is literary critic Leo Marx’s 1964 text The Machine in the Garden, which examines the complicated relationships between a “pastoral ideal” and technological progress within American literature and popular imagination. Marx’s analysis reveals that sound is often used to convey the disruptive presence of technology within the bucolic landscape of the American continent. In Hawthorne’s Sleepy Hollow for example, it is the interrupting shriek of a locomotive whistle that breaks the author’s harmonious reverie: “Now tension replaces repose: the noise arouses a sense of dislocation, conflict, and anxiety” (15). In the decidedly un-pastoral modern office space, the noise of the computer installation nevertheless signifies a momentous social change and irrevocable loss. Picking out these tensions has always been one of the show’s strengths – whether it is the computer, Draper’s double identity, or the quiet endurance of women to the misogyny of midcentury work and domestic life.

Change, however, has significant consequences for Ginsberg, the young copywriter and Holocaust survivor who, as CBS’s Jessica Firger observes, has been deteriorating psychologically for some time. The proximity of the IBM 360, and the incessant drone of its mind-controlling waves eventually puts him over the edge. As Draper and Peggy enter the office early in Episode 5, Ginsberg glowers into the room housing the IBM 360. “Stop humming, you’re not happy!” he explodes. As Peggy attempts to soothe her colleague, our perspective shifts to look out at them from inside the glass-encased computer room. From here, the mainframe’s ambient noise muffles Peggy’s words, suggesting isolation between human and non-human. This play of speech and silence reoccurs later in the episode as Ginsberg, working alone on a Saturday with tissues wedged in his ears, spies Lou Avery and SC&P partner Jim Cutler inside the computer room, their voices made inaudible by the droning computer in a delicious homage to 2001 (see Vulture’s amusing gif). But the noise is clearly affecting Ginsberg. “It’s that hum at the office! It’s getting to me!” he tells Peggy later that evening. He even claims the computer has affected his sexuality.

Ginsberg’s noise complaints would have resonated in 1969 New York. In November of that year, the New York Times ran a feature on the city’s nerve-shattering noise pollution, calling it a “slow agent of death.” In addition to the myriad construction projects, subways, car horns, jet planes, and standing machinery populating the city soundscape, office workers found scant respite indoors where phones, air conditioners, “computers and typewriters and tabulators” whirred, whined, and clacked throughout the day. The article went on to report that scientists studying the impact of prolonged noise exposure on the human body had concluded a variety of ill effects on the heart and nervous system. Though no connection was made between computers and sexuality (as Ginsberg claimed), the article reported that laboratory rats under prolonged noise exposure had indeed “turned homosexual,” an opinion that underlined deterministic associations between sexuality, psychological disorder, and external stimuli.

An advertisement for the IBM 360. Borrowed from Wikimedia Commons.

An advertisement for the IBM 360. Borrowed from Wikimedia Commons.

As SO! editor Jennifer Stoever-Ackerman has argued, noise in midcentury New York also signified a sonic-racial politics, in which the mainstream “listening ear” recoiled at the “noise” created by Black and Puerto Rican others. In terms of Mad Men’s computer however, it is technology, economic anxiety, and mental illness, rather than ethnicity that frames sonic disruption. The basis of these tensions are similar however, and various interactions with SC&P’s IBM 360 demonstrate, as Stoever-Ackerman writes in SO!, “the ways in which Americans have been disciplined to consider some sounds as natural, normal, and desirable, while deeming alternate ways of listening and sounding as aberrant [and] dangerous.” Though similar, the conflict with technology on Mad Men does not suggest a clear us/them, or us/”it” binary. The banging of construction may be at first antagonistic, but it’s finite – eventually the computer is normalized within the SC&P office space to the extent that Peggy chides Ginsberg’s exasperation in Episode 5 by insisting “it’s just a computer!” Ginsberg’s reaction is more complex however, implicating a contradictory relationship with technology: once fully installed, has the droning computer become “natural, normal, and desirable” despite previous ambivalence? Is the keen awareness and anxiety towards technology symbolized through Ginsberg (albeit in a extreme form) suggested as the “aberrant” listening practice, or could it be Peggy’s apparent acceptance?

Like most cultural texts set in the past, it is possible to read Mad Men allegorically, as suggesting a certain ordering of meaning and values. From the perspective of those who have long since domesticated computers, the controversies and tropes activated by SC&P’s IBM 360 might strike us as familiar, even quaint. As the sociologist Bruno Latour has argued however, we would be wise to consider how technology exerts a kind of social agency that structures and impacts our daily lives. As historical symbolism, the sounds and noises of the IBM 360 on Mad Men should remind us that technological progress is not teleological, but a struggle over meaning in which anxieties (about jobs, mind-control, surveillance, subjectivity, etc.) may be variously accommodated, suppressed, or dismissed as irrational.

Featured image: An IBM 360 Mainframe. Borrowed from Wikimedia Commons CC 2.0

Andrew J. Salvati is a Media Studies Ph.D. candidate at Rutgers University. His interests include the history of television and media technologies, theory and philosophy of history, and representations of history in media contexts. Additional interests include play, authenticity, the sublime, and the absurd. Andrew has co-authored a book chapter with colleague Jonathan Bullinger titled “Selective Authenticity and the Playable Past” in the recent edited volume Playing With the Past (2013)and has written a recent blog post for Play the Past titled The Play of History.”

tape reelREWIND!…If you liked this post, you may also dig:

“DIY Histories: Podcasting the Past” -Andrew J. Salvati

“The Noise of SB 1070: Or Do I Sound Illegal to You?”– Jennifer Stoever-Ackerman

“DIANE… The Personal Voice Recorder in Twin Peaks” -Tom McEnaney