Archive | Video Games RSS for this section

Sounding Out! Podcast #29: Game Audio Notes I: Growing Sounds for Sim Cell

Sound and Pleasure2

DOI: https://doi.org/10.59350/055kj-6jq16

A pair of firsts! This is both the lead post in our summer Sound and Pleasure series, and the first podcast in a three part series by Leonard J. Paul. What is the connection between sound and enjoyment, and how are pleasing sounds designed? Pleasure is, after all, what brings y’all back to Sounding Out! weekly, is it not?

In today’s installment Leonard peels back the curtain of game audio design and reveals his creative process. For anyone curious as to what creative decisions lead to the bloops, bleeps, and ambient soundscapes of video games, this is essential listening. Stay tuned for next Monday’s installment on the process of designing sound for Retro City Rampage, and next month’s episode which focuses on the game Vessel. Today, Leonard begins by picking apart his design process at a cellular level. Literally! -AT, Multimedia Editor

–

CLICK HERE TO DOWNLOAD: Game Audio Notes I: Growing Sounds for Sim Cell

SUBSCRIBE TO THE SERIES VIA ITUNES

ADD OUR PODCASTS TO YOUR STITCHER FAVORITES PLAYLIST

–

Game Audio Notes I: Growing Sounds for Sim Cell

Sim Cell is an educational game released in Spring 2014 by Strange Loop Games. Published by Amplify, a branch of News Corp, it teaches students how the human cell works. Players take control of a small vessel that is shrunk down to the size of a cell and solve the tasks set by the game while also learning about the human cell. This essay unpacks the design decisions behind the simulation of a variety of natural phenomena (motion, impact, and voice) in Sim Cell.

For the design of this game I decided to focus on the elements of life itself and attempted to “grow” the music and sound design from synthetic sounds. I used the visual scripting language Pure Data (PD) to program both the sounds and the music. The music is generated from a set of rules and patterns that cause each playback of a song to be slightly different each time. Each of the sound effects are crafted from small programs that are based on the design of analogue modular synthesizers. Basic synthetic audio elements such as filtered noise, sawtooth waves and sine waves were all used in the game’s sound design.

An image of the game's space-like world.
A screenshot of Sim Cell’s microscopic space-like world. Used with permission (c) 2014 Amplify.

The visuals of the game give a feeling of being in an “inner space” that mirrors outer space. I took my inspiration for Sim Cell‘s sound effects from Louis and Bebe Baron’s score to Forbidden Planet. I used simple patches in PD to assemble all of the sounds for the game from synthesis. There aren’t any recorded samples in the sound design – all of the sound effects are generated from mathematics.

The digital effects used in the sound design were built to emulate the effects available in vintage studios such as plate reverb and analog delay. I found a simulation of a plate reverb that from a modified open source patch from the RjDj project, and constructed an analogue delay by using a low-pass filter on a delayed signal.

In keeping with the vintage theme, I used elements of sound design from the early days of video games as well. Early arcade games such as Space Invaders used custom audio synthesis microchips for each of their sounds. In order to emulate this, I gave every sound its own patch when doing the synthesis for Sim Cell. I learned to appreciate this ethic of design when playing Combat on the Atari 2600 while growing up. The Atari 2600 could only output two sounds at once thus had a very limited palette of tones. All of the source code for the sounds and music for Sim Cell are less than 1 megabyte, which shows how powerful the efficient coding of mathematics can be.

Vector image on the left, raster graphics on the right.
Vector image on the left, raster graphics on the right. Photo used with permission by the author.

Another cool thing about generating the sounds from mathematics is that users can “zoom in” on sounds in the same way that one can zoom in to a vector drawing. In vector drawings the lines are smooth when you zoom in, as opposed to rasterized pictures (such as a JPEG) which reveal a blurry set of pixels upon zooming. When code can change sounds in real-time, it makes them come alive and lends a sense of flexibility to the composition.

For a human feeling I often filter the sounds using formant frequencies which simulate the resonant qualities of vowel sounds, thus offering a vocal quality to the sample. For alarm sounds in Sim Cell I used minor second intervals. This lent a sense of dissonance which informed players that they needed to adjust their gameplay in order to navigate treacherous areas. Motion was captured through a whoosh sound that used filtered and modulated noise. Some sounds were pitched up to make things seem like they were travelling towards the player and likewise they exploited a sense of doppler shift when pitched down, giving the feel that a sound was traveling away from the player. Together, these techniques produced a sense of immersion for the player while simultaneously building a realistic soundscape.

Our decision to use libPD for this synthesis turned problematic and its processing requirements were too high. In order to remedy this, we decided to convert our audio into samples. Consider a photograph of a sculpture. Our sounds, like sculpture in the photograph, could now only be viewed from one direction. This meant that the music now only had a single version and that sound effects would repeat as well. A small fix was exporting the file with a set of five different intensities from 0 to 1.0. Like taking a photograph from several angles, this meant that the game could play a sample at intensity levels of 20%, 40%, 60%, 80% and 100%. Although a truly random sense of variation was lost, this method still conveyed the intensity of impacts (and other similar events) generated by the physics engine of the game.

An image of the inexpensive open-source computer: The Rasberry PI
An image of the inexpensive open-source computer: The Rasberry PI. Photo used with permission by the author.

PD is a great way to learn and play with digital audio since you can change the patch while it is running, just like you might do with a real analogue synthesizer. There’s plenty of other neat stuff that PD can do, like being able to be run on the Raspberry Pi, so you could code your own effects pedals and make your own synths using PD for around $50 or so. For video games, you can use libPD to integrate PD patches into your Android or iOS apps as well. I hope this essay has offered some insight as to my process when using PD. I’ve included some links below for those interested in learning more.

Additional resources:

–

Leonard J. Paul attained his Honours degree in Computer Science at Simon Fraser University in BC, Canada with an Extended Minor in Music concentrating in Electroacoustics. He began his work in video games on the Sega Genesis and Super Nintendo Entertainment System and has a twenty year history in composing, sound design and coding for games. He has worked on over twenty major game titles totalling over 6.4 million units sold since 1994, including award-winning AAA titles such as EA’s NBA Jam 2010, NHL11, Need for Speed: Hot Pursuit 2, NBA Live ’95 as well as the indie award-winning title Retro City Rampage.

He is the co-founder of the School of Video Game Audio and has taught game audio students from over thirty different countries online since 2012. His new media works has been exhibited in cities including Surrey, Banff, Victoria, São Paulo, Zürich and San Jose. As a documentary film composer, he had the good fortune of scoring the original music for multi-awarding winning documentary The Corporation which remains the highest-grossing Canadian documentary in history to date. He has performed live electronic music in cities such as Osaka, Berlin, San Francisco, Brooklyn and Amsterdam under the name Freaky DNA.

He is an internationally renowned speaker on the topic of video game audio and has been invited to speak in Vancouver, Lyon, Berlin, Bogotá, London, Banff, San Francisco, San Jose, Porto, Angoulême and other locations around the world.

His writings and presentations are available at http://VideoGameAudio.com

–

Featured image: Concept art for Sim Cell. Used with permission (c) 2014 Amplify.

—

tape reel

REWIND! . . .If you liked this post, you may also dig:

Sounding Out! Podcast #10: Interview with Theremin Master Eric Ross- Aaron Trammell

Papa Sangre and the Construction of Immersion in Audio Games- Enongo Lumumba-Kasongo

Playing with Bits, Pieces, and Lightning Bolts: An Interview with Sound Artist Andrea Parkins

—


Digital Analogies: Techniques of Sonic Play

Sound and Play

DOI: https://doi.org/10.59350/9xkfz-dep85

Editor’s Note: Welcome to Sounding Out!‘s fall forum titled “Sound and Play,” where we ask how sound studies, as a discipline, can help us to think through several canonical perspectives on play. While Johan Huizinga had once argued that play is the primeval foundation from which all culture has sprung, it is important to ask where sound fits into this construction of culture; does it too have the potential to liberate or re-entrench our social worlds? Here, Roger Moseley challenges us to rethink the philosophical discourses of both sound and play and locates the moments in which they intersect and interface. From games of Telephone to Guitar Hero, Moseley considers the ways in which sonic play can help us understand the phantasmic binaries of the analog and digital.–AT

—

Throughout the distinguished intellectual lineage of play (where it is touched on by notable philosophers such as Plato, Montaigne, Kant, Schiller, Gadamer, Derrida, and Baudrillard), little attention has been paid to the parallels that can be drawn between sound and play as both media and phenomena. The very name of today’s most prominent cultural and technological locus of play, the video game, overtly privileges the eye at the expense of the ear. As recent research and creative work by such figures as Aaron Oldenburg, Aaron Trammell, George Karalis, and Enongo Lumumba-Kasongo indicates, a surge of interest in audio games, as well as video games that emphasize the importance of sound while eschewing or minimizing visual stimuli, is acting as a salutary corrective to this oculocentrism. In what follows, I suggest that bringing sonic and musical techniques to bear on this history might afford new insights into play and its myriad configurations. Conceiving of play sonically entails thinking of sound playfully. This intersectional logic can, I argue, unpick binarisms that enforce problematic distinctions and constrict thought. To demonstrate this, I conclude by deploying the concept of play to redefine the relationship between the digital and the analog—and vice versa.

How can play be defined in a manner that encompasses its farrago of meanings and associations? For video game designers and theorists Katie Salen and Eric Zimmerman, the answer is deceptively simple: play is “free motion within a more rigid structure” (Rules of Play, 304). To illustrate the flexibility of this definition, Salen and Zimmerman allude to the phenomenon of light playing upon the ocean waves. They leave unexamined, however, the intimacy and richness of the relationship between play and sound. From a scientific perspective, the patterned oscillation of which a sequence of sound waves is constituted consists of free motion within the limits set forth by the laws of physics. When disciplined and deployed as a cultural technique–take the play of musical instruments for example–sonic play is humanized and rendered transitive. But, we might also suggest that instruments play people, citing the sensation of automation with which fingers flash over fretboard or keyboard. Moving further away from anthropocentrism, we can observe how sonic technologies render play intransitive once more. From the barrel organ to the iPod, sound plays without human aid when mechanically reproduced. This way of framing reproduction invokes and extends Roger Caillois’s playful category of mimicry, which can be construed as faithful imitation, deceptive fakery, or even a Baudrillardian attempt to simulate a phenomenon that never existed.

In order to pay due attention to both the technologies through which sonic play is mediated as well as the cultural techniques imbue it with significance, I suggest that we supplement Salen and Zimmerman’s definition by thinking of freedom, motion, and structure in both digital and analogical terms. To an extent, the adoption of this modish epistemological framework acknowledges that conceptions of play are always constrained by their prevailing intellectual context. More importantly, however, I contend that technologies of sonic generation and representation from the seventeenth to the nineteenth centuries can be understood to play with the categories of the digital and the analog avant la lettre (ou le chiffre). The two categories are not mutually exclusive and to treat them as such would be to subjugate the granularity of the analog to the binary logic of the digital. Rather, they co-exist as modes between which sounds and players freely oscillate.

The origins of digital sonic play lie within the human body.  As Johan Huizinga put it, “the link between play and instrumental skill is to be sought in the nimble and orderly movements of the fingers.” In the course of musical performance, human digits perform innumerable calculations. At its crudest level, musical performance from a score can be construed as a sort of algorithmic play through which mimetic fidelity is evaluated (and wrong notes relentlessly tallied). This ludic logic is at its most visible in rhythm-action video games such as Guitar Hero in which the score is no longer a text but rather a quantitative analysis. The iconography of these games usually indexes a set of digital technologies used primarily for the recording, editing, and playback of music. On the one hand, this relationship can be traced back to Leibniz’s exposition of ars combinatoria and his “invention” of binary; on the other, it is realized by the hydraulic organ and composing machine devised and programmed by the Jesuit polymath Athanasius Kircher, both of which are depicted in his Musurgia universalis (1650). In media-archaeological terms, the combination of Leibniz’s concepts and Kircher’s mechanisms gave rise to the hardware and software of Joseph Marie Jacquard’s revolutionary loom, Charles Babbage’s prototypical Analytical Engine, the player piano, the IBM punch card, and the MIDI sequencer before resurfacing in Guitar Hero, a piece of software that, in purely algorithmic terms, enlists the player’s digits to verify checksums.

An example of Kirscher and Scott's Organum.
An example of Kircher and Schott’s Organum, an early example of the digitization of music.

Such digital grids may constitute the field and the rules of sonic play, but they must be supplemented by analog elements if play is to flourish. As detailed in C. P. E. Bach’s Versuch über die wahre Art das Clavier zu spielen (1753/62), the clavichord and its descendants distinguished themselves from the harpsichord and the organ by endowing the keyboard with an infinite sensitivity to touch, thereby enabling a mimetic spectrum of emotional flow with unprecedented verisimilitude. Analogicity also provides another perspective on Caillois’s concept of mimicry, according to which one object or activity playfully stands in for another via imitation, deception, or make-believe. Additionally, the curves of Ernst Chladni’s figures, which materialized sound as sand, exemplify this sonic and mimetic trajectory as they rely on both Hermann von Helmholtz’s pioneering work on acoustics and the complex history of phonography to the development of analog synthesis.

In terms of sonic play, digital and analog elements can be chiastically recombined and reconfigured. A sonic communication game such as Telephone relies on the human propensity for analogy and its corrupting influence on the integrity of information transfer, playfully inverting the conditions and functions of the “real” telephone (which was engineered to compress informational content digitally without jeopardizing meaning). In much electronic dance music, the digital latticework, simultaneously visualized and rendered audible by the sequencer’s grid, constitutes a field of play overlaid with vocals, sweeps, and other analog elements that, in turn, have been captured via digital sampling. As a kind of meta-game, a mash-up plays with sonic elements whose relations can be parsed in the digital terms of Leibnizian recombinatorial play, but equally important are the unintended associations and analogies which inevitably emerge. And while games such as Guitar Hero foreground digital techniques of sonic reproduction, they simultaneously foster diverse forms of analogical play involving the player’s manipulation of the sonic (and social) behavior of her on-screen avatar—and vice versa.

Playing telephone. Image borrowed from diamondmanga @Wikimedia.
Playing telephone. Image borrowed from diamondmanga @Wikimedia.

There is no doubt that the status of sound and its mediation through and as play have too often gone unacknowledged. As well as seeking to rectify this state of affairs by stressing the importance of sound in relation to the playful operations of other media, we might also dwell on the distinguishing features that set it apart: sound and the techniques that shape it are unique in the ways they simultaneously trace and are traced by the materials, technologies, and metaphors of play.

–

Roger Moseley is an Assistant Professor in the Department of Music at Cornell University. His most recent research brings a media-archaeological perspective to bear on musical performance and improvisation. He is particularly interested in how the concept of play informs sonic practices and cultural techniques. Active as a collaborative pianist on both modern and historical instruments, he has recently published essays on digital games in the contexts of musical and visual culture. His current book project is entitled Digital Analogies: Interfaces and Techniques of Musical Play.

—

tape reel

REWIND! . . .If you liked this post, you may also dig:

Papa Sangre and the Construction of Immersion in Audio Games– Enongo Lumumba-Kasongo

Playing with Bits, Pieces, and Lightning Bolts: An Interview With Sound Artist Andrea Parkins— Maile Colbert

Experiments in Aural Resistance: Nordic Role-Playing, Community, and Sound— Aaron Trammell

—