Archive | sonification RSS for this section

The Sound of What Becomes Possible: Language Politics and Jesse Chun’s 술래 SULLAE (2020)

“To this day I think about all the strange words I missed out on, all the losses I’m still carrying from faraway…I still think of the time when I spoke one language, and that language was whole.”

Chun 2020

Language can be a site of loss, a wholeness with which one, due to migration, has never really known. In the above passage, artist, Jesse Chun, reflects on how her grandmother spoke words in a language she did not understand, but yearned to hear and feel those sounds after her passing. There is a sonic residue that sticks to diasporic experiences. There are sounds that can stir up a blend of affect and ideation that is comforting when whiteness is unsettling. It is this disjuncture between words, meaning, and their sounds, that drew me to Chun’s work, 술래 SULLAE (2020). This piece reminded me of how sound, in its most ambiguous and queer forms, can hold the contingencies of history, language, memory, family, and the genealogies of loss that mark these sites of colonial dispossession.

술래 SULLAE (2020) is a single channeled video that draws from ganggang sullae, a Korean seasonal harvest and fertility ritual that integrates song and dance and is typically performed by women under the glow of moonlight. The participants hold hands forming a circle that through their movement, expands, disassembles, and changes its form. The songs can be both impromptu or pre-determined and encourages the participants to express their feelings in chorus with one another.

Video made by the Cultural Heritage Administration of the Republic of Korea (2008) for UNESCO “intangible heritage” application

Diana Seo Hyung Lee (2020) suggests that historically ganggang sullae was meant to provide a forum for its participants to express emotions connected to living within patriarchal systems of power and oppression. She writes: “the women participating would not have been able to, in their everyday lives, sing, speak loudly, nor leave the house at night, in the patriarchal society of ancient Korea. This dance was a license for their one release.” In 술래 SULLAE, the dance proves to be a defiant presence. The women flash on screen as an unbreakable chain reinscribing a gendered history with new sounds and images that gesture to emancipatory possibilities.

술래 SULLAE combines archival clips of ganggang sullae, index pages from intonation books, images of Hangul and English consonants and audio splices from YouTube tutorials on how to pronounce English correctly. In the video, language becomes unhinged from expectation but at the same time, given form through history. The sound of the English language is disembodied and spliced into phonemic pulses.  In 술래 SULLAE, Chun has created an encounter with the grammars of polyphony; a simultaneity of sounds that are both restrained by and resistant to the imposition of English on the Korean diaspora. Through what Chun has described as a form of “unlanguaging” following Rey Chow, her audience is witness to new meanings produced through the abstraction, manipulation, and redaction of sounds and symbols from the English language.

Still from 술래 SULLAE (2020), courtesy of the artist

Chun’s editing and manipulation of English sounds is intentional. In an interview with Art Forum, Chun shares: “Taking the sound apart but still keeping it within the conceptual framework of English made me think about what else is embedded in making a language. English is tied up with legacies of imperialism; there’s so much unseen violence that is part of how this language is institutionalized.” What remains after the edits is an inventory of sounds that disrupts the primacy of the vowel as central to English word construction and thus, central to colonial imagination.  

Like Chun, I realize that my conceptualizing of language is within an English framework, but my hope is that when we turn to the affective and when we begin to pull language a part, something different, something resistant, is produced.  I am neither an expert in English nor Korean linguistics, it was the sounds in this work that pulled me into it. In thinking with 술래 SULLAE, I’m interested in what becomes possible in the absence of the vowel. I turn to the interruptive potential of consonant sounds to affect and incite methods of communication outside of those steeped in colonial dominance. What does it mean to de-emphasize the function of vowel sounds in language and reorient our listening to the consonant? What do consonant sounds teach us about the sonics of race that underwrite hierarchies of language? What methods of communication become possible when we do away with words and are left with only their sonic substance?

술래 SULLAE 3-channel excerpt

Through her assemblage of consonant sounds in 술래 SULLAE, Chun is making a deliberate choice to describe and animate a politics of language through refusing its colonial enclosures and turning to the aesthetic in order hold the excesses of description. She refuses the vowel in this piece, not by denying its presence, but instead relegating it to the soundless and the unfamiliar, a space of, in her words, “untranslatability.” In this undoing, consonants become the emotive force where new meanings and orientations to the sounds that mark our words are forged.

술래 SULLAE opens with the sound ssshhh; a pairing of consonant sounds that is often associated with insisting on silence, a sound meant to reprimand. Chun extracts and emplaces this sound in a new aesthetic landscape that is independent and unregulated by colonial schemas of enunciation and translation. The prominent soundscapes of the video are consonant sounds and when removed from their phonetic relations to vowels these sounds undo the presumptive structuring or potential reprimand of English. In 술래 SULLAE, we are meant to experience the fullness of the consonants’ timbre…ssshhh, ppp, ddd, tttt, kkkk…these edited clips of sound originally meant to instruct and assimilate speech into English pronunciations now serve a different function. For me, they secure Chun’s political orientation: one that is about the crafting of a world that involves the careful consideration of the logistics, function, and embedded emotions of the sounds that inhabit it.

score (for unlanguaging), 2020
graphite, watermarks, paper, aluminum frame, 13 x 16 inches

All languages contain their own unique set of vowels and consonants, but, Anne Carson reminds us that: “The importance of vowels to human speech has remained. There are words in English without consonants, but so central are vowels to word construction that there isn’t a word in English that doesn’t include a vowel.” In speech, consonants sounds are meant to break up the intended agenda of vowels. The ssshhh, ppp, ddd, tttt, kkkk, are antithetical to the circle or the rounded mouth needed to voice a vowel sound. Unlike the openness of a vowel, producing consonant sounds involves a narrowing of the vocal tract. This narrowing is referred to as constriction or the obstruction of breath whereby sound is produced by a form of corporeal tension. Consonant sounds also demand all the mechanics of the mouth: the lips, the teeth, the tongue, and the palette. Shhhh, requires the corners of the lips to lower and rather than rounding, the lips become pursed, and teeth become exposed. Parts of the mouth are drawn in. The soft palate is raised, and the tongue reaches upwards towards the roof of the mouth without touching it and then the tip of the tongue lowers behind the teeth.

Consonants emerge out of collectivity. Where a vowel is sounded without vocal constraint, consonants require more effort. Their sounds are produced through intricate bodily choreographies in the mouth that involve both constriction and collaboration. Ganggangsullae likewise relies on effort and interdependence. Participants collectively determine the speed and/or shape of their dance. They may even become serpentine or separate into smaller circles depending on what the group decides. The dance also provides an aesthetic space for its participants to voice frustration, anger, and tension through song with the hopes of producing reprieve from gendered hardships. Chun has decided to withhold these songs from her audience; we never hear the women singing. Through this erasure, Chun embeds the consonant sound with affective force whereby a politics of language and gendered presence is enunciated through and beyond a form of silencing. The dance redirects trajectories of dominance whereby the shushing takes on a new voice imbued with agency and hope. Because of how Chun isolates and amplifies its sound, ssshhhh is free to take on different meanings and associations. For me, I was reminded of rushing water or gusts of wind, or the sound used to lull my child to sleep. I was brought into another index of knowing and relating.

술래 SULLAE, 3-channel installation view, 2020, courtesy of the artist

The sounds of language hold erasures and layered histories often obfuscated by our mundane encounters with them. Largely understood as the most sonorous part of the syllable, vowels produce the loudest speech sounds and their capacity for holding larger amplitudes or louder volumes have been linked to the sonic expression of emotion. Consonant sounds are more pragmatic than vowels. They are known for their functionality, for the ways in which they assemble the semantic structure of words and for their capacities to hold vowels in place or as Anne Carson describes as “delineating meaning amid the flow of open vowel sounds.” Consonant and vowel sounds map out different functional trajectories by virtue of the shape of mouth and orientation of breath that these sounds demand. Like Chun, I’m interested in what political orientations become possible when we source emotion elsewhere, beyond the confines of spoken words imposed upon us.

The word consonant is a noun, a word used to identify or classify, a semantic enclosure that establishes a subject or object. But unlike the word vowel, consonant is also an adjective. A consonant possesses the capacity to describe, to name, to tell us more. Adjectives parcel out description on states of being, in this way, they are inherently phenomenological. In 술래 SULLAE, Chun empties vowels of their sonic substance leaving behind traces of fragmented characters and differently shaped circles in their wake. They are stripped of breath and their symbolic value forming a new method of communication that reroutes expectations of what language, as we know it, can do and sound like. Like, ganggang sullae, the vowel is premised on the shape of a circle, but in 술래 SULLAE, Chun provokes us to think about what becomes possible beyond the circular structuring device, what becomes possible beyond the purview of the violent embeddedness of English and its colonial exigencies.

new moons, 2020, graphite on wall, surveillance mirror, 18 x 6 inches each
untitled (ㄷ), 2020, a functional concrete stool, courtesy of artist

Chun has noted that the moon that hovers above the ganggangsullae is yet another site of imperial conquest. In Art Forum, Chun states: “when I look up at it to feel comforted or to find solace, I’m reminded of colonial violence and an agenda that’s projected onto it. In that way, the moon also reflected how I see language.” Chun’s turn to consonants signals a reshaping of the colonial frame that does not disavow or idealize the legacies of imperialism on systems of communication, but instead highlights the tensions and obstructions produced in its shadows.

Featured image: 술래 SULLAE, 2020, single-channel version, courtesy of artist

Casey Mecija is an Assistant Professor in the Department of Communication & Media Studies at York University. Her current research examines sound as a mode of affective, psychic and social representation, specifically in relation to diasporic experience. Drawing on sound studies, queer diaspora studies and Filipinx Studies, her research considers how sensorial encounters are enmeshed and disciplined by social and psychic conditions. She is also a musician and filmmaker, whose work has received a number of accolades and has been presented internationally.

tape-reel

REWIND! . . .If you liked this post, you may also dig:

Blank Space and “Asymmetries of Childhood Innocence”  –Casey Mecija

Re-orienting Sound Studies’ Aural Fixation: Christine Sun Kim’s “Subjective Loudness”–Sarah Mayberry Scott

Tape Hiss, Compression, and the Stubborn Materiality of Sonic Diaspora–Chris Chien

Critical bandwidths: hearing #metoo and the construction of a listening public on the web

“A focus on listening [with technology] shifts the idea of freedom of speech from having a platform of expression to having the possibility of communication” (K. Lacey)

One of the biggest social media event of the past decade, #metoo stands out as a pivotal shift in the future of gender relations. Despite its persistence since October 2017, #metoo is still under-theorized, and since its permutations generate countless hashtag sub-categories each passing week, making sense of it presents a conceptual quagmire. Tracing its history, identifying key moments, mapping its pro- and counter-currents present equally tough challenges to both data science and feminist scholars.

Meta-communication about #metoo abounds. Infographics and visualizations attempt to contain its organic growth into perceivable maps and charts; pop news media constantly report on its evolution in likes, counts, and retweets, as well as—and increasingly—in number of convictions, lawsuits, and reports. At the same time, #metoo has arguably created a discernible listening public in the way that Kate Lacey (2013) argues emerged with national radio: women’s stories have never been listened to with such wide reach and rapt attention.

How did #metoo create new listening publics? “#metoo” by Flickr User Prachatai, (CC BY-NC-ND 2.0)

The project I discuss here takes ‘hearing’ #metoo a step further into the auditory realm in the form of data sonification so as to to re-imagine an audience compelled to earwitness not just the scope but the emotional impact of women’s stories. Data sonification is a growing field, which from its inception has crossed between art and science. It involves a conceptual or semantic translation of data into relevant sonic parameters in a way that utilizes perceptual gestalts to convey information through sound.

Brady Marks and I created the #metoo sonification you’ll hear below by drawing from a public dataset spanning October 2017 to the early Spring of 2018 obtained from data.world. Individual tweets using the hashtag are sonified using female battle cries from video games; the number of retweets and followers forms a sort of swelling and contracting background vocal texture to represent the reach of each message. The dataset is then sped up anywhere between 10x to 1000x in order to represent perceivable ebbs and flows of the hashtag’s life over time. The deliberate aim in this design was to convey a different sensibility of social media content, one that demands emotional and intellectual attention over a duration of time. Given Twitter’s visual zeitgeist whereby individual tweets are perceived at a glance and quickly become lost in the noise of the platform the affective attitude towards “contagious” events becomes arguably impersonal. A sonification such as this asks the listener to spend 30 minutes listening to 1 month of #metoo: something impossible to achieve on the actual platform, or in a single visualization. The aim, then, is to interrupt social media’s habitual and disposable engagements with pressing civic debates. 

MeToo: It's About Power | Morningside Center for Teaching Social  Responsibility

A critique of big data visualization

To date, there have been more than 19 million #MeToo tweets from over 85 countries; on Facebook more than 24 million people participated in the conversation by posting, reacting, and commenting over 77 million times since October 15, 2017. In a global information society ‘big data’ is translated into creative infographics in order to simultaneously educate an overwhelmed public and elicit urgency and accord for political action. Yet ideological and political considerations around the design of visual information have lagged behind enthusiasm for making data ‘easy to understand’. At the other end of the spectrum, social media delivers personalized micro-trends directly and in real time to always-mobile users, reinforcing their information silos (Rambukkana 2015). Between these extremes, the mechanisms by which relevant local, marginalized or emergent issues come to be communicated to the wider public are constrained.

“A wordcloud featuring #metoo” by www.scootergenius.com (CC BY 2.0)

With this big idea in mind, the question we ask here is what would it mean to hear data?  Emergent work in sonification suggests that sound may afford a unique way to experience large-scale data suitable for raising public awareness of important current issues (Winters & Weinberg 2015). The uptake of sonification by the artistic community (see Rory Viner, Robert Alexander, among many others) signals its strengths in producing affective associations to data for non-specialized audiences, despite its shortcomings as a scientific analysis tool (Supper, 2018). Some of the more esoteric uses of sonification have been in the service of capturing what Supper calls ‘the sublime’ – as in Margaret Schedel’s “Sounds of Science: The Mystique of Sonification.”

Who’s listening on social media?

Within the Western canon of sound studies “constitutive technicities” (Gallope 2011) or what Sterne calls “perceptual technics” embody historically situated ways of listening that center technology as a co-defining factor in our relationship with sound. Within this frame, media sociologist Kate Lacey traces the emergence of the modern listening public through the history of radio. Using the metaphor of ‘listening in” and “listening out,” Lacey reframes media citizenship by pointing out that listening is a cultural as well as a perceptual act with defined political dimensions:

Listening out is the practice of being open to the multiplicity of texts and voices and thinking of texts in the context of and in relation to a difference and how they resonate across time and in different spaces. But at the same time, it is the practice and experience of living in a media age that produces and heightens the requirement, the context, the responsibilities and the possibilities of listening out (198)

According to Lacey, a focus on listening instead of spectatorship challenges the implicit active/passive dualism of civic participation in Western contexts. More importantly, she argues, we need to move away from the notion of “giving voice” and instead create meaningful possibilities to listen, in a political sense. Data sonification doesn’t so much ‘give voice to the voiceless’ but creates a novel relationship to perceiving larger patterns and movements.

Our interactions with media, therefore, are always already presumptive of particular dialogical relations. Every speech act, every message implies a listening audience that will resonate understanding. In other words, how are we already listening in to #metoo?  How and why might data sonification enable us to “listen out” for it instead? In order to get a different hearing, what should #metoo sound like?

What should #metoo sound like? “de #metoo à #wetogether” by Flickr user Jeanne Menjoulet, Paris 8 mars 2018 (CC BY 2.0)

Sonifying #metoo: the battle cries of gender-based violence

It is unrealistic to expect that your everyday person will read large archives of testimony on sexual harassment and gender-based violence. Because of their massive scale, archives of #metoo testimony pose a significant challenge to the possibility for meaningful communication around this issue.  Essentially drowning each other out, individual voices remain unheard in the zeitgeist of media platforms that automates quantification while speeding up engagement with individual contributions. To reaffirm the importance of voice would mean to reaffirm inter-subjectivity and to recognize polyphony as an “existential position of humanity” (Ihde 2007, 178). This was the problem to sonify here: how to retain individual voices while creating the possibility for listening to the whole issue at hand. Inspired by the idea of listening out, myself and artist collaborator Brady Marks set out to sonify #metoo as a way of eliciting the possibility for a new listening public.

Sophitia Alexandra of the Soul Calibur franchise, close up from Flickr User Ngo Quang Minh‘s image from Soul Calibur III (CC BY-SA 2.0)

The #metoo sonification project intersected deeply with my work on the female voice in videogames. My choice to use a mixed selection of battle cry samples from Soul Calibur, an arcade fighting game, was intuitive. Battle cries are pre-recorded banks of combat sounds that video game characters perform in the course of the story. Instances of #metoo on Twitter presumably represent the experiences of individual women, pumping a virtual fist in the air, no longer silent about the realities of gender-based violence. So hearing #metoo posts as battle cries of powerful game heroines made sense to me. But it’s the meta layers of meaning that are even more intuitive: as I’ve discussed elsewhere, female battle cries are notoriously gendered and sexualized. Listening to a reel of sampled battle cries is almost indistinguishable from listening to a pornographic soundscape. Abstracted in this sonification, away from the cartoonish hyper-reality of a game world, these voices are even more eerie, giving almost physical substance to the subject matter of #metoo. Just as the female voice in media secretly fulfils the furtive desires of the “neglected erogenous zone” of the ear (Pettman 2017, 17), #metoo is an embodiment of the conflation of sex with consent: the basis of what we now call ‘rape culture.’

Sonifying real-time data such as Twitter presents not only semantic (how should it sound like) but also time-scale challenges. If we are to sonify a month – e.g. the month of November 2017 (just weeks after the explosion of #metoo) – but we don’t want to spend a month listening, then that involves some conceptual time-scaling. Time-scaling means speeding up instances that already happen multiple times a second on a platform as instantaneous and global as Twitter. Below are samples of three different sonifications of #metoo data, following different moments in the initial explosion of the hashtag and rendered at different time compressions. Listen to them one at a time and note your sensual and emotive experience of tweets closer to real-time playback, compared to the audible patterns that emerge from compressing longer periods of time inside the same length audio file. You might find that the density is different. Closer to real-time the battle cries are more distinctive, while at higher time compressions what emerges instead is an expanding and contracting polyphonic texture.

Vocalizations of female pleasure/affect, video game battle cries already have a special relationship to technologies of audio sampling and digital reproduction as Corbett & Kapsalis describe in Aural sex: the female orgasm in popular sound.”  This means that the perceptual technics involved in listening to recorded female voices are already coded with sexual connotations. Battle cries in games are purposely exaggerated so as to carry the bulk of emotional content in the game’s experiential matrix. Roland Barthes’ notion of the “grain of the voice”—the presence of the body in (singing) voice—is frequently evoked in describing the substantive role that game voices play in the construction of game world immersion and realism. In the #metoo sonification, I decontextualize the grain of the voice—there are no visual images, narrative, or gameplay; the battle cries are also acousmatic, in that there are no bodies visually represented from which these sounds emanate.

The battle cry in this #metoo sonification is the ultimate disembodied voice, resisting what Kaja Silverman (1988) calls the “norm of synchronization” with a female body in The Acoustic Mirror (83). As acousmatic voices, these battle cries could be said to exist on a different conceptual and perceptual plane, “disturbing the taxonomies upon which patriarchy depends,” to quote Dominic Pettman in Sonic Intimacy. (22). In other words, the sounds exist in a boundary space between combat sounds and orgasmic sounds highlighting for the listener the dissonance between the supposed empowerment of ‘speaking out’ within a culture that remains staunchly set up to sexualize women; something one can hardly ignore given the media’s reserved treatment of #metoo.

“Princess Zelda’s New Mouth” by Flickr UserMouthGuy2013, Public Domain

Liberated from the game world these voices now speak for themselves in the #metoo sonification, their sensuality all the more hyper-real. The player has no control here, as the battle cries are not linked to specific game actions, rather they are synchronized autonomously to instances of #metoo confessionals.  In fact, the density of the sonification as time speeds up will overwhelm listeners with its boundlessness; echoing how contemporary media treats the sounds of the female orgasm as a renewable and inexhaustible resource, even as reports of sexual harassment and gender-based violence continue to pile on in 2021. Yet we intend that the subject matter resists pleasure, rendering the sonic experience traumatic as the chilling realization sets in that listeners are hailed to accountability by #metoo. The experience should instead be unsettling, impactful, grotesque, and deeply embodied. 

Concluding remarks

Listening both metaphorically and literally goes to the very heart of questions to do with the politics and experience of living and communicating in the media age. In her paper on the sonic geographies of the voice, AM Kanngieser notes in “A Sonic Geography of Voice“: “The voice, in its expression of affective and ethico-political forces, creates worlds” (337). It is not just in the grain but in the enunciation that battle cries find their political significance in this sonification. As the hyper-real gasps and moans of game heroines animate individual moments of #metoo the codification of cartoonish voices resists being subconsciously “absorbed into the dialogic exchange” (342) of habitual media consumption. Listening to the sonification is instead an experience of re-coding the voice, reconfiguring the embedded meanings of game sound to a new and contradictory context: a space that challenges neoliberal appropriations of radical communication and discourse (348). This is not data sonification that delights the listener or simply grants them access to ‘information’ in a different format; rather it calls on the listener to de-normalize their received technicity and perceptions and to connect to the emotional inter-subjectivity of this call to action.

Most importantly, the #metoo sonification invites the auditeur to listen in, to take an active role in the reconfiguration of meanings and absorb their political dimensions. These are the stories of #metoo; these are the voices of women, of men, of marginalized peoples, emerging from the zeitgeist of Twitter to ask us to earwitness gender-based violence. We are a new listening public, wanting and needing to create new worlds. A critical bandwidth is the smallest perceivable unit of auditory change, in psychology terms. This sonification begs the question, how many battle cries will it take for us to end gender-based violence by fostering equitable worlds?

Featured Image: “Listen to What You See” by Flickr User Hernán Piñera (CC BY-SA 2.0)

Milena Droumeva is an Assistant Professor and the Glenfraser Endowed Professor in Sound Studies at Simon Fraser University specializing in mobile media, sound studies, gender, and sensory ethnography. Milena has worked extensively in educational research on game-based learning and computational literacy, formerly as a post-doctoral fellow at the Institute for Research on Digital Learning at York University. Milena has a background in acoustic ecology and works across the fields of urban soundscape research, sonification for public engagement, as well as gender and sound in video games. Current research projects include sound ethnographies of the city (livable soundscapes), mobile curation, critical soundmapping, and sensory ethnography. Check out Milena’s Story Map, “Soundscapes of Productivity” about coffee shop soundscapes as the office ambience of the creative economy freelance workers. 

Milena is a former board member of the International Community on Auditory Displays, an alumni of the Institute for Research on Digital Learning at York University, and former Research Think-Tank and Academic Advisor in learning innovation for the social enterprise InWithForward.  More recently, Milena serves on the board for the Hush City Mobile Project founded by Dr. Antonella Radicchi, as well as WISWOS, founded by Dr. Linda O Keeffe.

tape reel

REWIND! . . .If you liked this post, you may also dig:

One Scream is All it Takes: Voice Activated Personal Safety, Audio Surveillance, and Gender ViolenceMaría Edurne Zuazu 

Gendered Sonic Violence, from the Waiting Room to the Locker Room–Rebecca Lentjes

SO! Amplifies: Mendi+Keith Obadike and Sounding Race in America

detritus 1 & 2 and V.F(i)n_1&2 : The Sounds and Images of Postnational Violence in Mexico–Luz María Sánchez

Hearing EugenicsVibrant Lives

%d bloggers like this: