The Cyborg’s Prosody, or Speech AI and the Displacement of Feeling
In summer 2021, sound artist, engineer, musician, and educator Johann Diedrick convened a panel at the intersection of racial bias, listening, and AI technology at Pioneerworks in Brooklyn, NY. Diedrick, 2021 Mozilla Creative Media award recipient and creator of such works as Dark Matters, is currently working on identifying the origins of racial bias in voice interface systems. Dark Matters, according to Squeaky Wheel, “exposes the absence of Black speech in the datasets used to train voice interface systems in consumer artificial intelligence products such as Alexa and Siri. Utilizing 3D modeling, sound, and storytelling, the project challenges our communities to grapple with racism and inequity through speech and the spoken word, and how AI systems underserve Black communities.” And now, he’s working with SO! as guest editor for this series (along with ed-in-chief JS!). It kicked off with Amina Abbas-Nazari’s post, helping us to understand how Speech AI systems operate from a very limiting set of assumptions about the human voice. Then, Golden Owens took a deep historical dive into the racialized sound of servitude in America and how this impacts Intelligent Virtual Assistants. Last week, Michelle Pfeifer explored how some nations are attempting to draw sonic borders, despite the fact that voices are not passports. Today, Dorothy R. Santos wraps up the series with a meditation on what we lose due to the intensified surveilling, tracking, and modulation of our voices. [To read the full series, click here] –JS
—
In 2010, science fiction writer Charles Yu wrote a story titled “Standard Loneliness Package,” where emotions are outsourced to another human being. While Yu’s story is a literal depiction, albeit fictitious, of what might be entailed and the considerations that need to be made of emotional labor, it was published a year prior to Apple introducing Siri as its official voice assistant for the iPhone. Humans are not meant to be viewed as a type of technology, yet capitalist and neoliberal logics continue to turn to technology as a solution to erase or filter what is least desirable even if that means the literal modification of voice, accent, and language. What do these actions do to the body at risk of severe fragmentation and compartmentalization?
I weep.
I wail.
I gnash my teeth.
Underneath it all, I am smiling. I am giggling.
I am at a funeral. My client’s heart aches, and inside of it is my heart, not aching, the opposite of aching—doing that, whatever it is.
Charles Yu, “Standard Loneliness Package,” Lightspeed: Science Fiction & Fantasy, November 2010
Yu sets the scene by providing specific examples of feelings of pain and loss that might be handed off to an agent who absorbs the feelings. He shows us, in one way, what a world might look and feel like if we were to go to the extreme of eradicating and off loading our most vulnerable moments to an agent or technician meant to take on this labor. Although written well over a decade ago, its prescient take on the future of feelings wasn’t too far off from where we find ourselves in 2023. How does the voice play into these connections between Yu’s story and what we’re facing in the technological age of voice recognition, speech synthesis, and assistive technologies? How might we re-imagine having the choice to displace our burdens onto another being or entity? Taking a cue from Yu’s story, technologies are being created that pull at the heartstrings of our memories and nostalgia. Yet what happens when we are thrust into a perpetual state of grieving and loss?
Humans are made to forget. Unlike a computer, we are fed information required for our survival. When it comes to language and expression, it is often a stochastic process of figuring out for whom we speak and who is on the receiving end of our communication and speech. Artist and scholar Fabiola Hanna believes polyvocality necessitates an active and engaged listener, which then produces our memories. Machines have become the listeners to our sonic landscapes as well as capturers, surveyors, and documents of our utterances.
The past few years may have been a remarkable advancement in voice tech with companies such as Amazon and Sanas AI, a voice recognition platform that allows a user to apply a vocal filter onto any human voice, with a discernible accent, that transforms the speech into Standard American English. Yet their hopes for accent elimination and voice mimicry foreshadow a future of design without justice and software development sans cultural and societal considerations, something I work through in my artwork in progress, The Cyborg’s Prosody (2022-present).
The Cyborg’s Prosody is an interactive web-based artwork (optimized for mobile) that requires participants to read five vignettes that increasingly incorporate Tagalog words and phrases that must be repeated by the player. The work serves as a type of parody, as an “accent induction school” — providing a decolonial method of exploring how language and accents are learned and preserved. The work is a response to the creation of accent reduction schools and coaches in the Philippines. Originally, the work was meant to be a satire and parody of these types of services, but shifted into a docu-poetic work of my mother’s immigration story and learning and becoming fluent in American English.
Even though English is a compulsory language in the Philippines, it is a language learned within the parameters of an educational institution and not common speech outside of schools and businesses. From the call center agents hired at Vox Elite, a BPO company based in the Philippines, to a Filipino immigrant navigating her way through a new environment, the embodiment of language became apparent throughout the stages of research and the creative interventions of the past few years.
In Fall 2022, I gave an artist talk about The Cyborg’s Prosody to a room of predominantly older, white, cisgender male engineers and computer scientists. Apparently, my work caused a stir in one of the conversations between a small group of attendees. A couple of the engineers chose to not address me directly, but I overheard a debate between guests with one of the engineers asking, “What is her project supposed to teach me about prosody? What does mimicking her mom teach me?” He became offended by the prospect of a work that de-centered his language, accent, and what was most familiar to him.The Cyborg’s Prosody is a reversal of what is perceived as a foreign accented voice in the United States into a performance for both the cyborg and the player. I introduce the term western vocal drag to convey the caricature of gender through drag performance, which is apropos and akin to the vocal affect many non-western speakers effectuate in their speech.
The concept of western vocal drag became a way for me to understand and contemplate the ways that language becomes performative through its embodiment. Whether it is learning American vernacular to the complex tenses that give meaning to speech acts, there is always a failure or queering of language when a particular affect and accent is emphasized in one’s speech. The delivery of speech acts is contingent upon setting, cultural context, and whether or not there is a type of transaction occurring between the speaker and listener. In terms of enhancement of speech and accent to conform to a dominant language in the workplace and in relation to global linguistic capitalism, scholar Vijay A. Ramjattan states in that there is no such thing as accent elimination or even reduction. Rather, an accent is modified. The stakes are high when taking into consideration the marketing and branding of software such as Sanas AI that proposes an erasure of non-dominant foreign accented voices.
The biggest fear related to the use of artificial intelligence within voice recognition and speech technologies is the return to a Standard American English (and accent) preferred by a general public that ceases to address, acknowledge, and care about linguistic diversity and inclusion. The technology itself has been marketed as a way for corporations and the BPO companies they hire to mind the mental health of the call center agents subjected to racism and xenophobia just by the mere sound of their voice and accent. The challenge, moving forward, is reversing the need to serve the western world.
A transorality or vocality presents itself when thinking about scholar April Baker-Bell’s work Black Linguistic Consciousness. When Black youth are taught and required to speak with what is considered Standard American English, this presents a type of disciplining that perpetuates raciolinguistic ideologies of what is acceptable speech. Baker-Bell focuses on an antiracist linguistic pedagogy where Black youth are encouraged to express themselves as a shift towards understanding linguistic bias. Deeply inspired by her scholarship, I started to wonder about the process for working on how to begin framing language learning in terms of a multi-consciousness that includes cultural context and affect as a way to bridge gaps in understanding.
Or, let’s re-think this concept or idea that a bad version of English exists. As Cathy Park Hong brilliantly states, “Bad English is my heritage…To other English is to make audible the imperial power sewn into the language, to slit English open so its dark histories slide out.” It is necessary for us all to reconfigure our perceptions of how we listen and communicate that perpetuates seeking familiarity and agreement, but encourages respecting and honoring our differences.
—
Featured Image: Still from artist’s mock-up of The Cyborg’s Prosody(2022-present), copyright Dorothy R. Santos
—
Dorothy R. Santos, Ph.D. (she/they) is a Filipino American storyteller, poet, artist, and scholar whose academic and research interests include feminist media histories, critical medical anthropology, computational media, technology, race, and ethics. She has her Ph.D. in Film and Digital Media with a designated emphasis in Computational Media from the University of California, Santa Cruz and was a Eugene V. Cota-Robles fellow. She received her Master’s degree in Visual and Critical Studies at the California College of the Arts and holds Bachelor’s degrees in Philosophy and Psychology from the University of San Francisco. Her work has been exhibited at Ars Electronica, Rewire Festival, Fort Mason Center for Arts & Culture, Yerba Buena Center for the Arts, and the GLBT Historical Society.
Her writing appears in art21, Art in America, Ars Technica, Hyperallergic, Rhizome, Slate, and Vice Motherboard. Her essay “Materiality to Machines: Manufacturing the Organic and Hypotheses for Future Imaginings,” was published in The Routledge Companion to Biology in Art and Architecture. She is a co-founder of REFRESH, a politically-engaged art and curatorial collective and serves as a member of the Board of Directors for the Processing Foundation. In 2022, she received the Mozilla Creative Media Award for her interactive, docu-poetics work The Cyborg’s Prosody (2022). She serves as an advisory board member for POWRPLNT, slash arts, and House of Alegria.
—
REWIND! . . .If you liked this post, you may also dig:
Your Voice is (Not) Your Passport—Michelle Pfeifer
“Hey Google, Talk Like Issa”: Black Voiced Digital Assistants and the Reshaping of Racial Labor–Golden Owens
Beyond the Every Day: Vocal Potential in AI Mediated Communication –Amina Abbas-Nazari
Voice as Ecology: Voice Donation, Materiality, Identity–Steph Ceraso
The Sound of What Becomes Possible: Language Politics and Jesse Chun’s 술래 SULLAE (2020)—Casey Mecija
Look Who’s Talking, Y’all: Dr. Phil, Vocal Accent and the Politics of Sounding White–Christie Zwahlen
Listening to Modern Family’s Accent–Inés Casillas and Sebastian Ferrada
Share this:
SO! Amplifies: Immigrants Wake America Podcast and the Work of Engaged Digital Humanities
SO! Amplifies. . .a highly-curated, rolling mini-post series by which we editors hip you to cultural makers and organizations doing work we really really dig. You’re welcome!
—
Conceptualized at a time of rampant increase in anti-immigrant violence, Immigrants Wake America is a creative response to the growing bias and violence against immigrant women in the U.S., as seen in the Atlanta shootings, the rise in hate crimes since the onset of Covid-19, and the US-Mexico border crisis. We believe that storytelling allows us to find similarities and differences between ourselves and others, offering a humanizing counterpart to harmful media narratives. The podcast creates a living archive of stories not yet heard, serving as an audio intervention into how immigrant women’s (hi)stories are narrated and passed on.
Immigrants Wake America is a public humanities, community-engaged project of digital storytelling through podcasts, in partnership with the Tenement Museum in New York. It features storytellers who share their family stories about migration and the centrality of women in their life histories. These storytellers have submitted stories to the Tenement Museum’s digital archive Your Story, Our Story (YSOS),
Founded in 1988, the Tenement Museum, focuses on immigration and immigrants to “foster a society that embraces and values the role of immigration in the evolving American identity.” YSOS cofounded by Annie Polland and Kathryn Lloyd, is a digital archive that houses stories associated with immigration, migration, and cultural identity. Some of the storytellers are first generation immigrants, while others are descendants of immigrants, born and raised in the US; their great-grandparents or grandparents migrated to the US ages ago. Through YSOS, the Tenement Museum invites people across the country to share their stories in the online digital storytelling exhibit. Each story reveals one individual’s experience. Together, the stories help us see how the unique histories shape the nation, and the patterns that bind us together.
Through exploring and curating stories from Your Story Our Story, we facilitate conversations that supplement and expand it. This makes possible the conception of an archive that is both dynamic and collaborative. Such an archive resists the colonization and appropriation of lives and narratives of our storytellers. We navigate through the ethical conundrums that one might structurally and personally face in this collaborative endeavor. In our engagement with the archives at the Tenement Museum, we believe that our podcasting project really opens up the possibilities for an expansion of the archive.
We released our first episode, the Introductory Episode on January 15th, 2022, and have since been consistently releasing one episode per month.
While our podcast does not claim to retrieve or lay out these microhistories in their entirety, at an early stage of its development, we came to realize the potential that the form of the podcast itself offers for a different kind of storytelling. In our podcast, we treat stories as primary documents instead of marginalia. Michelle Caswell (2014) uses the term “symbolic annihilation” to describe the absence or misrepresentation of marginalized communities in archives. She advocates the powerful forces of community archives in countering “symbolic annihilation.” In thinking about archives in The Archaeology of Knowledge, Michel Foucault is concerned with “the density of discursive practices” wherein he observes “systems that establish statements as events and things (145)” This system of statements (as events or things) is what contributes to the law of what can be said. Processes of digital communal archiving such as those done by South Asian American Digital Archive (SAADA) or the Tenement Museum attempt to extend or expand the systematic possibility of events and things. Caswell and her colleagues have demonstrated the importance and success of the SAADA project. They have also pointed to the impossibility of representation in a traditional archive which is built on violence committed on colonized and enslaved bodies, also eloquently pointed out by Saidiya Hartman’s scholarship.
Through our experience we’ve learnt that podcasts can serve as a transgressive-dynamic expansion of digital archiving, given their unique ability to cut across racial and gendered lines of preconceived sonic notions and their potential to expand the current techniques and media of digital archiving. We map this formal potential of the podcast in the way it intersects with digital archiving in the following ways:
First, narratorial voice.
We wanted our project to act as an intervention in the way in which immigrant women’s (hi)stories are consumed and passed on. We wanted to provide counter narratives. It was essential that the storytellers share their stories in their own voices, literally! The audio medium allows us to produce a space for listening to voices that are otherwise marginalized and/or demonized.
–Le Li and Shruti Jain
Among the several unique and inspiring stories of resilience that the Tenement Museum houses, one such is a story by an immigrant case manager at the American Civic Association in Binghamton, Goretti Mugambwa. The museum and our podcast make it possible for her story to be narrated by herself in her voice. With her experience of working with the refugee and immigrant community she also does not just remain an individual voice, but acts to further a collective assertion.
Next, sonic variations.
Our storytellers’ voices are not just “characteristics” of the story but are an essential part of the story itself. We believe that each immigrant and their descendent brings to the story their unique tonal texture. This diversity destabilizes what immigrants and their descendants are expected to sound like. The sounds we add in the editing process are minimal. We try not to impose emotional cues and responses upon our listeners.
–Shruti Jain and Le Li
The multiplicity of voices in our podcast–and therefore in the archive–are not just “characteristics” of the aural storytelling or listening process, but are as much an essential part of the story itself. In line with what The Sonic Color Line reminds us, our work also finds that, “sound frequently appears to be visuality’s doppelgänger in U.S. racial history” (Stoever 4). This leads to the coding of race as not just visual but aural too. We want to clarify that the white constructed ideas of how people of color must sound flatten out the complexities in how people within and across communities do sound. At the same time, these notions of white sonic normativity also create a strong sense of what one must or must not sound like in order to succeed in the racial capitalist world order. The storytellers of our podcast and we ourselves are of diverse backgrounds. This, for us, is a way to demonstrate the “complex range of sounds actually produced by people of color” (Stoever 43). As Nancy Morales argues in “Óyeme Voz: U.S. Latin@ & Immigrant Communities Re-Sound Citizenship and Belonging,” the sound of ‘everyday voices’ mobilized against—and remarking on—the nation-state’s attempts to mark immigrant communities as vulnerable exerts an impactful and profoundly material agency.” With its conversational and collaborative format, our podcast serves as a dynamic medium to represent (his)stories that complicate generic conventions in critical ways.
Then, collaboration.
We have also been personally deeply impacted by the process of working on this podcast. We have made lasting bonds with our colleagues and storytellers alike. The storytellers of our podcast act not just as guests, but as collaborators and stakeholders. Instead of interpreting the stories in our own way and retelling the stories, we collaborate with the storytellers, and facilitate the unfolding of hidden stories by the storytellers. Dr. Lisa Yun, Professor of English at Binghamton University, and Kathryn Lloyd, Senior Director of Programs, Tenement Museum, have been advisors and the executive producers of the podcast. Together with Lloyd and Yun, we built a project on the ethos of collaboration.
The editing process of IWA too, is different. Rather than making individual editorial decisions, we engage the storytellers directly in post-production. After finishing a first edit of an episode collaboratively between ourselves, we then send it to the storytellers for their feedback and approval before releasing it. Sometimes, the storytellers do suggest changes. Based on their feedback, we re-edit the episode and eventually release it after the storytellers approval. We have also innovated methods of community editing, where we edit in groups of as large as 15 people.
Finally, accessibility.
The podcast medium makes Immigrants Wake America an ideal project for the public humanities. As opposed to lengthier podcasts, each episode of our podcast is edited down to 15-20 minutes. These can be used by educators as an in-class resource to generate discussion and activities. Community listeners could tune in during lunch breaks, get-togethers, cooking, driving or doing chores. Our episodes can also serve as conversation starters and help facilitate affective bonds among immigrants and non-immigrants alike.
The final episode of our first season, “Finding Our Grandmother in the Records,” aired just last week, and a second season is in the works.
As a way to expand this project, our second season will feature storytellers from our local community in addition to Your Story, Our Story. We plan to have units within our project dedicated to translation, recording and editing, and creating teaching resources. We aim for meaningful and engaged conversations and try to blur the supposed boundaries between the university and the community. Join us!
—
The first season of Immigrants Wake America was sponsored through the Institute for Advanced Studies in the Humanities at Binghamton University and a Public Humanities Grant from Humanities New York. Dr. Lisa Yun, Professor of English at Binghamton University, and Kathryn Lloyd, Senior Director of Programs, Tenement Museum, have been our advisors and the executive producers of the podcast. IWA is available on major streaming platforms such as Spotify, Google Podcasts, Apple Podcasts, Amazon Music, Soundcloud, and Audible.
—
Le Li and Shruti Jain are pursuing their PhDs at Binghamton University in the Translation Research and Instruction Program and the English Department respectively. They were Humanities New York Public Humanities fellows (2021-22) and graduate fellows of the Institute for Advanced Studies in the Humanities (IASH) at Binghamton University (2021-22). Through their podcast project and their work with digital community archives, Le and Shruti are currently working on exploring intersection between podcasts and digital archiving. They try to capitalize on the unique ability that the form of the podcast offers to cut across racial and gendered lines of preconceived sonic notions, which makes possible the conception of an archive that can be both dynamic and collaborative. Le’s research interests include translation studies, cultural studies, diaspora studies, and public humanities. Shruti’s PhD focuses on the Enlightenment, British Empire and the relationalities between race and caste formations.
—
REWIND!…If you liked this post, you may also dig all this good stuff about sound studies pedagogy! Good luck with Fall semester, folks!:
“Heavy Airplay, All Day with No Chorus”: Classroom Sonic Consciousness in the Playlist Project—Todd Craig
SO! Podcast #79: Behind the Podcast: deconstructing scenes from AFRI0550, African American Health Activism – Nic John Ramos and Laura Garbes
The Sounds of Anti-Anti-Essentialism: Listening to Black Consciousness in the Classroom- Carter Mathes
Making His Story Their Story: Teaching Hamilton at a Minority-serving Institution–Erika Gisela Abad
Teaching Soundwalks in a Course on Gentrification, Black Music, and Corporate America–Rami Toubia Stucky
Deejaying her Listening: Learning through Life Stories of Human Rights Violations– Emmanuelle Sonntag and Bronwen Low
Audio Culture Studies: Scaffolding a Sequence of Assignments– Jentery Sayers
Deep Listening as Philogynoir: Playlists, Black Girl Idiom, and Love–Shakira Holt
“Toward A Civically Engaged Sound Studies, or ReSounding Binghamton”–Jennifer Lynn Stoever
Listening to #Occupy in the Classroom–D. Travers Scott
SO! Podcast #71: Everyday Sounds of Resilience and Being: Black Joy at School–Walter Gershon
Sounding Out! Podcast #13: Sounding Shakespeare in S(e)oul– Brooke Carlson
A Listening Mind: Sound Learning in a Literature Classroom–Nicole Brittingham Furlonge
My Voice, or On Not Staying Quiet–Kaitlyn Liu
(Re)Locating Soundscapes of Schooling: Learning to Listen to Children’s Lifeworlds–Cassie J. Brownell
If You Can Hear My Voice: A Beginner’s Guide to Teaching–Caroline Pinkston
Mukbang Cooks, Chews, and Heals – David Lee
SO! Podcast #80: Refugee Realities Miniseries–Steph Ceraso
Share this:
Current Top Posts
April 2021
January 2020
November 2019
October 2019
August 2019
April 2014
November 2017
March 2015
December 2017
October 2015
Sounding Out! 
Recent Comments