The Cyborg’s Prosody, or Speech AI and the Displacement of Feeling
In summer 2021, sound artist, engineer, musician, and educator Johann Diedrick convened a panel at the intersection of racial bias, listening, and AI technology at Pioneerworks in Brooklyn, NY. Diedrick, 2021 Mozilla Creative Media award recipient and creator of such works as Dark Matters, is currently working on identifying the origins of racial bias in voice interface systems. Dark Matters, according to Squeaky Wheel, “exposes the absence of Black speech in the datasets used to train voice interface systems in consumer artificial intelligence products such as Alexa and Siri. Utilizing 3D modeling, sound, and storytelling, the project challenges our communities to grapple with racism and inequity through speech and the spoken word, and how AI systems underserve Black communities.” And now, he’s working with SO! as guest editor for this series (along with ed-in-chief JS!). It kicked off with Amina Abbas-Nazari’s post, helping us to understand how Speech AI systems operate from a very limiting set of assumptions about the human voice. Then, Golden Owens took a deep historical dive into the racialized sound of servitude in America and how this impacts Intelligent Virtual Assistants. Last week, Michelle Pfeifer explored how some nations are attempting to draw sonic borders, despite the fact that voices are not passports. Today, Dorothy R. Santos wraps up the series with a meditation on what we lose due to the intensified surveilling, tracking, and modulation of our voices. [To read the full series, click here] –JS
—
In 2010, science fiction writer Charles Yu wrote a story titled “Standard Loneliness Package,” where emotions are outsourced to another human being. While Yu’s story is a literal depiction, albeit fictitious, of what might be entailed and the considerations that need to be made of emotional labor, it was published a year prior to Apple introducing Siri as its official voice assistant for the iPhone. Humans are not meant to be viewed as a type of technology, yet capitalist and neoliberal logics continue to turn to technology as a solution to erase or filter what is least desirable even if that means the literal modification of voice, accent, and language. What do these actions do to the body at risk of severe fragmentation and compartmentalization?
I weep.
I wail.
I gnash my teeth.
Underneath it all, I am smiling. I am giggling.
I am at a funeral. My client’s heart aches, and inside of it is my heart, not aching, the opposite of aching—doing that, whatever it is.
Charles Yu, “Standard Loneliness Package,” Lightspeed: Science Fiction & Fantasy, November 2010
Yu sets the scene by providing specific examples of feelings of pain and loss that might be handed off to an agent who absorbs the feelings. He shows us, in one way, what a world might look and feel like if we were to go to the extreme of eradicating and off loading our most vulnerable moments to an agent or technician meant to take on this labor. Although written well over a decade ago, its prescient take on the future of feelings wasn’t too far off from where we find ourselves in 2023. How does the voice play into these connections between Yu’s story and what we’re facing in the technological age of voice recognition, speech synthesis, and assistive technologies? How might we re-imagine having the choice to displace our burdens onto another being or entity? Taking a cue from Yu’s story, technologies are being created that pull at the heartstrings of our memories and nostalgia. Yet what happens when we are thrust into a perpetual state of grieving and loss?
Humans are made to forget. Unlike a computer, we are fed information required for our survival. When it comes to language and expression, it is often a stochastic process of figuring out for whom we speak and who is on the receiving end of our communication and speech. Artist and scholar Fabiola Hanna believes polyvocality necessitates an active and engaged listener, which then produces our memories. Machines have become the listeners to our sonic landscapes as well as capturers, surveyors, and documents of our utterances.
The past few years may have been a remarkable advancement in voice tech with companies such as Amazon and Sanas AI, a voice recognition platform that allows a user to apply a vocal filter onto any human voice, with a discernible accent, that transforms the speech into Standard American English. Yet their hopes for accent elimination and voice mimicry foreshadow a future of design without justice and software development sans cultural and societal considerations, something I work through in my artwork in progress, The Cyborg’s Prosody (2022-present).
The Cyborg’s Prosody is an interactive web-based artwork (optimized for mobile) that requires participants to read five vignettes that increasingly incorporate Tagalog words and phrases that must be repeated by the player. The work serves as a type of parody, as an “accent induction school” — providing a decolonial method of exploring how language and accents are learned and preserved. The work is a response to the creation of accent reduction schools and coaches in the Philippines. Originally, the work was meant to be a satire and parody of these types of services, but shifted into a docu-poetic work of my mother’s immigration story and learning and becoming fluent in American English.
Even though English is a compulsory language in the Philippines, it is a language learned within the parameters of an educational institution and not common speech outside of schools and businesses. From the call center agents hired at Vox Elite, a BPO company based in the Philippines, to a Filipino immigrant navigating her way through a new environment, the embodiment of language became apparent throughout the stages of research and the creative interventions of the past few years.
In Fall 2022, I gave an artist talk about The Cyborg’s Prosody to a room of predominantly older, white, cisgender male engineers and computer scientists. Apparently, my work caused a stir in one of the conversations between a small group of attendees. A couple of the engineers chose to not address me directly, but I overheard a debate between guests with one of the engineers asking, “What is her project supposed to teach me about prosody? What does mimicking her mom teach me?” He became offended by the prospect of a work that de-centered his language, accent, and what was most familiar to him.The Cyborg’s Prosody is a reversal of what is perceived as a foreign accented voice in the United States into a performance for both the cyborg and the player. I introduce the term western vocal drag to convey the caricature of gender through drag performance, which is apropos and akin to the vocal affect many non-western speakers effectuate in their speech.
The concept of western vocal drag became a way for me to understand and contemplate the ways that language becomes performative through its embodiment. Whether it is learning American vernacular to the complex tenses that give meaning to speech acts, there is always a failure or queering of language when a particular affect and accent is emphasized in one’s speech. The delivery of speech acts is contingent upon setting, cultural context, and whether or not there is a type of transaction occurring between the speaker and listener. In terms of enhancement of speech and accent to conform to a dominant language in the workplace and in relation to global linguistic capitalism, scholar Vijay A. Ramjattan states in that there is no such thing as accent elimination or even reduction. Rather, an accent is modified. The stakes are high when taking into consideration the marketing and branding of software such as Sanas AI that proposes an erasure of non-dominant foreign accented voices.
The biggest fear related to the use of artificial intelligence within voice recognition and speech technologies is the return to a Standard American English (and accent) preferred by a general public that ceases to address, acknowledge, and care about linguistic diversity and inclusion. The technology itself has been marketed as a way for corporations and the BPO companies they hire to mind the mental health of the call center agents subjected to racism and xenophobia just by the mere sound of their voice and accent. The challenge, moving forward, is reversing the need to serve the western world.
A transorality or vocality presents itself when thinking about scholar April Baker-Bell’s work Black Linguistic Consciousness. When Black youth are taught and required to speak with what is considered Standard American English, this presents a type of disciplining that perpetuates raciolinguistic ideologies of what is acceptable speech. Baker-Bell focuses on an antiracist linguistic pedagogy where Black youth are encouraged to express themselves as a shift towards understanding linguistic bias. Deeply inspired by her scholarship, I started to wonder about the process for working on how to begin framing language learning in terms of a multi-consciousness that includes cultural context and affect as a way to bridge gaps in understanding.
Or, let’s re-think this concept or idea that a bad version of English exists. As Cathy Park Hong brilliantly states, “Bad English is my heritage…To other English is to make audible the imperial power sewn into the language, to slit English open so its dark histories slide out.” It is necessary for us all to reconfigure our perceptions of how we listen and communicate that perpetuates seeking familiarity and agreement, but encourages respecting and honoring our differences.
—
Featured Image: Still from artist’s mock-up of The Cyborg’s Prosody(2022-present), copyright Dorothy R. Santos
—
Dorothy R. Santos, Ph.D. (she/they) is a Filipino American storyteller, poet, artist, and scholar whose academic and research interests include feminist media histories, critical medical anthropology, computational media, technology, race, and ethics. She has her Ph.D. in Film and Digital Media with a designated emphasis in Computational Media from the University of California, Santa Cruz and was a Eugene V. Cota-Robles fellow. She received her Master’s degree in Visual and Critical Studies at the California College of the Arts and holds Bachelor’s degrees in Philosophy and Psychology from the University of San Francisco. Her work has been exhibited at Ars Electronica, Rewire Festival, Fort Mason Center for Arts & Culture, Yerba Buena Center for the Arts, and the GLBT Historical Society.
Her writing appears in art21, Art in America, Ars Technica, Hyperallergic, Rhizome, Slate, and Vice Motherboard. Her essay “Materiality to Machines: Manufacturing the Organic and Hypotheses for Future Imaginings,” was published in The Routledge Companion to Biology in Art and Architecture. She is a co-founder of REFRESH, a politically-engaged art and curatorial collective and serves as a member of the Board of Directors for the Processing Foundation. In 2022, she received the Mozilla Creative Media Award for her interactive, docu-poetics work The Cyborg’s Prosody (2022). She serves as an advisory board member for POWRPLNT, slash arts, and House of Alegria.
—
REWIND! . . .If you liked this post, you may also dig:
Your Voice is (Not) Your Passport—Michelle Pfeifer
“Hey Google, Talk Like Issa”: Black Voiced Digital Assistants and the Reshaping of Racial Labor–Golden Owens
Beyond the Every Day: Vocal Potential in AI Mediated Communication –Amina Abbas-Nazari
Voice as Ecology: Voice Donation, Materiality, Identity–Steph Ceraso
The Sound of What Becomes Possible: Language Politics and Jesse Chun’s 술래 SULLAE (2020)—Casey Mecija
Look Who’s Talking, Y’all: Dr. Phil, Vocal Accent and the Politics of Sounding White–Christie Zwahlen
Listening to Modern Family’s Accent–Inés Casillas and Sebastian Ferrada
Share this:
SO! Amplifies: The Electric Golem (Trevor Pinch and James Spitznagel)
—
On March 24th, 2019 the record release party for The Electric Golem’s 6th CD Golemology was held at the Loft in Ithaca, New York. The Electric Golem is an avant-garde synthesizer duo featuring Trevor Pinch and James Spitznagel, that has been in existence for about ten years.
Trevor Pinch is a local sound artist and professor at Cornell University. He is an STS (Science and Technology Studies) and Sound Studies scholar. As a key thinker of STS, Trevor is the coproducer of theories about Sociology of Scientific Knowledge, Social Construction of Technology (SCOT), and the role of users in technological history and innovation. However, Trevor’s interest in dates back much farther; he built his first modular synthesizer when he was a physics student in London in the 1970s.
The other half of The Electric Golem, James Spitznagel, is a multi-media artist who uses the iPad as a musical instrument and to create digital paintings. While he has played many roles in the music and culture industries—guitarist in a rock band, record store owner, art gallery and guitar shop investor, and even business manager for the Andy Warhol Museum—he moved to Ithaca to focus on producing abstract art: digital paintings and experimental, improvisational music. Being an energetic and enthusiastic person who has unrestrained fantasies, James finds that everything around him can be his inspiration.
Pinch and Spitznagel formed the group after Spitznagel read Analog Days: The Invention and Impact of the Moog Synthesizer (by Trevor Pinch and Frank Trocco) and realized Pinch also lived in Ithaca. Spitznagel simply looked his name up in the phone book and called him up: “I go, ‘is this Trevor Pinch?’ He said, ‘yes.’ I said, ‘well, you don’t know me, but I just read your book and I love it.’” And then they got together for a beer and have been best friends and collaborators ever since. Once Spitznagel heard about Pinch’s homemade synthesizer, he asked Trevor to try to make something together and it turned out to be a fascinating mixture of analog–Trevor’s synth, Moog Prodigy, and a Minimoog–and James’s digital instruments.
Building from this first moment of discovery, The Electric Golem’s music is electronic, experimental, and totally improvised. Typically, the pieces of music last twenty minutes to half an hour and expresses their interaction with the machines and with each other in the studio. James is much more controlling of the tone and rhythm, and patches the sound as he goes along, whereas Trevor is much more about making spontaneous weird sounds. They complement each other and the creation process is usually by random and spontaneous, as Spitznagel describes: “I didn’t tell Trevor what to do or what to play, but I said, here’s the piece of music I’ve written. He just instinctively knew what add to it.” Reciprocally, “he might just play something that I go, oh, I can weave in and out of the ambient sound he’s putting there.”
Trevor Pinch, Electric Golem at Elmira College, 2012
For the duo, the process of producing music becomes a shared experience with their listeners. The music is ever changing and evolving. In addition, unexpected drama adds vitality to the palette. “The iPad might freeze up or synthesizer might break somehow,” Spitznagel notes, “that’s happened to us, but we carry on. Like Trevor looks at me and says, it’s not working there. Or, I look at him and go, I have to reboot my computer, it’s not working. But, those times actually inspire us to try new things and go beyond what we are doing.” James explained. Their inspiration comes from the unknown, which just emerges from their practice. “Generally, this sort of music is completely unique to Electric Golem.” Trevor concluded.
The name “Electric Golem” comes from a series of books with Golem in the titles that Trevor collaborated on with his mentor Harry Collins. “The golem is a creature of Jewish mythology,” Pinch and Collins wrote in The Golem, What You Should Know about Science, “it is a humanoid made by man with clay and water, with incantations and spells. It is powerful, it grows a little more powerful every day. It will follow orders, do your work, and protect you from the ever threatening enemy. But it is clumsy and dangerous. Without control, a golem may destroy its masters with its flailing vigour” (1). Noting Trevor’s association with the concept of the Golem, Spitznagel added the “Electric” twist not just as a metaphor for their sound but also because “it’s kind of like a retro name.” The Electric Golem mushroomed from there, and in the past decade they have had many invitations and bookings to play out, receiving the first recording contract from the Ricochet Dream label, and have played with a bunch of notable musicians, such as Malcolm Cecil of Tonto’s Expanding Head Band, Simeon of Silver Apples, and “Future Man” (aka Roy Wooten), and they haven’t stopped there.
According to Pinch, the key feature of The Electric Golem’s music is its ability to encompass different moods. “I think Electric Golem has become good at one thing: its changing and transitioning from one sort of mood of music to another. And we have become quite good at those transitions. I think people would say that’s what they kind of like about us.” These sorts of slow transitions construct a unique texture of sound that can be quite cinematic, so much so that in 2012, the Electric Golem performed the accompaniment to the silent movie A Trip to the Moon, a special Cornell cinema event. Overall, as improvised experimental music, it is sometimes challenging to listen to, with no regular rhythm or reliable melody. Trevor produces warm, rich drones from the analog side that contrast with the sharper digital rhythms that James programs. In short, the Electric Golem varies between these two affects but the music goes far beyond the representation of emotional states; sometimes it conjures up the feeling of the vastness of space and time.
Experimental music, is a collaboration and negotiation process between instruments and their users. No matter if analog or digital, instruments have autonomy; they are non-human actors with their own agency to some extent. As Trevor Pinch intimates, “I understand the general sort of sound that can be produced, but the particular details of how it will work out, you don’t really know, that’s much more spontaneous, you have to react to that.” Instruments can often be uncontrollable–making their own sounds—so that Electric Golem must respond in kind. “So, it’s sort of like higher level meta-control versus actually doing what you’re doing in response to the instrument that combines together,” Trevor describes, “which I think is the secret to controlling these sorts of instruments.” It is incredible that Pinch and Spitznagel know each other so well—and each know their instruments so well–that they can improvise for long periods with no trouble. Trevor says: “Follow the use of these instruments! Follow the instruments! They are not essentialized. They are just stabilized temporarily.”
On the whole, The Electric Golem shows an artistic form which breaks the traditional paradigm, deconstructs and then reconstructs it, seeking to free sound from the instruments. Their music is beyond pure melody and rhythm, beyond the expression of existence, expressing more of an aesthetic state of transcendence. They challenge what music is, and what musical instruments are; they challenge divisions between the identities of engineer and musician. Electric Golem’s music co-constructs art and technology and binds them together; art, for them, is a mode of presenting technology, and vice versa, technology is a pathway through which art can flourish.
My favorite Electric Golem piece is called “Heart of the Golem.” What is the heart of the Golem? According to Pinch, “It is a mystery, a process of unfolding and discovery. It is somewhere where analog and digital sound meet, and an improvisation.” What the magic is remains unknown and unlimited, just like the future of the Electric Golem.
—
Featured Image: Courtesy of The Electric Golem
—
Qiushi Xu is a PhD candidate in the subject of Philosophy of Science and Technology in Tsinghua University, Beijing and in a joint PhD program in the Department of Science and Technology Studies in Cornell University, working with Prof. Trevor Pinch. Her research areas are Sound Studies, STS, Cultural Studies and Gender Studies. Her current research focuses on the sociology of piano sound and the negotiation and construction of piano sound in the recording studio (PhD dissertation), gender issues in recording industry, experimental music, auscultation and sound therapy. She holds an MA in Cultural and Creative Industries from King’s College London; a BA in Recording Arts and a BA in Journalism and Communication from the University of China, Beijing. She is also an amateur pianist, writer, and traditional Chinese painter. As a multiculturalist, she is am fascinated by different forms of art and culture in different cultural contexts.
Share this:
Current Top Posts
April 2021
January 2020
November 2019
October 2019
August 2019
April 2014
November 2017
March 2015
December 2017
October 2015
Sounding Out! 
Recent Comments