Beyond the Every Day: Vocal Potential in AI Mediated Communication


In summer 2021, sound artist, engineer, musician, and educator Johann Diedrick convened a panel at the intersection of racial bias, listening, and AI technology at Pioneerworks in Brooklyn, NY. Diedrick, 2021 Mozilla Creative Media award recipient and creator of such works as Dark Matters, is currently working on identifying the origins of racial bias in voice interface systems. Dark Matters, according to Squeaky Wheel, “exposes the absence of Black speech in the datasets used to train voice interface systems in consumer artificial intelligence products such as Alexa and Siri. Utilizing 3D modeling, sound, and storytelling, the project challenges our communities to grapple with racism and inequity through speech and the spoken word, and how AI systems underserve Black communities.” And now, he’s working with SO! as guest editor for this series for Sounding Out! (along with ed-in-chief JS!). It starts today, with Amina Abbas-Nazari, helping us to understand how Speech AI systems operate from a very limiting set of assumptions about the human voice– are we training it, or is it actually training us?
—
DOI: https://doi.org/10.59350/hdkc2-hrd04
Hi, good morning. I’m calling in from Bangalore, India.” I’m talking on speakerphone to a man with an obvious Indian accent. He pauses. “Now I have enabled the accent translation,” he says. It’s the same person, but he sounds completely different: loud and slightly nasal, impossible to distinguish from the accents of my friends in Brooklyn.
The AI startup erasing call center worker accents: is it fighting bias – or perpetuating it? (Wilfred Chan, 24 August 2022)
This telephone interaction was recounted in The Guardian reporting on a Silicon Valley tech start-up called Sanas. The company provides AI enabled technology for real-time voice modification for call centre workers voices to sound more “Western”. The company describes this venture as a solution to improve communication between typically American callers and call centre workers, who might be based in countries such as Philippines and India. Meanwhile, research has found that major companies’ AI interactive speech systems exhibit considerable racial imbalance when trying to recognise Black voices compared to white speakers. As a result, in the hopes of being better heard and understood, Google smart speaker users with regional or ethnic American accents relay that they find themselves contorting their mouths to imitate Midwestern American accents.
These instances describe racial biases present in voice interactions with AI enabled and mediated communication systems, whereby sounding ‘Western’ entitles one to more efficient communication, better usability, or increased access to services. This is not a problem specific to AI though. Linguistics researcher John Baugh, writing in 2002, describes how linguistic profiling is known to have resulted in housing being denied to people of colour in the US via telephone interactions. Jennifer Stoever‘s The Sonic Color Line (2016) presents a cultural and political history of the racialized body and how it both informed and was informed by emergent sound technologies. AI mediated communication repeats and reinforces biases that pre-exist the technology itself, but also helping it become even more widely pervasive.

Mozilla’s commendable Common Voice project aims to ‘teach machines how real people speak’ by building an open source, multi-language dataset of voices to improve usability for non-Western speaking or sounding voices. But singer and musicologist, Nina Sun Eidsheim describes how ’a specific voice’s sonic potentiality [in] its execution can exceed imagination’ (7), and voices as having ‘an infinity of unrealised manifestations’ (8) in The Race of Sound (2019). Eidsheim’s sentiments describe a vocal potential, through musicality, that exists beyond ideas of accents and dialects, and vocal markers of categorised identity. As a practicing vocal performer, I recognise and resonate with Eidsheim’s ideas I have a particular interest in extended and experimental vocality, especially gained through my time singing with Musarc Choir and working with artist Fani Parali. In these instances, I have experienced the pleasurable challenge of being asked to vocalise the mythical, animal, imagined, alien and otherworldly edges of the sonic sphere, to explore complex relations between bodies, ecologies, space and time, illuminated through vocal expression.

Following from Eidsheim, and through my own vocal practice, I believe AI’s prerequisite of voices as “fixed, extractable, and measurable ‘sound object[s]’ located within the body” is over-simplistic and reductive. Voices, within systems of AI, are made to seem only as computable delineations of person, personality and identity, constrained to standardised stereotypes. By highlighting vocal potential, I offer a unique critique of the way voices are currently comprehended in AI recognition systems. When we appreciate the voice beyond the homogenous, we give it authority and autonomy, ultimately leading to a fuller understanding of the voice and its sounding capabilities.
My current PhD research, Speculative Voicing, applies thinking about the voice from a musical perspective to the sound and sounding of voices in artificially intelligent conversational systems. Herby the voice becomes an instrument of the body to explore its sonic materiality, vocal potential and extremities of expression, rather than being comprehended in conjunction to vocal markers of identity aligning to categories of race, gender, age, etc. In turn, this opens space for the voice to be understood as a shapeshifting, morphing and malleable entity, with immense sounding potential beyond what might be considered ordinary or everyday speech. Over the long term this provides discussion of how experimenting with vocal potential may illuminate more diverse perspectives about our sense of self and being in relation to vocal sounding.
Vocal and movement artist Elaine Mitchener exhibits the disillusion of the voice as ‘fixed’ perfectly in her performance of Christian Marclay’s No!, which I attended one hot summer’s evening at the London Contemporary Music Festival in 2022. Marclay’s graphic score uses cut outs from comic book strips to direct the performer to vocalise a myriad of ‘No”s.

Mitchener’s rendering of the piece involved the cooperation and coordination of her entire body, carefully crafting lips, teeth, tongue, muscles and ligaments to construct each iteration of ‘No.’ Each transmutation of Mitchener’s ‘No’s’ came with a distinct meaning, context, and significance, contained within the vocalisation of this one simple syllable. Every utterance explored a new vocal potential, enabled by her body alone. In the context of AI mediated communication, we can see this way of working with the voice renders the idea of the voice as ‘fixed’ as redundant. Mitchener’s vocal potential demonstrates that voices can and do exist beyond AI’s prescribed comprehension of vocal sounding.
In order to further understand how AI transcribes understandings of voice onto notions of identity, and vocal potential, I produced the practice project Polyphonic Embodiment(s) as part of my PhD research, in collaboration with Nestor Pestana, with AI development by Sitraka Rakotoniaina. The AI we created for this project is based upon a speech-to-face recognition AI that aims to be able to tell what your face looks like from the sound of your voice. The prospective impact of this AI is deeply unsettling, as its intended applications are wide-ranging – from entertainment to security, and as previously described AI recognition systems are inherently biased.

This multi-modal form of comprehending voice is also a hot topic of research being conducted by major research institutions including Oxford University and Massachusetts Institute of Technology. We wanted to explore this AI recognition programme in conjunction with an understanding of vocal potential and the voice as a sonic material shaped by the body. As the project title suggests, the work invites people to consider the multi-dimensional nature of voice and vocal identity from an embodied standpoint. Additionally, it calls for contemplation of the relationships between voice and identity, and individuals having multiple or evolving versions of identity. The collaboration with the custom-made AI software creates a feedback loop to reflect on how peoples’ vocal sounding is “seen” by AI, to contest the way voices are currently heard, comprehended and utilised by AI, and indeed the AI industry.
The video documentation for this project shows ‘facial’ images produced by the voice-to-face recognition AI, when activated by my voice, modified with simple DIY voice devices. Each new voice variation, created by each device, produces a different outputted face image. Some images perhaps resemble my face? (e.g. Device #8) some might be considered more masculine? (e.g. Device #10) and some are just disconcerting (e.g. Device #4). The speculative nature of Polyphonic Embodiment(s) is not to suggest that people should modify their voices in interaction with AI communication systems. Rather the simple devices work with bodily architecture and exaggerate its materiality, considering it as a flexible instrument to explore vocal potential. In turn this sheds light on the normative assumptions contained within AI’s readings of voice and its relationships to facial image and identity construction.
Through this artistic, practice-led research I hope to evolve and augment discussion around how the sounding of voices is comprehended by different disciplines of research. Taking a standpoint from music and design practice, I believe this can contest ways of working in the realms of AI mediated communication and shape the ways we understand notions of (vocal) identity: as complex, fluid, malleable, and ultimately not reducible to Western logics of sounding.
—
Featured Image: Still image from Polyphonic Embodiments, courtesy of author.
—
Amina Abbas-Nazari is a practicing speculative designer, researcher, and vocal performer. Amina has researched the voice in conjunction with emerging technology, through practice, since 2008 and is now completing a PhD in the School of Communication at the Royal College of Art, focusing on the sound and sounding of voices in artificially intelligent conversational systems. She has presented her work at the London Design Festival, Design Museum, Barbican Centre, V&A, Milan Furniture Fair, Venice Architecture Biennial, Critical Media Lab, Switzerland, Litost Gallery, Prague and Harvard University, America. She has performed internationally with choirs and regularly collaborates with artists as an experimental vocalist.
—

REWIND! . . .If you liked this post, you may also dig:
What is a Voice?–Alexis Deighton MacIntyre
Voice as Ecology: Voice Donation, Materiality, Identity–-Steph Ceraso
Mr. and Mrs. Talking Machine: The Euphonia, the Phonograph, and the Gendering of Nineteenth Century Mechanical Speech – J. Martin Vest
One Scream is All it Takes: Voice Activated Personal Safety, Audio Surveillance, and Gender Violence—María Edurne Zuazu
Echo and the Chorus of Female Machines—AO Roberts
On Sound and Pleasure: Meditations on the Human Voice– Yvon Bonefant
On Sound and Pleasure: Meditations on the Human Voice
After a rockin’ (and seriously informative) series of podcasts from Leonard J. Paul–a three part “Inside the Game Sound Designer’s Studio”– and a post on sound and black women’s sexual freedom from SO! Regular Regina Bradley, our summer Sound and Pleasure series keeps doin’ it and doin’ it and doin’ it well, this week with a beautiful set of meditations from scholar, artist, performer, and voice activist, Yvon Bonenfant. EVERYBODY SCREAM!!!–-JS, Editor-in-Chief
—
DOI: https://doi.org/10.59350/29wsv-zgd60
1
What I have to say about sound and pleasure can mostly be summed up this way: everyone deserves to take profound pleasure in their body’s sound.
Not only this, everyone deserves to both engage passionately with social sound and negotiate the exchange of social sound on pleasurable terms.
Like other expressive systems, however, these inalienable sonic human rights are mostly ignored, curtailed, or otherwise ‘disciplined and punished’ in the Foucauldian sense by our social systems. So, we are mostly neurotic, or otherwise hung up on, what kinds of sounds we make, where and when. We fetishise sound, particularly virtuosically framed sound, because it is part of a series of sublimated impulses, or we repress it because we think we aren’t supposed to emit it, or we ignore it.
2
In any given human relationship within which all parties can vocalize, the voice is an evident, key relational tool. It is full of gesture and meaning and text and sends rapid-fire, complex, layered, even self-contradictory or oxymoronic messages. It is a truly tangled web, and of course, for those who can use speech, transmits language.
However, I’d like to disentangle our sound from our language for a moment. Indeed, sound is not necessary in order to develop and transmit linguistically carried ideas, information and impulses. It has long been accepted that sign languages are fully developed languages, with intricate grammatical systems, vocabularies, and all of the other features of spoken languages. It is thus not necessary to use sound as a carrier of language. Yet if we have a voice, we almost always use sound to carry our language. And we force deaf people to try to fake having a voice and to fake listening to voices through lip reading and gesturing.
The last twenty years has seen a real boom in speculation and even scientific experiments that theorise why human bodily sound – the most evident aspect of which is our vocal sound – is so important to us. Musicology, biomusicology, evolutionary psychology, neuropsychology, and cultural studies of many kinds have tried to account for this. I have my own favorite reason, one I’ve tried to describe in a number of scholarly articles. This is that sound is much like touch. Like, yet unalike. It reaches and vibrates bodies, but at distance. It voyages through space in other ways, but it evokes haptic responses.
3
Sound isn’t solid, but it takes up space. This is expressed by Stephen Connor within his concept of the vocalic body. When we sound, there is a resonant field of vibration that moves through matter, which behaves according to the laws of physics – it vibrates molecules. This vibratory field leaves us, but is of us, and it voyages through space. Other people hear it. Other people feel it.
I’ve said that sound is like touch. However, one key way that it is not like touch is that it can do this thing. It can leave our bodies and travel away from us. We don’t need to grip it. We don’t need to hold on. And once emanated, it is out of our control.
More than one emanation can co-exist within matter. Their vibrations interact with one another, waves colliding and travelling in similar or different directions, and the vocalic bodies that they represent are morphed, hybridized: they intersect and invent composite bodies.
We hear the resulting harmonies. Historically policed into ‘consonances’ and ‘dissonances’, we have the power to let the negativizing connotations of either of these words go and simply hear the results of the collisions. Voices sounding simultaneously create choreographies of gesture that can be jubilant, depressing, assertive, aggressive, delightful, morose… or many of these simultaneously and in rapid alternation.
The fields of human sound in which we bathe are a continually self-knitting web of sensation. They are full of gestures pregnant with intention, filled with improvisatory spontaneity, success, failure and experimentation. They are filled with a desire to act upon matter, and to reach and engage one another.
4
My Ukrainian-origin mother was ‘loud’, I guess, at least by Anglo-Saxon standards, and her voice was timbrally very rich. And my father was a radio announcer (he disliked being called a DJ immensely, even though he worked in commercial radio and worked on shows that spun discs – he preferred being associated with talking). His voice was also very rich, as well as extremely crafted. It could be pointed and severe: a weapon. He had professional command of its qualities. We were not a quiet family; none of us were vocal wallflowers. But were our soundings pleasure-filled? Certainly, we were allowed to make lots of sound in some circumstances. However, just being allowed to be loud – though it might sometimes be a pleasure – does not necessarily lead to a pleasure-filled dynamic. Weightlifting makes us stronger, but it doesn’t necessarily feel good.
The amount of sound and whether ‘lots’ of it, or heightenings of its qualities – lots of amplitude, or lots of other kinds of distinctness, let’s say things like pitch or emotional timbre – are key variable features of family life in our cultures. Sound takes us directly into the meatiest of interpersonal dynamics – the dynamics of space and gesture, the dynamics of who takes up space with their sound and when. Families are, of course, microcosms of this sonic dynamic, but any group within which we generate relationships and encounters is subject to this dynamic, too. Our very own bodies end up developing what Thomas Csordas might call a ‘somatic mode’ that embodies our experience of these dynamics.
Whether we start from psychodynamic, neuropsychiatric, or even habitus-based models, it’s clear that repressing the expression of bodily sound regulates breathing impulses and other metabolic processes in ways that might become, well, habits.
Let’s put this in other ways.
The classic, Freudian, psychodynamic model of neurosis – as disputed as it is, and with all of its colonial, sexist, homophobic, racist and even abuse-denying overtones – did at least one thing for our understanding of what repressed emotion does. Repressed emotion affects the body.
Today, a popular understanding of this kind of emotional repression from a biophysical perspective might be: the use of the conscious mind to hold back emotional flow, and along with it, the emotional qualities of certain associations, memories, or even the content of the memories themselves.
Repressing this thing we might call emotional flow represses the voice. The literal, physical voice. Now, this kind of repression of the voice can become what Freudians would call unconscious. To allow it out isn’t any longer a choice that can be made, because we’re so used to holding back, that we don’t realize we’re doing it any more.
Somatics have taught us, through the contended practices of the body psychotherapies descended from Wilhelm Reich’s work, or Bonnie Bainbridge Cohen’s Body-Mind Centering, or any numerous other somatic practices – from certain styles of yoga through to Zen meditation and beyond – that emotional flow is at least partly dependent on how we breathe. And neuropsychology and physiology bear this out.
Whatever might ‘cause’ an emotion – and the roots of the causes of emotion are a source of debate – once it gets going, it isn’t just a thought process. Emotion is meaty and full of pumping hormones and breath pattern alterations and gestures and rushes of fluid. Chemicals get released. Chemicals get washed away. Heart rates speed up and slow down. Our breath rises and falls and its patterns change. Digestion patterns speed up or slow down or get interrupted. What happens in the body affects the body. What happens in the body affects the voice. Ever heard that kind of voice that seems hardened against the world? Or that media voice – the voice that is carefully shaped to invoke reason? Maybe these vocalisers can never let go of that sound: maybe it’s the only sound they can do, now. It’s just too habitual to let it change.
So, these habits can become so habitual that we don’t notice them anymore. We might change our breathing in some way to modify our expressive states. Because the exact nature of the sound our voices make is exquisitely dependent on how we breathe, and on everything else we do with our bodies, it then changes as well. Our choices to not let impulses flow – and the breath is only one bodily impulse among many – get caught up in this web. What were once choices can become embedded, difficult, and stubborn. To go far beyond the psychoanalytic and neurophysiological models, we can end up embodying a culture of these choices, and invent together a cultural body that regulates vocal sound based on groups of people making similar choices or playing by similar rules of sonic exchange.
This can end up perpetuating itself within our very tissues, and it can be an incredibly subtle dynamic to identify and shift. The way we embody the complexities of how we structure our physical and psychological engagement with the world – the ways we breathe, look, move, gesture… the ensemble of these is how Bourdieu defined the habitus. Where these complexities start and end is perhaps an infinite loop, a continual cycle of turning and exchange and influence flowing from ourselves to our culture and back again. Our bodies are cultural, counter-cultural, infra-cultural, extra-cultural bodies: we react to culture; we interact with it: we take positions.
Sound – who gets to do it, and when and how – is negotiated, with others, but also, within our own bodies. The traces that others leave there, the things we might call sonic and vocal inhibitions, tensions, these held-back-nesses, eventually become ours to carry, live with, and/or dissolve. They are gifted to us by our culture…. by our environment… by our experience … and by our bodies themselves.
We negotiate sounding.
Pleasure is negotiated, too.
5
We do this to our children: we shut them up. Oh, of course, we also facilitate their sound, and some do this more than others. But even if we give them sonic liberty at home, someone will shut them up, somewhere. We all know and we all remember being silenced as children by somebody, or at least, made to raise our hands in a classroom to ensure one speaker at a time, chosen by the authority in question. Later, teenagers, more often girls than boys, are called mouthy. The mouth: implicitly loud, and if too active, implicitly offensive. The term has been used against feminists, every identity we might include within LGBTI+, African-Americans, and the list goes on.
The wet, open, loud, loud mouth, just ready to mouth off, just ready to make trouble with its irritating, nasty, and above all, bothersome noise – bothersome because it makes us have to react – to have to consider the existence, the needs, the demands of those we might otherwise ignore – that moist orifice can be a source of great pleasure.
6
And on the score of that poor mouthy mouth, let’s consider some other colloquial terms, like ‘sucker’. Sucking is bad, apparently. It expresses need. Thumb out of the mouth! Stop wanting intimacy, reassurance, warmth, contact, and above all stop wanting to satisfy your hard-wired, biological need to suck for comfort and food (my little child). And you there, you sexually active adult! You fucking cocksucker. You ass-licker. That gaping mouth should shut itself up: its gooey pleasures are disgusting. These pleasures involve direct skin-to-skin contact.
Perhaps there is a revolution to be had, in the simple facilitation of gape-mouthed drool.
The vocal tract – that long tunnel surrounded by tongue and palates and teeth and various bits of throat, with at its bottom, the resonant buzz of elastic membranes, through which air is squeezed – also grips the world with direct contact. It’s not just a resonating and sound-shaping cave.
7
I’m making some artworks for children and families right now, and I group them together under the project moniker “Your Vivacious Voice” [See SO! Amplifies post from 6/19/14 to learn more about the free Voice Bubbles App aspect of YB’s project—ed]. I’m collaborating with some scientists and clinician-scientists on this project. They all work with the voice – in psycholinguistics, in understanding infant language acquisition, in voice medicine, and even in laryngeal surgery. We interview these scientists, and use inspiration from our conversations as sources of metaphors for art-making.
One of these is the head Speech and Language Therapist at the Royal National Ear, Nose and Throat Hospital in London, Dr Ruth Epstein. She sees and/or oversees some of the most difficult cases of vocal problems in the whole of the UK. When we asked her what concerns she’d most like us to address in artworks for children and families, she responded along the lines of: please, find a way to get through to them that voice is contact, human contact. She has begun using communication skills, such as eye contact and turn-taking exercises, in addition to vocal skills, in families with children who have injured voices – because she realized at some point that in many of these families, the near exclusive modality of contact was yelling: yelling without contact – without relationship.
The contactless yell is the thrashing arm that somehow remains alone in a void. It’s a yell that might strike if it lands on other flesh, but somehow doesn’t grip, and can’t convert to a caress. It can’t hold… it only punches.
This reminds me of a rockish tune by Carole Pope and Rough Trade from the Canadiana of my childhood – the refrain went:
It hit me like, it hit me like, it hit me like a slap, oh-oh-oh, all touch…
All touch and all touch and no contact…..
8
Back to our children, and to us.
Bodily sound can be a pointed weapon. It can be violent, in that it can frighten, dominate, attack, evoke deep fear, and engage other mechanisms of terror and control and subjugation, and that it can attempt to annihilate our ability to recognize the existence of others. We can drown out others’ sounds. We can drown out their gesture. We can drown their vocalic bodies in our own through amplitude and clashes of timbral spectra. We can shut them up.
Let us consider, here, the desire for amplification and how amplified sound represents an exaggeration of this power, a cybernetic enhancement of the ability to dominate with our emanating waves. We can drown out the social ability for whole groups to hear anyone but ourselves.
However, if, in our cultural environments, everyone is allowed to sound – if, indeed, we facilitate social environments in which everyone’s sound is welcome, then those who are subjected to vocal and sonic violence have an incredible counter-power to this power: they have the power to make sound too.
Although making sound back to violent sound, back to annihilating sound, is not always easy, possible or permitted, it is a power that can’t be easily erased. And we can almost always feel, if not cognitively hear, our own sound vibrate within our own skulls and through our own bones, no matter what is coming from the outside, no matter what waves of vocalic body are streaming toward us. Our sound waves continue to exist, even if transformed.
We can give voice to ourselves. We can change our habits. We can expand away from them.
It isn’t even necessary to fight back. It’s only necessary to vibrate.
And we can take it further.
We can actively encourage each other’s sound. We can actively encourage our children’s sound. We can actively encourage social sound. We can actively encourage a dance with others’ voices. We can facilitate, make space for, enjoy being touched by, the uniqueness of other voices. We can play with how our voices collide and create children with the vocalic bodies of others. After all, our composite vocal bodies are the products of our intensive exchange. We can jublilate in the massages we receive by making our own sound, by vibrating our own skulls, flesh, blood, lymph, interstitial fluid, and the air near us, and we can make it so that we can engage in passionate exchange with the vibrations of others.
This might be something like music. Or other kinds of art. Or it might be simple conversation. Or it might be cooing with a baby. Or it might be making comforting sounds while a toddler cries. Or it might be screaming with rage together.
What it always is, though, is focusing on, opening up to, enjoying the dynamics of the dance of individual, idiosyncratic, messy, fleshly, bodily, sonic emanations reacting with one another.
In the end, the policing of our sound is under our control. We can find ways to unpolice, and enjoy the unbridledness of our sound.
Our bodily sound is a means of engaging passionately with relationship and of glorying in its results.
—
Featured image: “Faces 529” by Flickr user Greg Peverill-Conti, CC BY-NC-ND 2.0
—
Yvon Bonenfant is Reader in Performing Arts at the University of Winchester. He likes voices that do what voices don’t usually do, and he likes bodies that don’t do what bodies usually do. He makes art starting from these sounds and movements. These unusual, intermedia works have been produced in 10 countries in the last 10 years, and his writing published in journals such as Performance Research, Choreographic Practices, and Studies in Theatre and Performance. He currently holds a Large Arts Award from the Wellcome Trust and funding from Arts Council England to collaborate with speech scientists on the development of a series of participatory, extra-normal voice artworks for children and families; see www.yourvivaciousvoice.com. Despite his air of Lenin, he does frighteningly accurate vocal imitations of both Axl Rose and Jon Bon Jovi. www.yvonbonenfant.com.
—

REWIND! . . .If you liked this post, you may also dig:
Experiments in Aural Resistance: Nordic Role-Playing, Community, and Sound— Aaron Trammell
This Is Your Body on the Velvet Underground— Jacob Smith
Sound Designing Motherhood: Irene Lusztig & Maile Colbert Open The Motherhood Archives— Maile Colbert



























Recent Comments