Tag Archive | Science

Sounds of Science: The Mystique of Sonification

Hearing the Unheard IIWelcome to the final installment of Hearing the UnHeardSounding Out!s series on what we don’t hear and how this unheard world affects us. The series started out with my post on hearing, large and small, continued with a piece by China Blue on the sounds of catastrophic impacts, and Milton Garcés piece on the infrasonic world of volcanoes. To cap it all off, we introduce The Sounds of Science by professor, cellist and interactive media expert, Margaret Schedel.

Dr. Schedel is an Associate Professor of Composition and Computer Music at Stony Brook University. Through her work, she explores the relatively new field of Data Sonification, generating new ways to perceive and interact with information through the use of sound. While everyone is familiar with informatics, graphs and images used to convey complex information, her work explores how we can expand our understanding of even complex scientific information by using our fastest and most emotionally compelling sense, hearing.

– Guest Editor Seth Horowitz

With the invention of digital sound, the number of scientific experiments using sound has skyrocketed in the 21st century, and as Sounding Out! readers know, sonification has started to enter the public consciousness as a new and refreshing alternative modality for exploring and understanding many kinds of datasets emerging from research into everything from deep space to the underground. We seem to be in a moment in which “science that sounds” has a special magic, a mystique that relies to some extent on misunderstandings in popular awareness about the processes and potentials of that alternative modality.

For one thing, using sound to understand scientific phenomena is not actually new. Diarist Samuel Pepys wrote about meeting scientist Robert Hooke in 1666 that “he is able to tell how many strokes a fly makes with her wings (those flies that hum in their flying) by the note that it answers to in musique during their flying.” Unfortunately Hooke never published his findings, leading researchers to speculate on his methods. One popular theory is that he tied strings of varying lengths between a fly and an ear trumpet, recognizing that sympathetic resonance would cause the correct length string to vibrate, thus allowing him to calculate the frequency. Even Galileo used sound, showing the constant acceleration of a ball due to gravity by using an inclined plane with thin moveable frets. By moving the placement of the frets until the clicks created an even tempo he was able to come up with a mathematical equation to describe how time and distance relate when an object falls.

Illustration from Robert Hooke's Micrographia (1665)

Illustration from Robert Hooke’s Micrographia (1665)

There have also been other scientific advances using sound in the more recent past. The stethoscope was invented in 1816 for auscultation, listening to the sounds of the body. It was later applied to machines—listening for the operation of the technological gear. Underwater sonar was patented in 1913 and is still used to navigate and communicate using hydroacoustic phenomenon. The Geiger Counter was developed in 1928 using principles discovered in 1908; it is unclear exactly when the distinctive sound was added. These are all examples of auditory display [AD]; sonification-generating or manipulating sound by using data is a subset of AD. As the forward to the The Sonification Handbook states, “[Since 1992] Technologies that support AD have matured. AD has been integrated into significant (read “funded” and “respectable”) research initiatives. Some forward thinking universities and research centers have established ongoing AD programs. And the great need to involve the entire human perceptual system in understanding complex data, monitoring processes, and providing effective interfaces has persisted and increased” (Thomas Hermann, Andy Hunt, John G. Neuhoff, Sonification Handbook, iii)

Sonification clearly enables scientists, musicians and the public to interact with data in a very different way, particularly compared to the more numerous techniques involving vision. Indeed, because hearing functions quite differently than vision, sonification offers an alternative kind of understanding of data (sometimes more accurate), which would not be possible using eyes alone. Hearing is multi-directional—our ears don’t have to be pointing at a sound source in order to sense it. Furthermore, the frequency response of our hearing is thousands of times more accurate than our vision. In order to reproduce a moving image the sampling rate (called frame-rate) for film is 24 frames per second, while audio has to be sampled at 44,100 frames per second in order to accurately reproduce sound. In addition, aural perception works on simultaneous time scales—we can take in multiple streams of audio data at once at many different dynamics, while our pupils dilate and contract, limiting how much visual data we can absorb at a single time. Our ears are also amazing at detecting regular patterns over time in data; we hear these patterns as frequency, harmonic relationships, and timbre.

Image credit: Dr. Kevin Yager, data measured at X9 beamline, Brookhaven National Lab.

Image credit: Dr. Kevin Yager, Brookhaven National Lab.

But hearing isn’t simple, either. In the current fascination with sonification, the fact that aesthetic decisions must be made in order to translate data into the auditory domain can be obscured. Headlines such as “Here’s What the Higgs Boson Sounds Like” are much sexier than headlines such as “Here is What One Possible Mapping of Some of the Data We Have Collected from a Scientific Measuring Instrument (which itself has inaccuracies) Into Sound.” To illustrate the complexity of these aesthetic decisions, which are always interior to the sonification process, I focus here on how my collaborators and I have been using sound to understand many kinds of scientific data.

My husband, Kevin Yager, a staff scientist at Brookhaven National Laboratory, works at the Center for Functional Nanomaterials using scattering data from x-rays to probe the structure of matter. One night I asked him how exactly the science of x-ray scattering works. He explained that X-rays “scatter” off of all the atoms/particles in the sample and the intensity is measured by a detector. He can then calculate the structure of the material, using the Fast Fourier Transform (FFT) algorithm. He started to explain FFT to me, but I interrupted him because I use FFT all the time in computer music. The same algorithm he uses to determine the structure of matter, musicians use to separate frequency content from time. When I was researching this post, I found a site for computer music which actually discusses x-ray scattering as a precursor for FFT used in sonic applications.

To date, most sonifications have used data which changes over time – a fly’s wings flapping, a heartbeat, a radiation signature. Except in special cases Kevin’s data does not exist in time – it is a single snapshot. But because data from x-ray scattering is a Fourier Transform of the real-space density distribution, we could use additive synthesis, using multiple simultaneous sine waves, to represent different spatial modes. Using this method, we swept through his data radially, like a clock hand, making timbre-based sonifications from the data by synthesizing sine waves using with the loudness based on the intensity of the scattering data and frequency based on the position.

We played a lot with the settings of the additive synthesis, including the length of the sound, the highest frequency and even the number of frequency bins (going back to the clock metaphor – pretend the clock hand is a ruler – the number of frequency bins would be the number of demarcations on the ruler) arriving eventually at set of optimized variables.

Here is one version of the track we created using 10 frequency bins:


Here is one we created using 2000:


And here is one we created using 50 frequency bins, which we settled on:


On a software synthesizer this would be like the default setting. In the future we hope to have an interactive graphic user interface where sliders control these variables, just like a musician tweaks the sound of a synth, so scientists can bring out, or mask aspects of the data.

To hear what that would be like, here are a few tracks that vary length:




Finally, here is a track we created using different mappings of frequency and intensity:


Having these sliders would reinforce to the scientists that we are not creating “the sound of a metallic alloy,” we are creating one sonic representation of the data from the metallic alloy.

It is interesting that such a representation can be vital to scientists. At first, my husband went along with this sonification project as more of a thought experiment rather than something that he thought would actually be useful in the lab, until he heard something distinct about one of those sounds, suggesting that there was a misaligned sample. Once Kevin heard that glitched sound (you can hear it in the video above), he was convinced that sonification was a useful tool for his lab. He and his colleagues are dealing with measurements 1/25,000th the width of a human hair, aiming an X-ray through twenty pieces of equipment to get the beam focused just right. If any piece of equipment is out of kilter, the data can’t be collected. This is where our ears’ non-directionality is useful. The scientist can be working on his/her computer and, using ambient sound, know when a sample is misaligned.


It remains to be seen/heard if the sonifications will be useful to actually understand the material structures. We are currently running an experiment using Mechanical Turk to determine this kind of multi-modal display (using vision and audio) is actually helpful. Basically we are training people on just the images of the scattering data, and testing how well they do, and training another group of people on the images plus the sonification and testing how well they do.

I’m also working with collaborators at Stony Brook University on sonification of data. In one experiment we are using ambisonic (3-dimensional) sound to create a sonic map of the brain to understand drug addiction. Standing in the middle of the ambisonic cube, we hope to find relationships between voxels, a cube of brain tissue—analogous to pixels. When neurons fire in areas of the brain simultaneously there is most likely a causal relationship which can help scientists decode the brain activity of addiction. Computer vision researchers have been searching for these relationships unsuccessfully; we hope that our sonification will allow us to hear associations in distinct parts of the brain which are not easily recognized with sight. We are hoping to leverage the temporal pattern recognition of our auditory system, but we have been running into problems doing the sonification; each slice of data from the FMRI has about 300,000 data points. We have it working with 3,000 data points, but either our programming needs to get more efficient, or we have to get a much more powerful computer in order to work with all of the data.

On another project we are hoping to sonify gait data using smartphones. I’m working with some of my music students and a professor of Physical Therapy, Lisa Muratori, who works on understanding the underlying mechanisms of mobility problems in Parkinsons’ Disease (PD). The physical therapy lab has a digital motion-capture system and a split-belt treadmill for asymmetric stepping—the patients are supported by a harness so they don’t fall. PD is a progressive nervous system disorder characterized by slow movement, rigidity, tremor, and postural instability. Because of degeneration of specific areas of the brain, individuals with PD have difficulty using internally driven cues to initiate and drive movement. However, many studies have demonstrated an almost normal movement pattern when persons with PD are provided external cues, including significant improvements in gait with rhythmic auditory cueing. So far the research with PD and sound has be unidirectional – the patients listen to sound and try to match their gait to the external rhythms from the auditory cues.In our system we will use bio-feedback to sonify data from sensors the patients will wear and feed error messages back to the patient through music. Eventually we hope that patients will be able to adjust their gait by listening to self-generated musical distortions on a smartphone.

As sonification becomes more prevalent, it is important to understand that aesthetic decisions are inevitable and even essential in every kind of data representation. We are so accustomed to looking at visual representations of information—from maps to pie charts—that we may forget that these are also arbitrary transcodings. Even a photograph is not an unambiguous record of reality; the mechanics of the camera and artistic choices of the photographer control the representation. So too, in sonification, do we have considerable latitude. Rather than view these ambiguities as a nuisance, we should embrace them as a freedom that allows us to highlight salient features, or uncover previously invisible patterns.


Margaret Anne Schedel is a composer and cellist specializing in the creation and performance of ferociously interactive media. She holds a certificate in Deep Listening with Pauline Oliveros and has studied composition with Mara Helmuth, Cort Lippe and McGregor Boyle. She sits on the boards of 60×60 Dance, the BEAM Foundation, Devotion Gallery, the International Computer Music Association, and Organised Sound. She contributed a chapter to the Cambridge Companion to Electronic Music, and is a joint author of Electronic Music published by Cambridge University Press. She recently edited an issue of Organised Sound on sonification. Her research focuses on gesture in music, and the sustainability of technology in art. She ran SUNY’s first Coursera Massive Open Online Course (MOOC) in 2013. As an Associate Professor of Music at Stony Brook University, she serves as Co-Director of Computer Music and is a core faculty member of cDACT, the consortium for digital art, culture and technology.

Featured Image: Dr. Kevin Yager, data measured at X9 beamline, Brookhaven National Lab.

Research carried out at the Center for Functional Nanomaterials, Brookhaven National Laboratory, is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, under Contract No. DE-AC02-98CH10886.

tape reelREWIND! ….. If you liked this post, you might also like:

The Noises of Finance–Nicholas Knouf

Revising the Future of Music Technology–Aaron Trammell

A Brief History of Auto-Tune–Owen Marshall


Catastrophic Listening

Hearing the Unheard IIWelcome back to Hearing the UnHeard, Sounding Out‘s series on how the unheard world affects us, which started out with my post on the hearing ranges of animals, and now continues with this exciting piece by China Blue.

From recording the top of the Eiffel Tower to the depths of the rising waters around Venice, from building fields of robotic crickets in Tokyo to lofting 3D printed ears with binaural mics in a weather balloon, China Blue is as much an acoustic explorer as a sound artist.  While she makes her works publicly accessible, shown in museums and galleries around the world, she searches for inspiration in acoustically inaccessible sources, sometimes turning sensory possibilities on their head and sonifying the visual or reformatting sounds to make the inaudible audible.

In this installment of Hearing the UnHeard, China Blue talks about cataclysmic sounds we might not survive hearing and her experiences recording simulated asteroid strikes at NASA’s Ames Vertical Gun Range.

— Guest Editor Seth Horowitz

Fundamentally speaking, sound is the result of something banging into something else. And since everything in the universe, from the slow recombination of chemicals to the hypervelocity impacts of asteroids smashing into planet surfaces, is ultimately the result of things banging into things, the entire universe has a sonic signature. But because of the huge difference in scale of these collisions, some things remain unheard without very specialized equipment. And others, you hope you never hear.

Unheard sounds can be hidden subtly beneath your feet like the microsounds of ants walking, or they can be unexpectedly harmonic like the seismic vibrations of a huge structure like the Eiffel Tower. These are sounds that we can explore safely, using audio editing tools to integrate them into new musical or artistic pieces.

Luckily, our experience with truly primal sounds, such as the explosive shock waves of asteroid impacts that shaped most of our solar system (including the Earth) is rarer. Those who have been near a small example of such an event, such as the residents of Chelyabinsk, Russia in 2013 were probably less interested in the sonic event and more interested in surviving the experience.

But there remains something seductive about being able to hear sounds such as the cosmic rain of fire and ice that shaped our planet billions of years ago. A few years ago, when I became fascinated with sounds “bigger” than humans normally hear, I was able to record simulations of these impacts in one of the few places on Earth where you can, at NASA Ames Vertical Gun Range.

The artist at the AVGR

The artist at the AVGR

The Vertical Gun at Ames Research Center (AVGR) was designed to conduct scientific studies of lunar impacts. It consists of a 25 foot long gun barrel with a powder chamber at one end and a target chamber, painted bright blue, that looks like the nose of an upended submarine, about 8 feet in diameter and height at the other. The walls of the chamber are of thick steel strong enough to let its interior be pumped down to vacuum levels close to that of outer space, or back-filled with various gases to simulate different planetary atmospheres. Using hydrogen and/or up to half a pound of gun powder, the AVGR can launch projectiles at astonishing speeds of 500 to 7,000 m/s (1,100 to 16,000 mph). By varying the gun’s angle of elevation, projectiles can be shot into the target so that it simulates impacts from overhead or at skimming angles.

In other words, it’s a safe way to create cataclysmic impacts, and then analyze them using million frame-per-second video cameras without leaving the security of Earth.

My husband, Dr. Seth Horowitz who is an auditory neuroscientist and another devotee of sound, is close friends with one of the principal investigators of the Ames Vertical Gun, Professor Peter Schultz. Schultz is well known for his 2005 project to blow a hole in the comet Tempel 1 to analyze its composition, and for his involvement in the LCROSS mission that smashed into the south pole of the moon to look for evidence of water. During one conversation discussing the various analytical techniques they use to understand impacts, I asked, “I wonder what it sounds like.” As sound is the propagation of energy by matter banging into other matter, this seemed like the ultimate opportunity to record a “Big Bang” that wouldn’t actually get you killed by flying meteorite shards. Thankfully, my husband and I were invited to come to Ames to find out.

I had a feeling that the AVGR would produce fascinating new sounds that might provide us with different insights into impacts than the more common visual techniques. Because this was completely new research, we used a number of different microphones that were sensitive to different ranges and types of sound and vibrations to provide us with a selection of recording results. As an artist I found the research to be the dominant part of the work because the processes of capturing and analyzing the sounds were a feat unto themselves. As we prepared for the experiment, I thought about what I could do with these sounds. When I eventually create a work out of them, I anticipate using them in an installation that would trigger impact sounds when people enter the room, but I have not yet mounted this work since I suspect that this would be too frightening for most exhibition spaces to want.

Part of my love (and frustration) for sound work is figuring out how to best capture that fleeting moment in which the sound is just right, when the sound evokes a complex response from its listeners without having to even be explained. The sound of Mach 10 impacts and its effects on the environment had such possibilities. In pursuit of the “just right,” we wired up the gun and chamber with multiple calibrated acoustic and seismic microphones, then fed them into a single high speed multichannel recorder, pressed “record” and made for the “safe” room while the Big Red Button was pressed, launching the first impactor. We recorded throughout the day, changing the chamber’s conditions from vacuum to atmosphere.

Simple impact on the AVGR sand target.

Simple impact on the AVGR sand target.

When we finally got to listen that afternoon, we heard things we never imagined. Initial shots in vacuum were surprisingly dull. The seismic microphones picked up the “thump” of the projectile hitting the sand target and a few pattering sounds as secondary particles struck the surfaces. There were of course no sounds from the boundary or ultrasonic mics due to the lack of air to propagate sound waves. While they were scientifically useful–they demonstrated that we could identify specific impact events launched from the target—they weren’t very acoustically dramatic.

When a little atmosphere was added, however, we began picking up subtle sounds, such as the impact and early spray of particles from the boundary mic and the fact that there was an air leak from the pitch shifted ultrasonic mic. But when the chamber was filled with an earthlike atmosphere and the target dish filled with tiny toothpicks to simulate trees, building the scenario for a tiny Tunguska event (a 1908 explosion of an interstellar object in Russia, the largest in recorded history), the sound was stunning:


After the initial explosion, there was a sandstorm as the particles of sand from the target flew about at Mach 5 (destroying one of the microphones in the process), and giving us a simulation of a major asteroid explosion.


66 million years ago, in a swampy area by the Yucatan Penninsula, something like this probably occurred, when a six mile wide rock burned through the atmosphere to strike the water, ending the 135 million year reign of the dinosaurs. Perhaps it sounded a little like this simulation:



Any living thing that heard this – dinausaurs, birds, frogs insects – is long gone. By thinking about the event through new sounds, however, we can not only create new ways to analyze natural phenomena, but also extend the boundaries of our ability to listen across time and space and imagine what the sound of that impact might have been like, from an infrasonic rumble to a killing concussion.

It would probably terrify any listener to walk in to an art exhibition space filled with simply the sounds of simulated hypervelocity impacts, replete with loud, low frequency sounds and infrasonic vibrations. But there is something to that terror. Such sounds trigger ancient evolutionary pathways which are still with us because they were so good at helping us survive similar events by making us run, putting as much distance between us and the cataclysmic source, something that lingers even in safe reproductions, resynthesized from controlled, captured sources.


China Blue is a two time NASA/RI Space Grant recipient and an internationally exhibiting artist who was the first person to record the Eiffel Tower in Paris, France and NASA’s Vertical Gun. Her acoustic work has led her to be selected as the US representative at OPEN XI, Venice, Italy and at the Tokyo Experimental Art Festival in Tokyo, Japan, and was the featured artist for the 2006 annual meeting of the Acoustic Society of America. Reviews of her work have been published in the Wall Street Journal, New York Times, Art in America, Art Forum, artCritical and NY Arts, to name a few. She has been an invited speaker at Harvard, Yale, MIT, Berkelee School of Music, Reed College and Brown University. She is the Founder and Executive Director of The Engine Institute www.theengineinstitute.org.


Featured Image of a high-speed impact recorded by AVGR. Image by P. H. Schultz. Via Wikimedia Commons.


tape reel

REWIND! If you liked this post, check out …

Cauldrons of Noise: Stadium Cheers and Boos at the 2012 London Olympics— David Hendy

Learning to Listen Beyond Our Ears– Owen Marshall

Living with Noise— Osvaldo Oyola

%d bloggers like this: