What if we could translate data into sound?

The human ear can process information far better than the eye, so why are data almost always presented to us in visual forms? Composer Glenn McClure is the principal investigator of a National Science Foundation project, which aims to translate big data into music

GLOSSARY

ALGORITHM – a sequence of instructions used by computers to solve complex problems using data

BIG DATA – a field of research that develops ways to accurately extract useful information from huge datasets

SONIFICATION – the use of non-speech sound to convey information and perceive data

Whether we notice them or not, we are surrounded by sounds at every moment of our waking lives. And when we interpret these sounds, our brains make decisions about what to do next. “Your alarm clock delivers a sound to wake you up. Social media sites have sound alerts when you receive a new message. When you’re about to cross the road, you might hear an ambulance siren before you see it,” says Glenn McClure. He is a composer, co-founder of the Antarctic Artists and Writers Collective (AAWC) and principal investigator of the Music in the Numbers project.

Emerging research shows that the human ear has the capacity to process information far better than the eye and yet, despite the importance of sound in so many aspects of our lives, its role in the interpretation of scientific data is fairly limited. Indeed, sound tends to be used for threshold analysis – in other words, alarms that are triggered by certain thresholds. For example, an alarm will go off to alert hospital staff when a patient’s heart rate falls or increases to a dangerous level. But, for Glenn, this barely scratches the surface for the potential use of sound when interpreting data. This is where his Music in the Numbers project comes in.

INSPIRATION FOR MUSIC IN THE NUMBERS

In 2016, Glenn travelled to Antarctica on a grant from the National Science Foundation’s (NSF) Antarctic Artists and Writers Fellowship. Its mission is to inspire and educate the public about this polar region and the science that goes on there. With a team of scientists from the Scripps Oceanographic Institute in San Diego, Glenn studied data gathered by seismometers placed on the Ross Ice Shelf. By picking up subtle movements in the ice, these seismometers detected vibrations or infragravity waves travelling through the ocean. Since the ice shelf sits on water, it responds to water movement affected by events around the globe. The data collected by these instruments even registered wave movements from the 2011 Fukushima tsunami in Japan!

Glenn used the seismological data to compose several pieces of choral music, which told a story of our changing planet. “During my time on the ice, I started to realise that the algorithms created for this artistic project may have applications beyond music. Musicians are professional listeners and highly trained in analysing multiple, parallel datasets coming from every instrument of the orchestra. This training allows musicians to listen in ways that the average guy on the street does not. After performing the initial choral pieces, I started researching challenges in data science and found that there is a need for new strategies to analyse big data,” Glenn explains.

SIMILARITIES BETWEEN DATA AND MUSIC

Think about your favourite piece of music. When you listen to this music, your ears will be presented with deeply intricate soundscapes, with each voice and instrument contributing its own small part to the overall sound. According to Glenn, data gathered through scientific experiments – or in other contexts such as in banking – display similarly complex patterns. “Just as we listen to a guitar, bass, keyboard, drums and singers in a rock band, data analysis involves studying many parallel datasets, including the ups and downs and ongoing relationships between multiple sets of numbers,” he says.

With this in mind, music could therefore be used to convey complex datasets (see the print of some sonification translation below). Of course, successfully converting raw scientific data into sound is no easy task. To do this, Glenn and his team use advanced mathematical techniques.

SONIFICATION

Glenn’s team is developing useful algorithms that are able to instruct computers to ‘sonify’ data. This involves creating a set of rules that determine how data points are plotted on a musical canvas. When data are plotted on a graph, the relationship between sets of variables is shown with lines, bars and dots. With music, however, data need to be presented as notes, with varying sounds, pitches and volume.

Not only that, but the sonified data have to be tailored to different users. “For those cultures that widely use the seven note Do Re Mi scale, we will translate the data points into musical pitches that fit that scale. For cultures that use microtones and other musical systems, other algorithms work better,” Glenn explains. Handling large datasets and allowing users to tailor the music to their desired settings were a challenge for the team, but a working prototype known as ‘Music in the Numbers’ has been developed. Excitingly, Music in the Numbers can sonify datasets and generate musical melodies, which can be played using a wide range of instruments and even in different musical genres, from jazz to rock or classical music.

WIDER IMPACT

Technology plays a part in so many aspects of our daily lives. Many careers, from scientific research to finance and city planning, require people – or computers – to accurately interpret vast amounts of incoming information. With Music in the Numbers, Glenn’s team is introducing an entirely new way for us to experience these data. Indeed, work environments could become more productive, and even less stressful.

“Music in the Numbers will allow users to customise these sounds to make their experience both useful and fun,” says Glenn. “This product will provide a screen-free data tool that allows users to track data away from a computer screen, perhaps while they’re jogging, driving, making coffee for their co-workers, etc. This will open up a new avenue for productivity, mitigate current problems with eye strain over too much screen use, and engage a new part of our brain in data analysis – our audio cortex!”

But that is not all. Music in the Numbers will be a new tool for visually impaired scientists. Like visual analysis, braille representations of data have their limitations. Music in the Numbers will put visually impaired scientists on equal footing with their seeing counterparts.

GLENN MCCLURE
Composer
Co-founder of the Antarctic Artists and Writers Program (AAWP)
Principal Investigator of the Music in the Numbers project
USA

RESEARCH PROJECT: eveloping computational techniques for expressing large sets of data using music. The Music in the Numbers team advocates the wider use of sound to present and interpret information.

FUNDER: National Science Foundation (NSF)

ABOUT THE AAWC

The Antarctic Artists and Writers Collective (AAWC) is a gathering of creators that have participated in the National Science Foundation Artists and Writers Fellowship, which provides travel grants to artists of all kinds. With this funding, artists team up with scientific research teams in the Antarctic and tell their stories in imaginative ways. “The project recognises the long heritage of artists that have told these stories since the earliest days of Antarctic exploration,” says Glenn.

The Collective now has over 70 members, all of whom communicate their work to the public with regular exhibitions and presentations – both online and in person.

WHY IS AAWC’S WORK SIGNIFICANT?

Firstly, while the public might understand what scientists study closer to home, it is more difficult to engage the public in the study of Antarctica. AAWC artists have played a vital role in not only informing but also inspiring the public with artistic stories of the vital science done by courageous scientists at the bottom of the planet.

Secondly, misinformation spread by anti-science groups threatens several aspects of our lives, from health to the economy and climate change. AAWC is taking innovative steps to combat this problem. Through paintings, poems, music, photography and other media, AAWC is drawing attention towards scientific research in ways that may not have been possible just a few years ago. Members strongly believe that the integration of art and science will be key to ensuring a widespread understanding of science among the public.

WHY IS IT IMPORTANT FOR SCIENTISTS AND ARTISTS TO COLLABORATE?

Glenn argues that our current educational system forces a divide between students of art and science that really should not exist. In reality, the arts and sciences are strongly interlinked: for example, the physics of sound waves, and the biological mechanisms used to convert them into brain signals, are key to understanding how we play music and how it moves us emotionally.

In order for scientists to effectively communicate their ideas, especially about pressing issues such as climate change and public health, a strong collaboration between art and science has never been more important.  “Scientists and artistic creators have a great deal in common,” says Glenn. “They both try to understand the world around them, study intensely to gain the necessary skills and the proficiencies, and present to audiences with the hope to move them to understanding, compassion and action.”

DOES MUSIC REALLY HAVE A PLACE IN SCIENCE?

From ancient Greece (think of Pythagorus) to Renaissance-era Italy (think of Galileo), many of the greatest thinkers in history believed that science and music were intricately linked. In contrast, our modern culture tends to separate both fields entirely – and, as a student, you might have been made to believe that you must choose one over the other, i.e. the sciences over the arts. Glenn proposes that both science and society could benefit greatly if we shifted back to the attitudes of ancient thinkers towards music.

Music in the Numbers certainly presents an ideal opportunity to do this. “As we increase our understanding of the brain, we are learning that our audio capacity of processing data is equal to or greater than our visual capacity,” Glenn says. “Sonification offers an opportunity to confront the biggest scientific and economic questions of our time.”

MEET GLENN

WHAT DID YOU WANT TO BE WHEN YOU WERE YOUNGER?

I always wanted to be a musician. I was born a severe stutterer. At age 11, I learned a skill to control the disability. Before learning this control, the only time I could open my mouth without the physical and emotional distress of stuttering, was when I sang. Music circumnavigates the part of the brain affected by stuttering. When I sang, it was the only time I could freely communicate what was inside of me. Music had a scientifically measurable effect on my disability, which is why I seek to find hidden voices in the silence of data: the ice, space and students with disabilities.

DURING YOUR NSF FELLOWSHIP IN ANTARCTICA, YUKI TAKHASHI SHOWED YOU AN IONISATION DETECTOR AND CHALLENGED YOU TO LISTEN FOR SOUNDS HE COULD NOT HEAR. HAVE YOU ALWAYS HAD AN ABILITY FOR THE DEEP LISTENING YOU USE IN CONCERT HALLS OR A RECORDING STUDIO, OR IS IT SOMETHING THAT COMES WITH PRACTISE?

It has come with both nature and nurture. On the nature side, stutterers feel the visceral link between the sounds of words and the complicated mechanism that produces them. Some studies suggest that hypersensitive hearing is linked to stuttering, and I have been diagnosed. On the nurture side, a musician learns to deeply listen to, among other things, be able to imitate the professionals they emulate. I listened and learned to imitate many of the musicians in my upbringing. I studied sophisticated audio analysis techniques as a musican.

WHO OR WHAT HAS BEEN YOUR INSPIRATION?

My grandfather Charlie McClure was a Vaudeville revival performer in the 1930s. He always had songs to sing and stories to tell, and also used his music to help people in his small, rural New York State town. My oldest brother’s rock and roll band practised every Wednesday night in my living room. I fell asleep listening to songs from the Beatles, the Rolling Stones and Wilson Pickett. My parents were medical volunteers for the seasonal communities of migrant farmworkers. As they were doing medical check-ups, I would wander over to any informal music events, hearing Mexican, Puerto Rican and Southern Black Music. Music was never isolated in my life, it was always part of something bigger.

YOU CALL YOURSELF A MUSICAL ACTIVIST. HOW HAVE YOU USED MUSIC AS A FORM OF PROTEST?

Music not only serves as a voice of protest for the individual musician but also has the power to draw together diverse constituencies toward a common goal. My career was launched when I joined a local environmental group that was protesting a proposal to fill the empty cavities of our local rock salt mine with incinerated garbage ash. Scientists, political consultants and volunteers were all doing great work, but all this hard work wore them down. I found that a series of concerts that celebrated the good work of these people, and articulated our goals, played an important role in sustaining the protest until the proposal was rejected by our state government. I went on to perform with folksinger Pete Seeger and many others in support of this issue. My musical career has always held on to science and the environment in big and small ways.

WHY IS IT SO IMPORTANT TO END THE DIVIDE BETWEEN SCIENTISTS AND ARTISTS IN SCHOOLS?

In the US, our current educational culture is still based upon a 19th century industrial model that was originally designed to help young farm kids learn how to work in an urban factory, because the factory was the driver of their future employment. Our schools mimic the industrial model of dividing tasks and separating math, science, English, music and art. Today, the drivers of future employment are connectivity, innovation, effective communication and leadership. Our school systems need the inspiration of countless musicians and scientists to reimagine how to prepare students for the world unfolding in front of us, rather than the world of 150 years ago.

GLENN’S TOP TIPS

01 We all feel the burdens of information overload these days and no doubt you’ll feel it more than anyone else. We are constantly asking ourselves, “Is this important or can I scroll to something else?” “Is this going to be on the test or can I just ignore it?” Don’t wait for a teacher to tell you what is important or not. EVERYTHING MATTERS.

02 You should always be looking for ways that your studies of science or music connect to other things. You should always be curious, willing to try something that looks difficult, something totally out of your comfort zone. You should be joyfully willing to fail, then laugh, then try again.

03 Our lives are already integrated with learning, beauty, risk, success, mystery, failure and wonder. Embrace it all and watch where it takes you. As a high school student, I never put together a plan to get to Antarctica. I never imagined it. But I did good work and followed the advice of good people.

Do you have a question for Glenn?
Write it in the comments box below and Glenn will get back to you. (Remember, researchers are very busy people, so you may have to wait a few days.)