Earlier this month I wrote a story for Ideas on an equation that describes how people move in crowds. Afterward I received an e-mail that said, in effect, if you’re interested in things that are hard to model, look at how sound waves bounce around a room.
That e-mail came from Dinesh Manocha, a computer scientist at the University of North Carolina who works on “three-dimensional audio” — creating technology that produces audio for video games, movies, and virtual reality environments that mimics the way we hear real noises in the real world. It’s an improvement that Manocha says has been a long time coming.
“Visual effects in movies and games are stunning, but audio is about 20-30 years behind,” he says, noting that unlike visual effects, which can be computer-generated, audio effects are still physically recorded.
There are a few reasons for the lag. One is simply that we care about visual effects more — we’re want to see a dazzling high-definition movie, but don’t mind listening to it on tinny earbuds. A second is that real-to-life audio is harder to generate. Standing in the middle of a concert hall, sound waves strike our left and right ears differently and what we end up hearing depends on subtleties like the shape of our head and the exact direction the sound waves are traveling when they reach us. Plus, there are all sorts of environmental effects that change the way a note from a trumpet moves — or propagates — through space, long before we register it as music.
“Audio sounds in movies and games are pre-recorded with simple manipulations like increasing the bass, but the echoes and diffractions are not right,” Manocha says. “We need a simple, automatic technology that overcomes that barrier for propagation.”
To do this, Manocha’s lab, which he runs with his colleague Ming Lin, is creating software that uses the wave equation from physics to model how sounds move in different environments. The software takes into account complexities like the materials the sound waves interact with (a shiny window reflects more sound than a rough carpet), the rate at which a sound-emitting object is moving relative to the person who hears it (the Doppler effect), and differences between how sound strikes your (virtual) left and right ears. These are relatively simple calculations to perform in isolation from each other, and given a lot of time, but they’re extremely difficult to perform at the real-time pace of a video game.
“Physics tells us how we do it fast in huge environments like a city,” Manocha says.
Two of Manocha’s former students have started a company, Impulsonic, that provides three-dimensional audio technology to video game designers and architects (who might want to test the sound properties of building features). There’s still a lot of work to do, though, before simulated environments sound as good as they look. Researchers like Manocha are working on expanding the database of synthetic noises — the raw material behind sound propagation in virtual environments — and they have their eyes, or ears, set on one group of sounds in particular.
“Many sound effects in nature we can’t simulate,” Manocha says. “We want to figure out how these ambient sounds are generated.”
The movement represents the antithesis of the chintzy glamor of President Trump’s personal aesthetic.Continue reading »
The logic of the game’s strange new nicknames.Continue reading »
Criminal justice looks at the severity of the crime. But the scales are often tipped by random chance.Continue reading »
The way we use the word racism has become so imprecise, abusive, and even antithetical to genuine activism that change is worth addressing.Continue reading »
Is Western democracy failing? The march of right-wing populism has spawned a cottage industry of pessimism. But the real problem is that liberalism has been a rousing success.Continue reading »
The creator of “The Hobbit” gets more credit than he deserves.Continue reading »
Dynamite and the atomic bomb led to soul-searching among chemists and physicists. Computer scientists have not yet witnessed negative consequences of their work — and behave accordingly.Continue reading »