All things considered, Matt Rosen has had a pretty good year. After graduating this past spring from the BS in Computer Science and Digital Audio program at DigiPen and wrapping up a brief stint at Amazon Web Services, he’s recently embarked on a brand new journey altogether — living and working abroad in Quebec City. As a new hire for the Canadian branch of the Belgium-based Larian Studios, Rosen began working this month on audio programming for the highly anticipated Baldur’s Gate 3, a computer role-playing game set in the Dungeons & Dragons universe.
“It’s going to be weird learning French and moving that far north, but I’m very excited to be working on something I’m going to be passionate about,” Rosen said during an interview in late September.
On top of all that, there’s yet another career highlight to mark Rosen’s remarkable year. This past fall, Rosen co-wrote a research paper that’s been making waves among the audio programming community — or at the very least, simulating them. His paper, titled “Interactive sound propagation for dynamic scenes using 2D wave simulation,” was published by the ACM SIGGRAPH / Eurographics Symposium on Computer Animation 2020. The project was recently spotlighted on the Microsoft Research website, alongside a video demonstration for an open-source software program called “Planeverb,” which Rosen also developed.
Both the paper and the Planeverb program were part of a senior capstone project that Rosen began in 2019 for his MUS 470 course, taught by Dr. Matthew Klassen. What began as a solo class project eventually became a collaborative effort with contributions from two Microsoft research scientists.
Essentially an audio engine with built-in plugin support for Unity, Planeverb represents a new technique by which video games and other real-time simulations can render audio effects, such as reverberation and occlusion. “When you have, for example, a wall in between you and someone who is speaking, it sounds quieter, right? That’s the basis of occlusion. If there’s something in between you and something else, it gets quieter,” Rosen said.
While the computer industries continue to make astonishing breakthroughs in areas like graphical fidelity and dynamic lighting, leading to the advent of what we often refer to as “photorealism,” even some of the biggest games still rely on less-sophisticated techniques when it comes to simulated audio effects, Rosen said. He points to the occlusion effect as one example.
“That effect for audio is usually neglected or kind of phoned in, because it’s really hard to do — and really hard to do fast and well. So a lot of games just use a ray-cast approach. They just cast a ray in a direction [from the source of the sound to the player’s location]; if it gets blocked by a wall, then it’ll be quieter,” Rosen said. “And that doesn’t always work. For example, if there’s a lamp post, if the ray just intersects with one tiny section of a lamp post, it’ll sound a lot quieter, even though it’s just a lamp post — it doesn’t actually make things quieter. So in order to get the effect to sound real, you’ve got to use something called diffraction, which is how real soundwaves travel around corners and through portals.”
With Planeverb, Rosen and his research colleagues demonstrated a way to more accurately replicate how soundwaves interact with various obstacles within an environment — even if those obstacles happen to shift and move in real time.
Rosen’s idea for the project came about last fall after he and a friend watched an on-campus guest presentation by Dr. Nikunj Raghuvanshi, a principal researcher at Microsoft who was demonstrating a new piece of audio technology called Project Acoustics. Similar to Rosen’s eventual program, Project Acoustics was designed as a means to achieve a more authentic type of audio simulation.
“With Project Acoustics, they run a real physically based wave simulation on a 3D environment that you give it. And then they’ll tell you — for each point in the map — what the acoustical parameters would be,” Rosen said. “Then they render that in real-time in a video game. And they do it by using the power of Microsoft Azure, which is a cloud-distributed computing system, so that it runs across dozens of machines.”
Rosen was impressed, not only with the concept, but also the results. “It sounded incredibly realistic, and I wanted to do something like that,” he said.
With that in mind, Rosen and his friend introduced themselves to Raghuvanshi after the event, asking for advice on how to build something similar as part of a one-year student project. “And he was like, ‘Well, maybe if you did a 2D simulation instead of a 3D simulation, you could try to start there and see if you could do something with dynamic geometry,’” Rosen said.
Whereas Project Acoustics accomplished a major feat — accounting for how soundwaves will travel and reverberate in all directions in a given 3D space — it only works for games and simulations where the environment is static and non-changing.
“I mean, it’s still amazing, but it’s a big limitation,” Rosen said, pointing to popular games like Minecraft and Fortnite that allow players to create and destroy parts of the game environment on the fly. “So if there was a door in the scene that could open, [with Project Acoustics] you’d have to have a separate bake of the scene with the door closed and one with the door open.”
By simplifying the soundwave simulation from full 3D space to a more manageable 2D plane, the idea was to build a system that could account for real-time changes in level geometry — a compromise in mathematical complexity and audio fidelity in order to achieve greater flexibility. And so, at Raghuvanshi’s suggestion, Rosen set about researching the idea that would become Planeverb.
“Wave simulation is a tricky thing to get right. If a number is slightly off, the whole thing will explode,” Rosen said. “Basically the entire first three months was working on wave simulation — testing it out, making sure it worked.”
When Raghuvanshi returned to campus for another guest lecturer, Rosen showed him what he had accomplished so far. “He was like, ‘Wow! You’re actually doing something,’” Rosen recalled. “And he gave me some more information about how to continue, and from there it was a little more collaborative. I would implement some things, and he would give suggestions.” Raghuvanshi later introduced Rosen to Dr. Keith Godin, a senior scientist at Microsoft also specializing in spatial audio and interactive acoustics, who also contributed to the academic paper submission published by SIGGRAPH.
Today, you can watch a video demonstration of the Planeverb system, which shows off several examples of smooth variation in acoustics effects in a simplistic 3D environment, all using 2D soundwave diffraction simulation techniques. Impressively, Rosen and his colleagues managed to achieve their goal with a limited CPU budget, relying on a single CPU core to render their audio effects. What does that mean?
“If you’re going to buy a computer, it’ll say something like, this CPU has four cores. This CPU has six cores or eight cores, or something like that. And basically what that means is each core has its own processing unit, and each processing unit can run in parallel. … Modern video games try to use all of the cores so that they can do as many things as possible at the same time,” Rosen explained. “Basically, what I want to do is minimize the amount of CPU usage that I use, so that the actual engine and the [graphics] rendering don’t have to worry about what the audio is doing.”
Not surprisingly, Rosen’s project attracted attention from a number of employers and helped him to land multiple audio-related job interviews, including with Larian Studios. “I’d been following the company for a while since they announced that they would be working on Baldur’s Gate, because I am a big Dungeons & Dragons fan,” Rosen said. “There was an opening and I applied, and I wasn’t expecting a response because I’m in the U.S. and they’re not. They contacted me pretty quickly, so I’m pretty happy with that.”
Fortunately, Rosen was prepared. When asked to write a game in C++ as part of a code test, Rosen went the extra mile by writing an entire game engine from scratch, something he had already done as a student.
“It was definitely overkill. They were like, ‘I can’t believe you actually did this,’” Rosen said. “My experiences from DigiPen directly helped me do a good job on this code test and get a job at Larian.”
Baldur’s Gate 3 became available on Steam Early Access in October.