From the very beginning, the development of artificial intelligence and video games have been deeply intertwined. The first computer designed exclusively to play a game, 1951’s Nimrod, also implemented one of history’s first examples of AI. Today, 68 years later, BA in Game Design graduate Winson Han is pushing AI and video games’ legacy of symbiotic growth even further through his job at Seattle’s Allen Institute for Artificial Intelligence, but not in the way you’re probably thinking. Han isn’t designing AI for games. Rather, he’s designing games for AI.
“It’s kind of an interesting thing,” Han says. “I got my game design degree from DigiPen, but instead of making games for people, I’m effectively creating game environments for AI to interact and play in, in order to more robustly train them.”
Founded by the late Microsoft co-founder and philanthropist Paul Allen in 2013, the Allen Institute for Artificial Intelligence (also known as AI2) is a non-profit AI research and engineering institute. Staffed by some 80 scientists and researchers, AI2’s goal is to push the field of artificial intelligence forward “for the common good,” releasing much of the software, tools, and data pipelines they develop as open-source materials for other researchers to use as well.
For nearly a year, Han has worked as a game designer at Perceptual Reasoning and Interaction Research, also referred to as PRIOR. That specialized team, acting as a part of the larger AI2, “seeks to advance computer vision to create AI systems that see, explore, learn, and reason about the world,” as PRIOR puts it on their website.
Imagine, Han says, a robot you could introduce into your home environment that could immediately start helping with regular cleaning, cooking, or assisted living tasks. That’s just one example of a future scenario that PRIOR seeks to help make possible. The team’s research, which often combines machine learning with computer vision, is concerned with any kind of large-scale visual processing and identification. That includes everything from the practical scenarios Han mentioned to more surreal tests of artificial aptitude. In one project, for example, PRIOR developed an AI that could create and compose complex new Flintstones videos based on whatever description was fed to it — along with the 25,000 original Flintstones videos it had seen and analyzed on its own.
During his time at PRIOR, Han has applied his game design skills to one project in particular, The House Of inteRactions, also known as AI2-THOR. The open-source project is essentially a photorealistic virtual house, designed in the Unity game engine. Using this virtual environment as a staging ground for various experiments, researchers across the world can set their AI systems loose to explore, learn, identify objects and patterns, and interact with the environment itself.
“The big strength of our simulation environment is that if you wanted to place an actual robot in a kitchen and let it find objects, you’d have to reset it every time and mess with all the hardware. It could take hours just to do a hundred runs or so,” Han explains. “But since everything is simulated in this photorealistic Unity environment, we can do thousands or millions of iterations of tests and runs and effectively get the same level of fidelity.”
During his time working on AI2-THOR, Han has designed and built 150 unique virtual rooms for AI to explore and interact with, all variations on five different categories: entryways, kitchens, bathrooms, living rooms, and bedrooms. While the project didn’t initially feature much interactivity, Han has been focusing on building up that element, adding in realistic physics, real-time cause and effect, and object collision to expand the possibilities for AI researchers.
“We’re trying to make the framework as flexible and robust as possible, so whatever a researcher might come up with, they should be able to use this framework to do it,” Han says. “Some researchers might say, ‘I want to see if this AI agent can identify and navigate to a toaster in each scene.’ And that’s as far as they take it, and they write a paper about that. Some might say, ‘We want to see if they can identify a mug, push it off the table so that it breaks and makes a mess.’”
For other researchers, the task could be simply to allow an AI to explore a space and learn what kinds of objects belong in certain environments. “So, an apple, you’re not going to find it in the bathroom, [but] you’ll probably find one in the kitchen,” Han says. “We don’t have any sort of defined task — the task is very user-designed.”
Unsurprisingly, there are some distinct differences between designing a game for humans versus designing a game for artificial intelligence agents. “That’s one big adjustment I had to make going into this. There are a lot of parallels,” Han says, “but the way you do certain things is very different.”
When designing a game for humans, having some kind of feedback, be it visual or auditory, is crucial to indicate when a game state changes. For AI, that principle flies out the window. “We’re grabbing the frame from the graphics card before it’s even rendered and exporting that as just a bunch of numbers, and that’s the metadata that’s transferred over and effectively the ‘feedback’ that the AI is getting,” Han says.
Despite that paradigm shift, Han has made it his mission to bring the modern game design principles he learned at DigiPen to the computer vision research at PRIOR, whether humans are playing or not. “How can we design this so it’s future-proof, so it makes sense within the context of what the end user, in this case the researcher, wants out of the project?” Han says. “The problem-solving approach is really similar to our GAT [game design] assignments. The way I learned to approach the design process, I use that process on an everyday basis.”
Coming in as the sole game designer in a building full of advanced AI specialists, Han says he initially had a bit of imposter syndrome when he started the job.
“Everyone is amazingly smart,” Han says. “I’m surrounded by doctorates, Ph.D. candidates, people submitting papers to the big fancy science journals, so there is sometimes a level of disconnect that’s funny. We’ll do meetings where we talk about what our tasks are for the day, and I’ll hear stuff like, ‘I’m preparing a speech for this paper.’ And then I say, ‘I’m adding collision to lettuce today!’ But in talking more with the people who work here, mutually tackling projects on the day-to-day, I realize there’s a lot you can learn from each other in this cross-disciplinary way. A lot of cool, unexpected things can happen.”
Some of those unexpected occurrences? Not only did Han get to design PRIOR’s current logo, he even playtested a new Pictionary-like game called Iconary that people can play with AI.
That Han is working at AI2 in the first place is something he also says he never expected when he first started sending out his resume, and it ties into a lesson he hopes to communicate to current DigiPen students.
“If I were to give advice to any DigiPen student, game design or whatever, [I would say] don’t be so laser-focused on: ‘I need to get an internship at my favorite game company,’” Han says. “You never know what’s going to happen. I didn’t even know this position existed — it was pure chance that I stumbled upon it. Since then, I’ve gotten to expand what my game design training can be applied to. You never know what sort of opportunities might come along.”