By Nathan Deen
Discovering something new and unexpected is a dream for any researcher who studies artificial intelligence (AI), but it doesn’t happen often.
But that’s exactly what happened to School of Interactive Computing Professor Mark Riedl and his collaborators, Xiangyu Peng and Prithviraj Ammanabrolu, when they were building a reinforcement learning model for an AI they created to play text adventure games.
AI agents typically learn computer gaming through reinforcement learning, which teaches them how to solve problems that require multiple steps. In text adventure gaming, the agent is presented with a problem scenario and must use text responses to solve the problem.
Text-adventure games were created more than 45 years ago. Riedl said he looked back to them because contemporary games often prove to be limiting for an AI that’s being taught through reinforcement learning.
“Our techniques will work on any agents that work in any other domain — other types of games, the real world, robots, things like that,” Riedl said. “We did these text adventure games because they’re complicated. There’s a lot going on in these games. It’s a lot of problem solving.”
Riedl and his team began researching artificial intelligence in text adventure gaming in 2018. When the agents they created started to become experts in winning the games, however, the researchers realized that they needed to make the agents explainable.
“Reinforcement learning agents have to deal with explanations on two different time scales,” Riedl said. “One is the immediate timescale — what is the agent seeing in this moment that makes it think certain actions are better than others?
“The other is a long-term timescale. They perform an action that will prompt a future action or allow a future goal to be achieved. We look at how current actions chain together with future actions and then try to provide an explanation.”
To explain the AI, Riedl and his team created an algorithm based on knowledge graphs. Riedl said each node on the graph represented a concept that a human can understand, and the nodes (concepts) are linked together in various ways. They used a concept called Hierarchically Explainable Reinforcement Learning to identify the most important data that the agent needed to perform its tasks.
But this method did more than show how the agent thought and made its decisions — it started to learn faster and earn higher scores each time it played the game.
“Basically, we helped it organize its memories into chunks, and it was able to use its memory more efficiently,” Riedl said. “Something about breaking out this structure helped it organize itself, make discoveries, and get rewarded faster.
“Often, when you change an algorithm for some additional purpose, you make the algorithm a little worse. There’s a sacrifice. If I make this algorithm more explainable, I might be giving up some performance or accuracy. We don’t entirely know what to make of this. Is there something special about explainability that supports learning as well? There are some interesting ‘ifs’ associated with that. This never happens, so I’m excited.”
Riedl, Peng, and Ammanabrolu will report their findings at the 36th Conference on Neural Information Processing Systems (NeurIPS), Nov. 28-Dec. 9 in New Orleans.
While Riedl said that the discovery of an improved AI after the introduction of an algorithm opens the door to deeper analysis, there are a couple of immediate implications: It can be translated over to robotics, and the AI community can be less afraid of introducing new algorithms.
“Reinforcement learning can be made explainable and this is going to be a big factor when it comes to human-AI interaction,” Riedl said. “We can put this into robots and have robots talk about what they are doing so we understand.
“Explanations don’t have to be an after-the-thought add-on that hurts our agents. We shouldn’t be saying ‘I don’t want to put an explanation on my system because it won’t work as well.’ We have evidence that this add-on doesn’t have to hurt the performance of our agent.”
To collect feedback of the agent’s performance, the team used a crowd-sourcing platform to poll 40 participants with a randomly selected subset of 10 explanation pairs generated by the hierarchical graphs.
“We gave these explanations to humans and asked them if they understood what the agent was doing,” Riedl said. “Was it explaining itself in a human-like way? Did people have confidence in what the agent was doing? These explanations tended to be rated very highly.”
Riedl said that feedback gave him confidence that he and his team had improved the agent’s performance and achieved their original goal of making it explainable.
“While it’s true we did some algorithm development, we did it for the sake of humans,” he said. “Here’s an agent that does its job well and makes itself accessible to humans. I think that’s a very powerful combination of messages coming together here.”