Can an application help intelligence analysts engage in better reasoning and produce reports that help decision makers make better decisions? A multidisciplinary team of researchers from Syracuse University, the University of Arizona, Colorado State University and SRC Inc. aims to answer this question by developing digital tools for improving reasoning and decision making.
Members of the TRACE team, from left: Lu Xiao, Carsten Oesterlund, Kate Kenski, Jennifer Stromer-Galley, James Folkestad, Rosa Mikeal Martey, Brian McKernan and Debi Plochocki. Not pictured: David Kellen and Lael Schooler.
The team, led by School of Information Studies (iSchool) Professor Jennifer Stromer-Galley, will develop TRACE (Trackable Reasoning and Analysis for Collaboration and Evaluation), a web-based application aimed at improving reasoning through the use of techniques—such as debate and analogical reasoning—along with crowdsourcing to enhance analysts’ problem-solving abilities and foster creative thinking in order to provide support and guidance where human reasoning falls short.
The 50-month project is supported by a $11.5 million contract from the CREATE (Crowdsourcing Evidence, Reasoning, Argumentation, Thinking and Evaluation) Program of the Intelligence Advanced Research Projects Activity (IARPA), an arm of the Office for the Director of National Intelligence, which heads the nation’s intelligence services.
“Our goal is to create a reasoning and reporting application that is not only effective, but also appealing to users by making the process intriguing and fun while not interfering with their natural reasoning and writing abilities,“ says Stromer-Galley. “What makes this project unique is that we are rigorously testing every aspect of our application using experimental research methods. When this project is done we will have a proven, effective tool for people to use.”
Improving team communication and creating an application that people actually want to use is a key concern of the TRACE team. “Asking people to change the way they work and collaborate is challenging, and with our focus on the process, not just the product of the work, we’ll be able to offer something that engages people in the different dimensions of analysis,” explains Rosa Mikeal Martey, associate professor in the Department of Journalism and Media Communication at Colorado State University.
“In other words, we don’t just want people to end up with an accurate, effective analysis, we also want them to find new ways of engaging with the work and with each other while they do so,” says Martey.
The TRACE team aims to investigate the role of several techniques for enhancing reasoning while also promoting better communication and discussion among teams. “Communication is key to effective and gratifying decision-making,” says Kate Kenski, associate professor of communication and government and public policy at the University of Arizona. “This research gives us an unparalleled opportunity to design an online discussion system that harnesses the best group communication strategies, resulting in successful, thoughtful online interactions.”
Using crowdsourcing to improve reasoning and reduce cognitive bias is an innovative idea that will enable the system to overcome common shortcomings in intelligence work by improving the division of labor and reducing both the systematic and random errors individuals may generate while promoting communication and interaction among teams.
To foster richer user experiences, the TRACE application will apply game-based principles of human-computer interaction (HCI) to create an engaging and intuitive solution that promotes efficiency, accuracy and clarity in analysis. The system, which will be created by SRC Inc., a not-for-profit research and development company, will also employ background software processes, such as machine learning, simple decision trees and advanced natural language processing, to help detect reasoning errors and provide support to a variety of tasks. The application will also adopt responsive and just-in-time mechanisms—called nudges—to guide users through reasoning and reporting, as well as provide recommendations.
In each of the project’s three phases, the TRACE team will conduct a series of experiments with different techniques to identify the best approach to improve human reasoning through the use of digital tools. Those include testing and comparing the effects of different analytic techniques and how different levels of interaction affect reasoning and reporting in order to improve user-interface and application design.
The project benefits from a large team of researchers, practitioners and graduate students with unique abilities and areas of expertise, ranging from human-computer interaction, deliberation, crowdsourcing, game and experimental design, interface and software design, cognitive and decision sciences, as well as computational techniques.
The potential benefits of a platform such as TRACE may go beyond the intelligence community and can provide guidance to those interested in improving group communication and teamwork. “Through TRACE, we hope to improve online communication platforms that help groups make more accurate and satisfying decisions when faced with important and sometimes urgent problem-solving tasks,” says Kenski. “While our system is being designed for the intelligence community, we see our product design as one that can potentially improve group communication in many types of settings, including political, public service and business arenas.”
The team brings together scholars from distinct backgrounds and expertise to tackle the challenge. In addition to Stromer-Galley, Martey and Kenski, the team includes: Research Associate Professor Nancy McCracken; Associate Professor Carsten Oesterlund and Associate Professor Lu Xiao from the iSchool; Professor Lael Schooler and Assistant Professor David Kellen in the Department of Psychology in the College of Arts & Sciences; Professor James Folkestad in the School of Education and Professor Ben Clegg in the Department of Psychology, both at Colorado University; Brian McKernan, assistant professor at Sage College; and two experts in intelligence analysis, Roc Myers and Sarah Taylor.
