This post is also available in: heעברית (Hebrew)

Dialog will be a critical capability for autonomous systems operating across multiple echelons of Multi-Domain Operations. US Army researchers developed a novel dialogue capability to transform Soldier-robot interaction and perform joint tasks at operational speeds. The fluid communication achieved by dialogue will reduce training overhead in controlling autonomous systems and improve Soldier-agent teaming.

The Joint Understanding and Dialogue Interface, or JUDI, capability, enables bi-directional conversational interactions between Soldiers and autonomous systems. The goal is to shift the paradigm of Soldier-robot interaction from today’s heads-down, hands-full joystick operation of robots to a heads-up, hands-free mode of interaction where a Soldier can team with one or more robots while maintaining situational awareness of their surroundings.

The technology was developed by researchers from the US Army Combat Capabilities Development Command’s Army Research Laboratory, in collaboration with the University of Southern California’s Institute for Creative Technologies – ICT. 

According to, the effort supports the Next Generation Combat Vehicle Army Modernization Priority and the Army Priority Research Area for Autonomy through reduction of Soldier burden when teaming with autonomous systems and by allowing verbal command and control of systems.

Dr. Matthew Marge, a research scientist at the laboratory, said the new technology “enables a Soldier to interact with autonomous systems through bidirectional speech and dialogue in tactical operations where verbal task instructions can be used for command and control of a mobile robot. In turn, the technology gives the robot the ability to ask for clarification or provide status updates as tasks are completed. Instead of relying on pre-specified, and possibly outdated, information about a mission, dialogue enables these systems to supplement their understanding of the world by conversing with human teammates.”

In this innovative approach, he said, dialogue processing is based on a statistical classification method that interprets a Soldier’s intent from their spoken language. The classifier was trained on a small dataset of human-robot dialogue where human experimenters stood in for the robot’s autonomy during initial phases of the research.

“JUDI’s ability to leverage natural language will reduce the learning curve for Soldiers who will need to control or team with robots, some of which may contribute different capabilities to a mission, like scouting or delivery of supplies,” Marge said.While commercial industry has largely focused on intelligent personal assistants like Siri and Alexa – systems that can retrieve factual knowledge and perform specialized tasks like setting reminders, but do not reason over the immediate physical surroundings,” Marge said. “These systems also rely on cloud connectivity and large, labeled datasets to learn how to perform tasks.” In contrast, Marge said, JUDI is designed for tasks that require reasoning in the physical world, where data is sparse because it requires previous human-robot interaction and there is little to no reliable cloud-connectivity. Moreover, he said, JUDI is a dialogue system adapted to autonomous systems like robots, allowing it to access multiple sources of context, like Soldier speech and the robot’s perception system, to help in collaborative decision-making.