{"id":87203,"date":"2025-08-13T10:00:00","date_gmt":"2025-08-13T08:00:00","guid":{"rendered":"https:\/\/aktuelles.uni-frankfurt.de\/?p=87203"},"modified":"2026-01-30T10:45:29","modified_gmt":"2026-01-30T09:45:29","slug":"when-ai-learns-from-gestures","status":"publish","type":"post","link":"https:\/\/aktuelles.uni-frankfurt.de\/en\/english\/when-ai-learns-from-gestures\/","title":{"rendered":"When AI learns from gestures"},"content":{"rendered":"

A subproject in the Priority Program \u201cVisual Communication\u201d endeavors to make body language analyzable<\/h2>\n\n\n\n
<\/div>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Gestures help people under-stand each other. They are also increasingly important for human-machine interaction.
All photos: Uwe Dettmar; all other photos: Technology Lab<\/p>\n\n\n\n

<\/div>\n\n\n\n

Visual means of communication such as gestures and facial expressions are age-old forms of human understanding. When someone speaks, they not only string words or sentences together but also use their hands, arms, head and face. ViCom, a Priority Program at Goethe University Frankfurt funded by the German Research Foundation (DFG), has been studying these nonverbal channels since 2021. One of the subprojects is examining the significance of body language with the help of virtual reality (VR) recording methods.<\/strong><\/p>\n\n\n\n

Transitioning from the office in the Faculty of Computer Science and Mathematics to the virtual world takes only a few seconds: The test person (in this case, the author) just puts on her VR goggles and is suddenly an avatar standing on a street that runs through a small town \u2013 past a chapel, through the park to the church and onwards to the town hall. Her task then is to describe the way to another test person so that they can follow.<\/p>\n\n\n\n

For computer linguist and computer scientist Professor Alexander Mehler, linguist Dr. Andy L\u00fccking and computer scientist Dr. Alexander Henlein in Frankfurt, the dialogs taking place in the VR lab can deliver valuable insights: What role do gesticulation and facial expressions play when describing the way? How is spoken language combined with pointing, facial cues or other nonverbal instructions? Which route does the second test person take and what helps them to arrive at their destination? These are questions that researchers in the GeMDiS project (Virtual Reality Sustained Multimodal Distributional Semantics for Gestures in Dialog) want to answer.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/div>\n\n\n\n

Pointing \u2013 a clear gesture even in early childhood that the computer must first learn with the help of VR technology.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Gestures: Long neglected by linguistics<\/h3>\n\n\n\n

Although gestures were long regarded in linguistic research as mere accessories to verbal utterances, we know today that they are much more than that. There is a difference between a guest in a restaurant saying \u201cThe food is too salty\u201d and banging his fist on the table as he says it. \u201cWe use gestures and facial expressions because they enable us to convey more information and make communication more efficient,\u201d says Alexander Mehler, and gives another example: \u201cIf I want to talk to my interlocutor about a certain plant, I automatically look or point in the corresponding direction. This makes it easier to identify the object without wasting a lot of words.\u201d<\/p>\n\n\n\n

This is exactly what most of the test persons also do when describing their route in the VR trials. In countless experiments, the GeMDiS team has observed, for example, that the participants use their hands to form a triangle when saying the word \u201cchurch\u201d to describe the roof more expressively. To indicate a pond in the virtual landscape, they draw a circle with their finger. The project hypothesizes that the more frequently the first test person couples spoken language with gestures and facial expressions, the more easily the second test person can follow the way.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Restricted gestures \u2013 less understanding<\/h3>\n\n\n\n

The GeMDiS team can also use virtual reality technology to explore what happens when body language is restricted. For example, the test person\u2019s auditory and visual ability can be influenced by manipulating the audio and video outputs of the VR goggles. It is also possible within the VR environment to influence to what extent they can grasp and use objects virtually: In one of the experiments, for example, the test person could no longer pick up a virtual cup from the table. The researchers discovered that such restrictions of a person\u2019s ability to act have a major impact on communication that is not, by contrast, seen when restricting auditory and visual quality via technical means: Test persons whose possibilities for interaction are curtailed move around far less in the virtual world, they talk more about their experiences there and are generally more negative in their assessment of how they experienced the experiment.<\/p>\n\n\n\n

Each test series in experiments of this kind lasts around 25 minutes. The GeMDiS team then looks in detail at the image sequences and analyzes them. The computer delivers data on hand and face movements from this three-dimensional space. These data are used over the course of the project to train artificial intelligence (AI) models. The AI recognizes when examples frequently recur \u2013 like when a word such as \u201ctree\u201d or \u201ccrossroads\u201d is often combined with a specific gesture when describing the way.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\n
\n

IN A NUTSHELL<\/p>\n\n\n\n