Mann schaut auf ein Display mit dem Gebärdenavatar von AVASAG.

AVASAG: High-quality, automated sign language interpreting

Barbara Gleißl Senior UX Researcher • Specialist UX Research

14/07/2020 • 5 minutes reading time

Did you know that text is difficult for many deaf people to use, almost like a foreign language?

In actuality, reading and writing are often a major barrier for people who are hard of hearing that not only makes everyday life significantly more difficult in general, but also analogue and digital searches for information as well as interacting with media. Together with our research partners, we have joined the AVASAG project to find a solution to overcome these barriers.

There are around 70 million deaf people worldwide, with around 80,000 living in Germany (sources: yomma GmbH und Deutscher Gehörlosen-Bund e.V.). Contrary to popular belief, written language is like a foreign language to many of these people. Why? Learning to read and write is a different process for deaf people than for hearing people, as text and writing are learned as a depiction of spoken language. Allocating letters to spoken sounds is difficult or even impossible for someone that cannot hear.

The value of sign language

Sign language is an alternative that many deaf and people hard-of-hearing use — a visual manual language that employs the hands, facial expressions, line of vision, head and upper body positions as well as mouth movements. Sign language also has strong regional differences and dialects, resulting in more than 100 different sign languages existing across the globe.
For deaf people globally, their own sign language is more than just a method of communication. It’s an important component of an independent deaf culture that manifested itself in the official recognition of German Sign Language (DGS) in 2002 as well as official legitimation in 2006 in the United Nations Convention on the Rights of Persons with Disabilities. And yet, the stark lack of sign language interpretation in everyday life and in the media, such as television and streaming services, shows just how much the deaf community is still excluded.

Ensuring accessibility for deaf people

A new online access law (OZG) means that German authorities are obliged to make all information available digitally without barriers to accessibility. The difficulties experienced by deaf people when dealing with text language, however, prevent any such barrier-free access. Efforts may be increasing, especially within the realms of public media, to make information and texts available in the form of short sign language videos, but these are often created manually so result in a high workload. Changes and updates to interpreted information require extensive production work.

Research Project AVASAG kicks off: high-quality automated sign language interpretation

A group project supported by the Federal Ministry for Education and Research, AVASAG (“Avatar-basierter Sprachassistent zur automatisierten Gebärdenübersetzung” — avatar-based language assistant for automated sign language interpretation) is intended to develop a solution to these challenges: an avatar that automatically translates German text into German sign language in real time, allowing deaf people and those who are hard of hearing more comprehensive social participation as well as more integration into “digital society”.

Video: Project description in Swiss German sign language by Roger Kistler, UX Designer Ergosign Switzerland, July 2020

Project & research partners

The project began on May 1st 2020 and will come to an end on April 30th 2023 after three years of research. This joint project focuses its research and development on applications within travel information and services.

We are working on this joint project together with five research and development partners:

  • Charamel GmbH (project management) — specialist for interactive avatar-based assistance systems

  • yomma GmbH — sign language experts in Hamburg

  • DFKI GmbH — research into “cognitive assistance systems”, “language technology and multilingualism”

  • TH Köln— institute for media and photo technology

  • Uni Augsburg— human-centered multimedia

As a member of this consortium, we particularly contribute our years of expertise in user analysis and evaluation, and would also like to contribute towards solutions that are equally human-centered and innovative.

Together, we and our research partners within the group will address the needs of deaf people in order to increase societal participation and contribute towards an accessible and inclusive society.

You can find more information about the project and partners on the project website.
We look forward to interdisciplinary collaboration with research partners and innovative research as well as the results that AVASAG will produce.