The Robotics and Three-Dimensional Vision (RoViT) research group at the University of Alicante has developed a real-time communication platform between the hearing community and the deaf community using sign language. All this in a simple and bidirectional way through natural language processing techniques and the camera and the screen of a computer or mobile device.
TECHNICAL DESCRIPTION
In order to achieve communication between the hearing community and the deaf community using sign language as a means of communication, it has been necessary to develop two functionalities:
1) Sign to text: On the one hand, the platform recognises and interprets what a user signs in front of a camera, providing a transcription in text and voice (see Figure 1). The system that implements this functionality consists of a camera, which can be of any type, including those cameras integrated in mobile devices and tablets, which will be in charge of capturing the images of a person signing in Sign Language (SL). These images are processed by neural networks to obtain the signed signs, generating a sentence or complete text in SL. This text is the input to a third module in charge of its translation into Oral Language (OL) through a natural language processing (NLP) technique. The generated text is displayed on a screen that can be a computer screen, any mobile device or tablet.
Figure 1. Diagram of the functionality corresponding to the generation of text and speech from signing in front of a camera.
2) From text and/or voice to sign: On the other hand, the platform obtains, from text or a voice message, the corresponding LS transcription and displays it on the screen by means of a virtual avatar (see Figure 2). Thus, in this case, the user will indicate the text he/she wishes to sign by voice, using any microphone or keyboard (including those that allow handwritten text to be typed). Once the text has been obtained, a Natural Language Processing (NLP) technique is used to obtain the text expressed in LSE signs. These signs are signed by a virtual avatar displayed on a screen, whether it is a computer screen, a mobile device or a tablet.
Figure 2: Diagram of the functionality corresponding to the generation of the signing by means of a virtual avatar from text or audio entered by a user.
TECHNOLOGY ADVANTAGES AND INNOVATIVE ASPECTS
MAIN ADVANTAGES OF THE TECHNOLOGY
• This system works in real time, in a simple and fast way.
• It is able to recognise complete sentences, unlike other platforms that only offer sign-by-sign recognition.
• To achieve this communication, it is only necessary to have a mobile device with a camera and a screen.
• The accuracy of the system reaches 95%.
INNOVATIVE ASPECTS
• Allows two-way communication, i.e. conversion from sign to text, but also from text and/or speech to sign. Until now only sign-to-text conversion was possible.
• The system includes facial keypoints that allow for more accurate sign language recognition by including the analysis of facial expressions and vocalisation of signs.
CURRENT STATE OF DEVELOPMENT
The technology is at an advanced stage of development (offering an accuracy of 95%)with a demo. So far, it has been trained with vocabulary especially in the field of catering and health but is intended to be extended to new areas.
COLLABORATION SOUGHT
We are looking for companies or entities interested in acquiring this technology for commercial exploitation through patent licensing agreements or for the development of the technology and its adaptation to the specific needs of their activity.
Ahead of the current Coronavirus outbreak, Innoget is fully committed to contributing to mobilizing scientific and expert communities to find a real solution to the Covid-19 pandemic. Therefore, we're supporting worldwide calls and programs that could help in any aspects of the coronavirus crisis.
Is your organization promoting or looking for innovation or research initiatives to mitigate the Covid-19 outbreak? Email us at covid19@innoget.com to list them.
Channeled through Innoget's online open innovation network, initiatives in the health, virology, medicine, or novel technologies applied to human health, among others, are listed and disseminated to Innoget members -ranging from hospitals, research institutes, scientists, businesses, and public administrations- and innovation partners worldwide.
Create your free account to connect with Universidad de Alicante and thousands of other innovative organizations and professionals worldwide
Send a request for information
to Universidad de Alicante
Technology Offers on Innoget are directly posted
and managed by its members as well as evaluation of requests for information. Innoget is the trusted open innovation and science network aimed at directly connect industry needs with professionals online.
Need help requesting additional information or have questions regarding this Technology Offer?
Contact Innoget support
