[PDF] [PDF] From VLibras to OpenSigns: Towards an Open - United Nations

[PDF] Augmented Service for Deaf People Using a Text to Sign Language

In the third way we can build our own renderization engine similar to the 3D Java Mobile API offers and translate the data the Text to Sign Language module 

[PDF] Audio to Sign Language Translation for Deaf People - International

API • The text is then pre-processed using NLP (Natural Language Processing) • Finally, Dictionary based machine translation is done Sign language is 

[PDF] Translation of Sign Language Into Text Using Kinect for - ThinkMind

For the sample dataset, 11 DSG gestures were selected for recording Video recording was decided to make with Microsoft Kinect for Windows v2 with SDK 2 0 It 

[PDF] English Text to Indian Sign Language Translator

There have been various mediums available to translate or to recognize sign language and convert them to text, but text to sign language conversion systems  

Arabic Text-to-Sign - ScienceDirectcom

6 nov 2017 · Arabic Text-to-Sign (ArTTS) Model from Automatic SR System Omar H Consequently, sign language translator and the 3D avatar There exist various online tools for machine translation; i e API's library such as Google

[PDF] From VLibras to OpenSigns: Towards an Open - United Nations

the translation of texts from spoken languages into sign languages ilarities in the structure and behavior of components, such as APIs and backends of 

[PDF] SIGN LANGUAGE CONVERTER - AIRCC Publishing Corporation

used for sign language conversion and a voice recognition system for voice open source drivers, SDKs and APIs could have been useful in this system communicate with the voice than text, therefore with the translation of voice to the  

[PDF] Conversion of Sign Language to Text and Speech - IJARnD

appropriate text Keyword: Sign Language, Leap Motion Controller, Bone API 1 INTRODUCTION For more than 250–300 million people around the world, 

[PDF] SignAR: A Sign Language Translator Application - cloudfrontnet

improve as the English words/ text are mapped to the sign language Due to the an SDK like Vuforia platform which tracks and registers the objects of the real 

[PDF] Sign Language Interpreter using Image Processing and - IRJET

develop an application which will translate sign language to text is then converted to audio We have used Google's Text to Speech API for transforming

[PDF] texte à transformer à l'imparfait ce2

[PDF] texte découverte imparfait ce2

[PDF] texte en français à lire fle

[PDF] texte français

[PDF] texte français débutant

[PDF] texte imparfait ce2

[PDF] texte lecture ce1 pdf

[PDF] texte lecture ce2 pdf

[PDF] texte lecture compréhension cp ce1

[PDF] texte lecture compréhension fin cp

[PDF] texte passé composé imparfait cm1

[PDF] textile recycling berkeley

[PDF] texting ruins literacy skills

[PDF] texts in computational science and engineering pdf

[PDF] tfl consultation bus

From VLibras to OpenSigns: Towards an Open Platform for Machine Translation of Spoken Languages into Sign Languages

Tiago AraujoRostand CostaManuella LimaGuido Lemos

June, 2018

1 Introduction

The World Health Organization estimates that approx- imately 466 million people worldwide have some level of hearing loss[31]. In Brazil, according to the 2010 cen- sus of the Brazilian Institute of Geography and Statis- tics (IBGE), there are approximately 9.7 million Brazil- ians with some type of hearing loss, representing around

5.1% of its population[13].

This relevant part of the population faces several challenges for accessing information, since it is gener- ally available in written or spoken language. The main problem is that most deaf people are not prociency in reading and writing the spoken language of their country. One of the possible explanations is the fact that these languages are based on sounds[25]. A study carried out in 2005 with 7 to 20 years old Dutch deaf persons found that only 25% of them had a reading ca- pacity equal or greater than a 9-year-old child without disability [30]. One of the reasons for this diculty is that the deaf communicate naturally through sign languages (SL), and spoken languages are just a \second language". Each SL is a natural language, with its own lexicon and grammar, developed by each deaf community over time, as well as each non-deaf community develop its spoken languages. Thus, there is no unique SL. Although there are some similarities between all these languages, each country usually has its own, some even more than one - by 2013, there were already over 137 sign languages cataloged around the world[4].Digital Video Applications Lab (LAVID) Informatics Center (CI) - Federal University of Paraiba (UFPB)

E-mail:fmaritan,rostand,manuella,guidog@lavid.ufpb.brIn order to allow adequate access of information

for deaf people, one solution is to translate/interpret spoken contents into the associated SL. However, con- sidering the volume and dynamism of information in some environments and platforms, such as the Web, performing this task using human interpreters is a very dicult task, considering the high volume of content that is published daily on the Internet. In the context of Digital TV, the support for sign languages is gener- ally limited to a window with a human sign language interpreter, which is displayed overlaying the video pro- gram. This solution has high operational costs for gen- eration and production of the contents (cameras, stu- dio, sta, among others), needs full-time human inter- preters, which ends up restricting its use to a small por- tion of the TV programming. To address this question pragmatically, one of the most promising approaches is the use of machine translation (MT) tools from spoken languages into SLs. Proportionately to the number of SL, there are also several parallel initiatives to develop machine trans- lation tools for SLs, usually focused on a single lan- guage/country[11,17,24]. Most of these text-to-sign 1ma- chine translation tools, although they were developed independently in their respective countries, have sim- ilarities in approach, scope, and architecture. In gen- eral, the basic functionalities are present in some form in most of them. Some examples are the extraction of the text to be translated from audio and subtitles, the generation of a sign language video, incorporation of the sign language videos into the original videos (e.g, on Digital TV), dactilology and rendering of signs by plugins and mobile applications, etc. There are also sim-1 In this paper, we use the acronym text-to-sign to repre- sent the translation of texts from spoken languages into sign languages.

2Tiago Araujo et al.

ilarities in the structure and behavior of components, such as APIs and backends of communication, transla- tion and control.

The main points of variation are usually the spe-

cic mechanism of machine translation and the sign lan- guage dictionary (visual representation of signs). Con- sidering the use of avatars for representing the content in the sign language, the process of creating the visual represention of signs is usually similar (e.g., a set of animations) and generally depends on the allocation of nancial and human resources, regardless of the tech- nology used.

To reduce this problem, the objective of this pa-

per is to propose an open, comprehensive and extensi- ble platform fortext-to-signtranslation in various us- age scenarios and countries, including Digital TV. In the proposed platform, the common components share generic functionalities, including the creation and ma- nipulation of the SL dictionaries. Only the translation mechanism and the dictionary itself are interchange- able, being specic for each SL. To accelerate the de- velopment, we used the Sute VLibras

2tools and com-

ponents as a basis[2]. Our proposal is the concentration of eorts and re- sources around an unique solution can provide some cutting edge gains, such as the denition of patterns for the industry standard and greater functional exibility for the common components, and also allow advances in the state-of-the-art, such as sharing techniques and heuristics among translation mechanisms. A single standardized platform with centralized pro- cessing of multiple sign languages can also serve as a catalyst for more advanced translation services, such as incorporating text-to-textIn this paper, we use the acronym text-to-text to represent the translation of texts between spoken or written languages. conversion. Thus, we can integrate available translation mechanisms be- tween spoken languages to allow Deaf in Brazil or Spain to understand a text in English, for example.

Another contribution is to leverage the emergence

of a common core machine translator that can be ex- tended/adapted to other languages and regionalisms.

Reducing the eort to make a new SL available may2

TheSuite VLIBRASis the result of a partnership be- tween the Brazilian Ministry of Planning, Development and Management (MP), through the Information Technology Sec- retariat (STI) and the Federal University of Paraba (UFPB), and consists of a set of tools (text, audio and video) for the Brazilian Sign Language (Libras), making computers, mobile devices and Web platforms accessible to deaf. Cur- rently, VLibras is used in several governmental and private sites, among them the main sites of the Brazilian government (brasil.gov.br), Chamber of Deputies (camara.leg.br) and the Federal Senate (senado.leg.br) ). Further information can be obtained from http://www.vlibras.gov.br.further enhance digital inclusion and accessibility, in technologies such as Digital TV, Web and Cinema, es- pecially in the poorest countries.

2 Machine Translation Platforms for Sign

Languages

2.1 Sign Languages

The communication of people with hearing impairment occurs through formal gestural languages, calledSign Languages(SL). SL are languages that use gestures, facial and body expressions, instead of sounds in com- munication. They have a proper linguistic system that is independent of spoken languages and eectively ful- lls the communication needs of the human being, be- cause they have the same complexity and expressiveness of the spoken languages [21].

SLs have properties common to other human lan-

guages[21], such as: {Flexibility and Versatility: SL present several possi- bilities of use in dierent contexts; {Arbitrarities: the word is arbitrary because it is al- ways a convention recognized by the speakers - the sign languages also have words where there is no relation between form and meaning; {Discontinuity: minimal dierences between words and their meanings are discontinued through the distribution they present at dierent linguistic lev- els; {Creativity/Productivity: there are innite ways of expressing the same idea with dierent rules; {Dual articulation: the human languages present units of smaller articulations, without meanings, that com- bined with others form units of meaning; {Standard: there is a set of rules shared by a group of people; {Structural Dependency: elements of the language can not be combined at random, there is a structural de- pendence between them. Generally, each country has its own sign language. The Brazilian Sign Language (Libras), Portuguese Ges- tural Language (LGP), Angolan Sign Language and Mozambican Sign Language (LMS) are the SL of Brazil, Portugal, Angola and Mozambique, respectively, just to name a few countries with the same oral linguistic base (ie, Portuguese). As in spoken languages, there are also variations within the sign language itself, caused by re- gionalisms and/or other cultural dierences.

It is relatively common to assume that sign lan-

guages are agged versions of their respective spoken languages. However, although there are similarities, SLs

VLibras to OpenSigns3

are autonomous languages, possessing singularities that distinguish them from spoken languages and each other

SL [21].

The relevant cultural dierences that impact the

modes of environmental representation are re ected in considerable dierences between sign languages.

2.2 Machine Translation Platforms

Machine translation systems for sign languages are gen- erally divided into three main classes: Rule-Based Ma- chine Translation (RBMT), Statistical Machine Trans- lation (SMT) and Example-Based Machine Translation (EBMT) [26]. One important challenge of these systems is to ensure that the content available to deaf has the same consistency and quality of the original content, allowing the adequate understanding of the message. Considering these systems may be a viable alterna- tive to minimize the marginalization of deaf, especially through digital inclusion, several researches have been developed around the world focusing on the develop- ment and oering of operational platforms for MT from spoken languages into SL.

With respect to machine translation for Brazilian

Sign Language, there are four platforms available for machine translation of Brazilian Portuguese digital con- tents into LIBRAS:Sute VLibras[1,2],HandTalk[6],

ProDeaf[20] eRybena[22].

TheSute VLibrasis a set of open source com-

putational tools that translates digital content from Brazilian Portuguese (BP) into Libras, making the in- formation available for deaf users on computers, TVs, mobile devices and Internet portals. An overview of its component architecture is given by Figure 1.

The VLibras main components are:

{VLibras-Plugin: a browser extension that allows the translation of selected texts to LIBRAS; {VLibras-Mobile: VLibras clients for mobile devices ( both iOS and Android); {VLibras-Desktop: is a tool used to translate into sign language marked texts taken from applications run- ning on personal computers; {VLibras-Video: is a portal that allows translation to LIBRAS of audio tracks or subtitles associated with videos; {LibrasTV: an adaption of VLibras for the Brazilian

Digital TV system.

It is also part of theSute VLibrasa backend

service calledVLibras-Service, which performs the ma-chine translation services for the other components

3 and also hosts the repository of 3D models of the Libras signs that are used by the avatar to render the acces- sible content after the translation. Currently, the Signs

Dictionary of theSute VLibrashas around 13,500

modeled signs, one of the largest bases of the kind in the world. Finally, there is theWikiLibras, a Web tool for the collaborative modeling of signs in Libras, which allows volunteers to participate in the process of building and expanding the signs dictionary, through the specica- tion of the movements of each signal.

3 Open Signs: A Proposal of a Multilingual

Machine Translation Platform

From our experience in the development of theSute

VLibras, we identied that a number of VLibras fea- tures were not dependent of the source and the tar- get language, and possibly applicable to other contexts. Among the technological tools potentially reusable, we can mention: { Plug-insfor three browsers (Google Chrome, Mozilla Firefox and Safari) that allow texts on web pages to be captured, submitted to a remote text-to-gloss 4 translator and the resulting glosses are rendered by an avatar. { TV applicationsfor the Brazilian Digital TV Sys- tem that allow the presentation of sign language contents available on Digital TV signal. { Mobile applicationsfor two platforms (Android and iOS) that allow the translation and rendering of signs from an input text, also using a remote text- to-gloss translator. { Desktop applicationsfor two operating systems (Windows and Linux) that allow contents from mul- tiple sources on the user's computer to be translated to SL and rendered oine. { Extraction mechanismsof texts from audio and videos for a text-to-gloss translation . {Aweb portalfor video translation resulting in a new video with a sign language window synchro- nized with the original audio. An integrated set of tools like this for machine trans- lation using avatars is not easy to develop and we be- lieve there are few initiatives in the world with this3

Except for theVLibras-Desktop, which operates au-

tonomously and oine, having a built-in machine translator and a copy of the Signs Dictionary.

4We use the acronym text-to-gloss to represent the transla-

tion of texts in spoken languages into a textual representation in sign language, called gloss.

4Tiago Araujo et al.Fig. 1Sute VLibras Component Architecture

reach and penetration, and still making available a dic- tionary composed of more than 13,500 Brazilian Sign

Language (Libras) 3D signs.

Based on these resources, our proposal is to oer a multilingual platform for machine translation text-to- sign, which accepts several spoken languages as input and performs a machine translation into several target sign languages. With the eort of generalization prac- ticed in this work, the VLibras framework could become available to be extended and used in other countries and languages. Thus, the main focus of our work was the transfor- mation of a complete platform of machine translation from Brazilian Portuguese (written or spoken) to LI- BRAS, calledVLibras, into an extensible and multilin- gual machine translation platform, calledOpenSigns.

3.1 Building an OpenSigns Platform Prototype

We started with an initial assessment that the majority of theSute VLibrascomponents has generic features that can be shared among several sign languages with minor changes. The main changes needed are aimed at making the components that access the translation services \agnostic", ie, independent of the source and taget languages. In addition, we also focus on enabling the solution to support multiple machine translation engines and multiple sign dictionaries. Figure 3 illustrates the architecture of theVLibras- Core[2]. Initially, it only translated content from Brazil- ian Portuguese (BP) to Libras. Figure 4 presents an

adapted version of this architecture, which includes sup-port for multiple source and target languages. We called

itOpenSigns-Core.

According to Figure 4, the components highlighted

in orange and red represent the points of variance and have been rebuilt to support multiple source and target languages. The components in blue had their basic be- havior maintained. The minor adjustments required are related to the generalization and internationalization of their interface. We will detail these changes in Sections

3.1.1 and 3.1.2.

3.1.1 Multilingual Text-to-Gloss Translator

Originally, the basic role of theSuite VLibrasMT

component was to receive texts in BP and convert them to a textual representation in Libras, called gloss. Since most of the related work uses a rules-based translation approach, including VLibras, we choose this translation approach for the OpenSigns MT component. The main reason for making this choice is the diculty in nding a large and representative bilingual corpus of several domains for all spoken language-sign language pairs. In addition, SLs are visuospatial languages which means that the gloss representation is intermediate. Therefore, there is no formal, structured and widely disseminated written form, which hinders the establishment of natu- ral conventions of writing a natural language and makes it dicult to implement a statistical or a neural MT component, for example. However, it is possible to use other MT approaches. Since this component has a restricted and well-dened function (ie translating a text from a spoken language into a gloss of sign language), it could be replaced by

VLibras to OpenSigns5(a)(b)

(c)(d)

Fig. 2VLibras Aplications (a) VLibras Desktop, (b) VLibras Plugin, (c) VLibras Mobile and (d) VLibras Video

other MT component (e.g., a neural MT implementa- tion, a statistical MT implementation, among others) with little or no modication to the overall architecture of the system. The four main steps of the VLibras translation pro- cess are as follows (see Figure 5)[15]: {Preprocessing: In this step, the input text is sepa- rated in sentences and then in tokens. {Classication: Two classication approaches are per- formed: Morphological, where we identify the gram- matical classes of the tokens in the sentence; Syn- tactic, which groups lexical items into multiple syn- tactical units. {Morphological and Syntactic Adequacy: This process is important because the Libras structure is gener- ally dierent from the Brazilian Portuguese. In addi- tion, Libras has some morphological elements that are not represented, such as articles and preposi- tions. It is also necessary to treat verbal tense, to identify common nouns of two genera, among oth- ers. {Postprocessing: This step renes the sequence of glosses to improve the quality of the translation.

Some examples are: treatment of numbers, plural,

synonyms, among others.As can be seen in Figure 4, one novelty of the Open- Signs architecture is the incorporation of a previous text-to-textconversion. In this new context, the text- to-gloss translation process now consists of two dis- tinct integrated processes: (1) a text-to-text machine translation, which converts the input text into a spo- ken language associated with the target sign language (eg, from English to Brazilian Portuguese); and (2) a specic text-to-gloss translation to the target sign lan- guage (eg, from Portuguese to Libras gloss). The in- ternal organization of the components of the generic text-to-glosstranslator ofOpenSignsis illustrated in

Figure 6.

It is important to note that thetext-to-glosstransla- tion ofOpenSignshas a machine translation algorithm similar to that used in the VLibras, containing the four original steps (see Figure 5). However, the architectural pattern of the translator has been changed to allow the development of several concrete implementations, so that the singularities of each sign language can be addressed punctually, with maximum reuse.

The conguration of the concrete components used

in the text-to-gloss machine translation for each pair of languagessup- ported is dened previously by a template and applied

6Tiago Araujo et al.Fig. 3Internal Architecture of VLibras-CoreFig. 4Internal Architecture Adapted to theOpenSigns

by the OpenSigns translator at run time. This exibility allows the implementation of specic steps that can be addressed for a specic sign language, whereas others can be shared by several sign languages. For example, the preprocessing and post-processing steps, which per- form common actions can be referenced in more than one template, whereas the specic morphological and syntactic classications can be inherited and adapted, or fully reimplemented.In some steps, whose behavior is usually based on external congurations such as models and translation rules, we can reference a generic concrete implementa- tion in template and make adjustments just in the con- gurations. As shown in Figure 7, this strategy allows to conciliate several dierent scenarios for the morpho- logical adequacy step, allowing that existing sign lan- guage implementations can be adapted and integrated to the OpenSigns, and new implementations can be cre-

VLibras to OpenSigns7Fig. 5Internal Organization of theSuite VLibrasTranslatorFig. 6Internal Organization of theOpenSignsTranslatorFig. 7Implementation of the Morphological Adequacy Step

ated using generic classiers and adapters oered by the platform.

3.1.2 Multilingual Signs Repository

Regardless of the translation approach used in the text- to-gloss MT, the quality of the text-to-sign translation generally also depends on the available sign vocabu- lary [14]. In this type of machine translator, the signing is usually done using 3D avatars, that use these signsquotesdbs_dbs17.pdfusesText_23