Japan Society for the Promotion of Science (JSPS)
International Joint Research Program with the U.K. (JRP-LEAD with UKRI):

Understanding cross-signing phenomena in video conferencing situations during and post-COVID-19

Principal investigator: BONO, Mayumi, Associate Professor, Information and Society Research Division

This study analyzes the improvisation and change of sign language communication styles in the context of videoconferencing systems, focusing on how deaf people living in different regions and countries modify and simplify their languages (e.g., translanguaging) when they engage in cross-signing (communication between deaf people who do not share the same sign language, achieved through a simple impromptu sign conversation). The scientific value of this study is in understanding how native sign language speakers have been affected by the dramatic changes in communication environment arising from the COVID-19 pandemic. This situation is an unusual case of the rapid penetration of information technology into communities that use a specific language (e.g., a sign language) and is significant from the viewpoint of cultural and linguistic anthropology, particular in view of the likelihood of humans encountering a similar situation in the future. The originality of this project lies not only in the pioneering nature of the linguistic research, aimed at investigating changes in linguistic practices in online videoconferencing, but also in the development of new AI techniques for generating sign language corpora. The 3D information and body movement information obtained using deep learning will be distributed to the sign language research community to promote further interdisciplinary research on sign languages and informatics. Specifically, this study involves interview surveys, extensive questionnaire surveys, linguistic analysis, and interaction analysis, both in Japan and the U.K. An online sign language dialogue corpus will be created and movements will be detected and annotated using AI techniques. The data collected in this study will be managed in the NII Research Data Cloud (NII RDC), with the aim of making parts of it available for academic use.