Manuscript Title:

ADVANCED TOUCHLESS HCI: A GESTURE RECOGNITION-BASED MULTILINGUAL VIRTUAL KEYBOARD FOR CHARACTER INPUT

Author:

MD. ABDUR RAHIM, JUNGPIL SHIN, MD. ABUL HASHEM, HEMEL SHARKER AKASH, MD. IMRAN HOSSAIN, MD. NAJMUL HOSSAIN, ABU SALEH MUSA MIAH

DOI Number:

DOI:10.5281/zenodo.14039789

Published : 2024-11-10

About the author(s)

1. MD. ABDUR RAHIM - Associate Professor and Head, Department of Computer Science and Engineering, Pabna University of Science and Technology, Rajapur, Pabna, Bangladesh.
2. JUNGPIL SHIN - Professor, School of Computer Science and Engineering, The University of Aizu, Fukushima, Japan.
3. MD. ABUL HASHEM - M.Sc. in Engineering Degree, Department of Computer Science and Engineering, Pabna University of Science and Technology, Rajapur, Pabna, Bangladesh.
4. HEMEL SHARKER AKASH - B.Sc. in Engineering Degree, Department of Computer Science and Engineering, Pabna University of Science and Technology, Rajapur, Pabna, Bangladesh.
5. MD. IMRAN HOSSAIN - Associate Professor, Department of Information and Communication Engineering, Pabna University of Science and Technology, Rajapur, Pabna, Bangladesh.
6. MD. NAJMUL HOSSAIN - Associate Professor, Department Electrical, Electronic and Communication Engineering, Pabna University of Science and Technology, Rajapur, Pabna, Bangladesh.
7. ABU SALEH MUSA MIAH - Post-Doc Researcher, School of Computer Science and Engineering, The University of Aizu, Fukushima, Japan.

Full Text : PDF

Abstract

Virtual keyboard-based non-touch character input systems present an advanced communication method between humans and computers, offering interaction in challenging environments like industrial settings. Extensive research has explored touch and touchless input methods, including hand gestures, aerial handwriting, sign language recognition, and finger alphabet systems. However, many systems require significant learning and complex processing for accurate character recognition. This reveals the need for more efficient, accessible, and low-overhead solutions in non-touch input technologies. This paper presents a virtual keyboard-based character input system that utilizes hand gesture detection to create a novel touchless human-computer interaction (HCI) interface. The study has two key components: a hand gesture recognition system and a character input method. The system leverages MediaPipe's pre-trained models to accurately detect human body keypoints, enabling mid-air typing through intuitive hand gestures. We calculated the angles and distances between various keypoints to extract the features for gesture recognition. OpenCV is used for data collection, and Pynput facilitates keyboard control. A CronoNet architecture-based model powers the system, translating hand gestures into precise keyboard inputs. The virtual keyboard supports seamless transitions between language layouts, including English and Bengali. It recognizes gestures for commands such as scrolling (up/down), swiping (left/right), thumbs up, and finger tapping for input. The system achieved an average accuracy of 96.54% in gesture recognition and 97.07% in character input, showcasing its superiority over state-of-the-art methods.


Keywords

Gesture Recognition, Virtual Keyboard Interface, Mediapipe, Crononet Architecture.