An Eye-Controlled Wearable Communication and Control System for ALS Patients Smarteyes
* Author to whom correspondence should be addressed.
Sigma Journal of Engineering and Natural Sciences 2017, Vol. 8, Suppl. 2, pp. 107-116; doi.org/10.62051/ytu.sigma-journal-of-engineering-and-natural-sciences-an-eye-controlled-wearable-communication-and-control-system-for-als-patients-sma
Abstract
Keywords: ALS; eye tracking; image processing; wearable technologies; hardware control.
1. Introduction
Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease which is the blocking of motor neurons in the nerve cells of the spinal cord. The degeneration of the motor neurons causes the unwanted result of losing muscle control. Communication between caregivers and ALS patients is crucial for life quality issues. These patients mainly suffer from communication difficulties due to deterioration of motor neurons. Therefore, it is very hard for them to communicate with outside world. Although the limbs get paralyzed gradually, the eye muscles
Corresponding Author/Sorumlu Yazar: e-mail/e-ileti: esumer@baskent.edu.tr, tel: (312) 246 66 66 / 1305
E. Sümer, İ.B. Uslu, M. Türker / Sigma J Eng & Nat Sci 8 (2), 107-116, 2017
tend to retain. This means, the eyes still function and the eye gaze is the only way for communication. Several efforts related with the eye-based communication systems for paralyzed individuals can be seen in the literature. Most of these studies either concentrated on computer interaction or hardware control. In a study conducted by [1], a virtual keyboard system (EyePad®) was developed based on eye blinking. The eye blinks are converted into electro-oculogram and electro-myogram signals to control the virtual keyboard. By this system, a better communication media is provided for ALS patients. In a different study, a communication system was developed for ALS patients, who are speechless and having no hand control [2]. The proposed system determines the gaze direction via tracking the eye movements by means of a head-mounted CCD camera. The gaze directions are associated with the predefined requests. The experiments were performed by considering 9 or 12 gaze directions and 90% accuracy was reported. In a similar study conducted by [3], a head mounted camera system that tracks the eye movements was proposed. The eye movements are used to write messages via a virtual keyboard. The Japanese character input experiment was done on three healthy students and the input speeds were computed as 3.45, 2.70 and 2.15 seconds per one character, respectively. [4] developed a wordprocessor based on eye’s natural focusing movements for paralyzed and ALS patients. The system has several functions such as computer mouse control and nurse calling. It is reported that the system allows a patient to input eight letters per minute. In a similar study conducted by [5], a communication device was developed for paralyzed and ALS patients. The device inputs signals received from the eyes, fingers, and lips to control the computer mouse and activate menu functions of TV, air condition, alarm and emergency call. It is reported that the developed device helped the people with disability for increasing their communication ability. In a different study conducted by [6], head movements are used to communicate with peripherals such as light, air conditioner and telephone. A laptop camera is utilized to capture the head movements and an infrared emitter is employed to control the peripherals. It is reported that the patients can communicate successfully and operate electric devices by using the proposed life support system. In this study, we developed a wearable communication and peripheral control system, SmartEyes, for ALS patients, which is based on the movements of eye gaze. In communication part, the system vocalizes the needs and requests of patients by tracking their looking direction. Besides, in control part, various peripherals such as TV, light, rolling curtain, air condition are operated in the same way. The main novelty of this system is that it is a wearable system that integrates the communication and control parts in a single framework. Furthermore, the users do not need to look at any screen with the help of voice guidance. This means that the system can be used in any position such as sitting or lying on. Moreover, due to the infrared LEDs attached on the camera, SmartEyes operates in day or night.
2. System Architecture
The architecture of SmartEyes is shown in Figure 1. The eye region images are captured by the glasses module that contains an infrared camera. The images are transmitted to the data processing unit by a wireless transmission module that is embedded into the glasses module. To extract the pupil, a number of image processing methods are employed. After determining the gaze direction, the patient initiates either the communication or the control module. In communication module, several needs and requests can be selected by the eye movements. Similarly, in control module, four peripherals can be controlled in the same way.
An Eye-Controlled Wearable Communication and … / Sigma J Eng & Nat Sci 8 (2), 107-116 2017
2.1. SmartEyes Glasses Module
The glasses module constitutes the fundamental component of the SmartEyes system. The prototype is designed ergonomically and the integrated camera apparatus can be adjusted to fit different users. Besides, the apparatus do not occlude the direction of the view. One earpiece of the glasses frame is produced by the 3-D printing technology and the original frame is used for the remaining parts. The developed earpiece contains the camera, wireless transmission module, and the battery compartment. The camera lens is directed to the eye by an elastic wire. The analog camera contains ¼ inch high definition (HD) sensor with the frame rate of 60 fps. It also contains the infrared (IR) LEDs mounted around the lens. Therefore, the camera can operate in low illumination levels. The captured images are transmitted to data processing unit via the wireless transmission module operated at 2.4 GHz frequency. The power is supplied by a single 3.7 V AAA-type battery, which has 1000 mAh current capacity. Under these circumstances, the glasses module can operate about three hours. The produced prototype is illustrated in Figure 2.
E. Sümer, İ.B. Uslu, M. Türker / Sigma J Eng & Nat Sci 8 (2), 107-116, 2017
2.2. Pupil Detection
The real-time pupil detection is achieved through processing the image frames using the Matlab programming language. The processing steps of the image frames sent by the wireless transmission module are listed in Figure 3.
Figure 3. The image processing steps for pupil detection In the first step, the image frames acquired in Red-Green-Blue (RGB) color space are converted into gray-level. Then, the image contrast is adjusted using the ‘imadjust’ function of Matlab by using the suitable parameters [7]. Next, the noisy pixels are removed using an 11x11 median filter [8]. This step is followed by specifying an intensity level for gray-level thresholding. This operation discriminates the pupil region from the background. The pupil region is then enhanced using the morphological connected component operators, such as opening and hole filling. Finally, the pupil regions of both eyes are extracted as shown in Figure 4. Note that the upper row in Figure 4 illustrates the captured eye region images while the lower row indicates the corresponding pupil regions colored in white.
Figure 4. The detected pupil regions for the left and right directions
2.3. SmartEyes Interface and Communication Modules
SmartEyes system is commonly configured by the caregivers of the patients. This configuration includes; the specification of menu items regarding to their needs and requests, adjustment of the environment’s light level and eye gaze calibration. After completing these steps, the control is passed to the patient. From now on, the patient can have the full control over the system through his/her eye movements. 2.3.1. Menu Generation The generation of a menu is the first configuration step performed by the caregiver. The caregiver selects the requests and needs from a predefined list by considering the patient’s current 110
An Eye-Controlled Wearable Communication and … / Sigma J Eng & Nat Sci 8 (2), 107-116 2017
condition, and constructs a custom menu using the SmartEyes Interface Module. The main menu can be listed as:
Emergency Case sounds an alarm to warn the caregivers. Easy Communication contains basic communication messages, such as ‘Yes’, ‘No’, ‘Okay’, etc. Some essential food and beverage needs as well as hygienic requests appear in the Basic Needs menu item. Control menu item contains the peripheral control commands, such as switching off the TV, turning on the light, etc. The menu list is compiled as a result of face-to-face interviews or surveys conducted on patients or their caregivers. The top 5 results of a sample survey conducted on 77 patients / caregivers are listed in Table 1. A responder can vote an option from 1 (most important) to 5 (least important). By considering the first two important responses, a notification of an emergency case has the highest rating, while the control ratings for TV and air conditioner are on the third and fourth places, respectively. Table 1. The results of a sample survey Requirements 1) Emergency Notification 2) Sickbed Position Adjustment 3) TV Control 4) Air Conditioner Control 5) Phone Call Answering
Each menu item contains different number of sub-menu items, and a total of 118 needs and requests are addressed in SmartEyes. The user can select the desired menu items and generate a customized menu. The items are also matched with the gaze directions that are automatically assigned by the system. For instance, Emergency Case is matched with the left gaze direction whereas Easy Communicaiton, Basic Needs and Control are matched with the up, straight, and right directions, respectively. The menu items are triggered when a valid gaze direction is detected. The details for the gaze direction detection and system execution are given in Section 2.3.4. An example screenshot from the main menu of SmartEyes is shown in Figure 5.
E. Sümer, İ.B. Uslu, M. Türker / Sigma J Eng & Nat Sci 8 (2), 107-116, 2017
2.3.2. Light Level Adjustment Patients live in different environments according to the phase of their diseases. Some patients are bedridden while some live with a wheelchair. Therefore, the illumination conditions of the environment may vary as the location of the patient changes. In order to minimize the illumination problems, a light adjustment step is essential. In this step, an intensity level is specified interactively. For each different lighting condition, the user changes the level of intensity to obtain a clear discrimination of the pupil. Once a satisfactory threshold level is determined, the parameters are stored to be used during the execution of the system. This adjustment step can be skipped for the patients that barely or never change their locations. 2.3.3. Eye Gaze Calibration The last step before the execution of a process is the calibration of the eye gaze. In this step, four reference directions (left, right, up, straight) are determined to navigate on menu items, explained in Section 2.3.1. To initiate the calibration step, the patient looks straight ahead for a while. This direction is labeled as ‘straight’. Then, the remaining 3 directions are calculated from the direction ‘straight’. To do that, the Center of Mass (CoM) determined for straight direction is shifted to the left, right and up directions by 100 pixels. This value was specified based on several experiments conducted on a number of users. The shifting procedure and the specified directions are illustrated in Figure 6. Note that the yellow spot designates the approximate pupil location when the patient looks straight. On the other hand, the blue, green and red spots designate the estimated pupil locations for the left, right and up directions, respectively.
Figure 6. An example calibration output 2.3.4. Gaze Direction Detection To navigate on custom menu items, the gaze direction must be detected. Navigation is based on voice messages. To make a choice the patient looks a direction for about 2 seconds. During this time period the gaze direction is estimated from the coordinates of the detected pupil’s CoM. The gaze duration starts with the “ready” and ends with the “stop” signals, generated by the system vocally. During this time period the patient should stare at that direction uninterruptedly. The steps to determine the gaze direction are as follows:
1. Detecting the pupil using the processing steps given in Section 2.2.
2. During the gaze duration, computing the CoM’s pixel coordinates of the pupil for each image frame.
3. Computing the average of CoM’s pixel coordinates within the gaze duration.
An Eye-Controlled Wearable Communication and … / Sigma J Eng & Nat Sci 8 (2), 107-116 2017
4. Comparing the average value with the reference direction coordinates found in calibration step. 5. Using the Euclidean distance metric, finding the closest reference direction (i.e. the gaze direction). According to the gaze direction of the patient, any desired menu item can be selected easily. In the case of a wrong selection, the system returns back to the previous menu by means of closing the eyes. Any needs/requests or a peripheral control message can be vocalized by the system when the patient reaches that menu item. For all signals, corresponding Turkish voice messages in .wav format are prepared and loaded to the system. In preparation of voice messages, text-to-speech technology is used.
2.4. SmartEyes Control Module
SmartEyes enables to control four peripherals: (i) Television, (ii) Air Conditioner, (iii) Rolling Curtain and (iv) Desk Lamp. The surveys conducted show that these peripherals have higher ratings compared with others such as telephone, door check, etc. However, there is an exceptional case, which is sickbed control. Although its ratings are quite satisfactory, it is not considered in this study due to its high cost and complex structure. The peripherals and their control signals included in SmartEyes are listed in Table 2. Table 2. The peripherals and their control signals
1) Television 2) Air Conditioner 3) Rolling Curtain 4) Desk Lamp
Control Signal List (i) Turning on, (ii) Turning off, (iii) Changing channel, (iv) Adjusting the volume, (v) Mute (i) Turning on, (ii) Turning off, (iii) Swing, (iv) Adjusting the fan speed (i) Rolling up, (ii) Rolling down, (iii) Rolling up gradually, (iv) Rolling down gradually (i) Turning on, (ii) Turning off, (iii) Dimming up, (iv) Dimming down
Curtain control is carried out through the radio frequency (RF) signals. To do that, 433 MHz ASK RF transmitter module is utilized. On the other hand, the remaining peripherals are controlled through the infrared (IR) signals. In particular, the Phillips RC5 infrared transmission protocol is used for the TV control. The remote control signals are analyzed with an oscilloscope and two different circuits are designed. The first circuit enables the transmission of RF and IR signals. The block diagram and the image of transmitter circuit are shown in Figure 7.
E. Sümer, İ.B. Uslu, M. Türker / Sigma J Eng & Nat Sci 8 (2), 107-116, 2017
Figure 7. The block diagram (left) and the image (right) of the transmitter circuit The second circuit is the receiver circuit for the desk lamp. Since the desk lamp does not contain a remote control mechanism, an external receiver circuit was built therefore. This circuit takes IR signals as input and performs a dimmer control for a tungsten filament lamp. The block diagram and the image of the receiver circuit are shown in Figure 8.
Figure 8. The block diagram (left) and the image (right) of the receiver circuit
3. The Preliminary Results
The preliminary tests are conducted on 3 user groups. The first group contains 4 people from the development team of SmartEyes. This group is regarded as the most experienced group in using the SmartEyes system. The second group is composed of 3 healthy people who have no experience in using the system. The last group contains 2 ALS patients, one of whom is an experienced user. The tests were conducted on different locations, and the users were selected such that the eye colors and eye structures were different from each other. The photos taken for the user groups during the tests are shown in Figure 9. Furthermore, the preliminary results are summarized in Table 3.
An Eye-Controlled Wearable Communication and … / Sigma J Eng & Nat Sci 8 (2), 107-116 2017
Figure 9. Three user groups: Group-1(left), group-2(middle), group-3 (right) Table 3. Summary of preliminary test results Test Group #1 #2 #3
Test Module Communication + Control Communication Communication Total
The experiments for group #1 were conducted in an office environment. The office is the development environment of SmartEyes and it contains the peripherals that are controlled by the system. The accuracy of the first group was computed to be 93.3%. This accuracy level was calculated by counting the number of eye movements that correctly selects a menu item. Of the total 15 selections, 14 were found to be correct. This is validated by comparing the actual requests (declared by the user) with the system outputs. The experiments for the second and third groups were conducted in a house. Groups #2 and #3 yielded the accuracy measures of 80% and 93.3%, respectively. Both groups tested the communication module only due to the difficulties in removing and taking the peripherals to the test side.
4. Conclusion
In this study an innovative communication and control system (SmartEyes) developed for the ALS patients is presented. SmartEyes is a wearable eye-controlled system that aims to improve the life qualities of the ALS patients and their caregivers. The system is composed of four modules, which are (i) Glasses Module, (ii) Interface Module, (iii) Communication Module, and (iv) Control Module. The system is based on voice guidance and the users can declare their needs / requests / control commands with the eye movements. We can say, based on the preliminary test results (about 89% accuracy), that the performance of SmartEyes is quite satisfactory. Besides, several feedbacks that were received from the testers will light the way for the future improvements of the system. Acknowledgments / Teşekkür The study was supported by TÜBİTAK TEYDEB-1512 program with Grant number 2120141 and KOSGEB R&D and Innovation Program. The authors would like to express their deepest gratitude to Dr. Alper Kaya (executive board member of the Alliance of ALS/MND Associations) for his support and encouragement throughout the SmartEyes project. The authors would also like
E. Sümer, İ.B. Uslu, M. Türker / Sigma J Eng & Nat Sci 8 (2), 107-116, 2017
to thank Asst. Prof. Dr. Fikret Arı for his tremendous support for the development of Control Module. REFERENCES / KAYNAKLAR [1] [2] [3]
Gürkan S., Gürkan G., Kaya A., Uşaklı A. B., (2012) Amiyotrofik Lateral Skleroz Hastaları İçin Göz Kırpma Esaslı Yeni Bir Kolay İletişim Cihazı: Eyepad®, Tıp Teknolojileri Ulusal Kongresi, TIPTEKNO’12, 01-03 November 2012, Antalya, Turkey. Maehara T., Uchibori A., Muzikami Y., Wakasa Y., Tanaka K., (2003) A Communication System for ALS Patients using Eye-direction, IEEE EMBS Asian-Pasific Conference on Biomedical Engineering, 20-22 October 2003, Kyoto-Osaka-Nara, Japan. Handa S., Ebisawa Y., (2008) Development of Head-mounted Display with Eye-gaze Detection Function for the Severely Disabled, IEEE International Conf. on Virtual Environments, Human-Computer Interfaces, and Measurement Systems, VECIMS 2008, 14-16 July 2008, Istanbul, Turkey. Yamada H., Fukuda T., (1987) Eye Word Processor (EWP) and Peripheral Controller for the ALS patient, IEE Proceedings, 134, Pt.A, No.4. Tanaka M., Yamanaka Y., Fukuda Y., Ishimatsu T., (2004) Sensor of Communication Device for Serious ALS Patients, SICE Annual Conf., 04-06 August 2004, Sapporo, Japan. Nakashima H., Shibata M., Tanaka M., Moromugi S., Ishimatsu T., Vision Device to Support Independent Life for People with Serious Disability, IEEE International Conf. on Multimedia Technology (ICMT), 26-28 July 2011, Hangzhou, China. Mathworks Inc. (2016) Imadjust function documentation. URL: https://www.mathworks.com/help/images/ref/imadjust.html. Accessed 17 November 2016. Mathworks Inc. (2016) 2-D median filtering documentation. URL: https://www.mathworks.com/help/images/ref/medfilt2.html. Accessed 17 November 2016.
Share and Cite
SÜMER, E.; USLU, İ.B.; TÜRKER, M. An Eye-Controlled Wearable Communication and Control System for ALS Patients Smarteyes. Sigma Journal of Engineering and Natural Sciences 2017, Vol. 8, pp. 107-116. https://doi.org/10.62051/ytu.sigma-journal-of-engineering-and-natural-sciences-an-eye-controlled-wearable-communication-and-control-system-for-als-patients-sma