Soft Exoskeleton Glove For Hand Assistance Based On Human-Machine Interaction and Machine Learning
Soft Exoskeleton Glove For Hand Assistance Based On Human-Machine Interaction and Machine Learning
Abstract—This paper proposes a human machine interaction Some work has been done in the field of exoskeleton
system in the field of stroke rehabilitation, based on the concept gloves [8-11]. M. Cortese et al. [12] develop a novel
of mirror therapy (MT). It aims to improve the hand motor mechatronics master-slave setup for hand rehabilitation. It
function of stroke patients, enabling a true synchronization consists of a sensor glove acting as a remote master and a
between the affected hand and non-affected hand (healthy hand) powered hand exoskeleton as a slave. The master unit can
for the stroke patient. It consists of a soft exoskeleton glove, a control the slave unit to assist the affected hand to complete
surface electromyography (sEMG) signal collecting armband mirror gestures, including “Rest”, “Pinch”, and “Lateral
and machine learning (ML) algorithms. The glove is developed grasp”. J. H. Bae et al. [13] design a wearable hand
by integrating low-power motors to provide force strength for
rehabilitation robot, DULEX-II. The DULEX-II is with three
the hand movement. Unlike the rigid exoskeleton devices, the
degrees of freedom, which can assist both wrists and all
glove is comfortable to wear and lightweight, so it is more
suitable for rehabilitation training of stroke patients in daily life. fingers movements. Serpelloni et al. [14] design a a soft
The armband collects the sEMG signals for pattern recognition robotic glove that is driven by EMG for hand mirror therapy.
by the ML algorithms. In the experiment, four subjects perform The system uses a brushless motor to push and pull the cable,
10 hand gestures to collect data for model training. A which drives the finger movement.
comparison of data preprocessing is conducted to find the In this paper, a rehabilitation system for a power-assisted
optimal data segmentation method and feature vector sets. A
glove control employing a commercial wearable MYO
series of pattern recognition algorithms are developed and
armband is proposed. The system consists of a soft
assessed in different aspects, including prediction accuracy,
training time and predicting time. All 10 gestures can be
exoskeleton glove, a sEMG signal collecting armband and ML
recognized in offline mode with an accuracy up to 99.4%. The algorithms. The armband is worn on a non-affected hand to
control of soft exoskeleton glove in real-time manner is also collect the sEMG signals to sense the hand motion of the
carried out, and the accuracy is 82.2%. The experiment result patient. Then the ML algorithms recognize what motions the
demonstrates the feasibility of the proposed system. The hand made and send classification results to the exoskeleton
innovations and limitations of the work are discussed at the end glove. The exoskeleton glove is controlled to assist the
of the paper. affected hand to accomplish the mirror movements. The
contribution of this work includes: provide the exoskeleton
Keywords—soft exoskeleton glove, surface electromyography glove which the patients can wear on the affected hand with
(sEMG), mirror therapy, machine learning higher comfort and safety than rigid exoskeletons. The patient
can take rehabilitation training at home without being
I. INTRODUCTION confined to the medical environment. Besides, the system has
Stroke is a life-threatening disease, caused by blockage of customized 10 kinds of fine-grained training gestures for
a blood vessel to the brain or bleeding in or around the brain mirror therapy, aiming to train each finger individually and
[1]. About 55% to 75% of survivors have serious disabilities train multiple fingers in coordination. Moreover, the accuracy
such as paralysis and aphasia [2-4]. Among them, the most and training time of different ML algorithms are analyzed and
common is the loss of part or all of the hand’s motor function compared.
[5], resulting in patients unable to take care of themselves.
The rest of this paper is organized as follows. The
Hand dysfunction can seriously affect the quality of patients’
framework of the proposed system is described in Section II.
life, which means more demand is needed on the hand
The methodology of gesture recognition and control of the soft
rehabilitation.
exoskeleton glove are introduced in Section III. Then the
Mirror therapy (MT) is one of the traditional methods for experiment results are evaluated and discussed in Section IV.
improving hand motor function, which is simple, effective, Finally, Section V concludes the whole paper.
and inexpensive [6]. The traditional mirror therapy refers to
placing a mirror in front of the patient. The patient observes II. FRAMEWORK OF THE PROPOSED SYSTEM
the image of the non-affected hand in the mirror, and imagines A. Overview of the System Architechture
the affected hand making the same action, so as to achieve the
therapeutic purpose through optical illusion and visual The architecture of the system implemented in this study
feedback. Mirror therapy is now extended to another method, is shown in Fig. 1. The proposed system consists of the
bilateral movement [1, 7]. In bilateral movement, the following parts: a human-machine interaction module
rehabilitation equipment is used to assist the affected hand to collecting sEMG signals from the user, a pattern recognition
make mirror gestures as the non-affected hand, generating module classifying different hand gestures, and an
direct visual feedback and promoting the rehabilitation of the exoskeleton glove providing power assistance.
patient’s hands. Studies have shown that bilateral The conventional mirror therapy blocks the visual input to
simultaneous exercise, for mirror therapy, is more effective “cheat” the user, and the affected hand does not make any
than unilateral exercise. movement in fact. In this paper, we propose a rehabilitation
therapy scenario that enables a true synchronization between
the affected hand and non-affected hand for the stroke patient.
* X. Chen and L. Gong contributed equally to this work The stroke patient wears MYO armband on the non-affected
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. Architecture of the stroke rehabilitation system. Fig. 3. System architecture of the glove.
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.
Fig. 7. Classification result of a signal feature.
Fig. 6. Accuracy using different overlapped and data windows.
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.
TABLE I. RESULTS OF HAND GESTURE RECOGNITION
Discrim
Neural
Tree inant SVM KNN Ensemble learning
network
analysis
Machine learning
algorithm Linear Fine Subspace hidden
Fine Linear Quadrat Cubic Fine Cosine Cubic Weighte Boosted Bagged Subspac
Discrimi Gaussia Discrimin layer:
Tree SVM ic SVM SVM KNN KNN KNN d KNN Trees Trees e KNN
nant n SVM ant 27, 21
Accuracy 83.9% 68.0% 92.2% 98.6% 99.1% 99.3% 99.4% 98.4% 98.5% 99.2% 64.9% 98.6% 65.7% 99.4% 99.3%
Input Training
16.20 3.99 122.11 106.88 106.52 364.63 36.87 36.72 1090.00 35.68 302.30 87.26 32.48 578.86 31.00
pattern 1 Time (s)
Predict
6.516 7.386 52.635 60.439 58.212 89.681 15.356 30.213 41.555 13.055 25.248 34.637 42.508 152.022 4.947
Time (ms)
Accuracy 81.0% 66.4% 90.7% 98.1% 98.7% 98.8% 98.9% 97.7% 97.9% 98.6% 64.0% 98.0% 64.3% 99.0% 98.3%
Input Training
14.84 2.65 112.64 101.79 103.52 404.70 32.38 32.09 1007.60 32.35 272.23 80.92 28.43 524.30 15.00
pattern 2 Time (s)
Predict
7.142 7.617 52.938 60.638 60.020 89.027 13.880 30.087 40.577 13.693 22.369 36.009 42.543 148.797 5.160
Time (ms)
Accuracy 78.4% 64.7% 88.6% 97.0% 98.0% 97.8% 98.0% 96.5% 96.7% 97.6% 62.1% 97.0% 62.6% 98.2% 97.3%
Input Training
15.85 3.00 122.66 109.62 112.05 456.55 32.83 32.28 1037.80 33.05 275.87 86.57 29.77 536.22 22.00
pattern 3 Time (s)
Predict
8.406 6.925 53.187 57.515 56.942 84.308 13.601 32.071 39.567 12.544 22.867 36.082 40.919 144.385 5.561
Time (ms)
Input pattern 1: Data length = 300ms, time increment = 50ms; Input pattern 2: Data length = 250ms, time increment = 50ms; Input pattern 3: Data length = 200ms, time increment = 50ms
It is also worth noting that the ensemble classifiers may lower than others, from 88.6% to 92.2% among the three
consume much more time compared with other classifiers. different input patterns. As to the kNN, different parameters
Therefore how to implement a proper ensemble classifier is result in similar results, which are fluctuating from 96.4% to
an essential issue in ensemble learning. 99.4% among different input patterns and distance metrics.
The performance of kNN and SVM are quite similar in terms
At last, a feedforward neural network is implemented. of accuracy. The ensemble learning also shows an interesting
The number of nodes for the input layer is equal to the result. The subspace kNN has the highest accuracy (99.4%)
number of feature extracted vectors. The hidden layer and among all the classifiers, which is slightly higher than kNN.
output layer transfer function is tan-sigmoid function. There Due to the combination of weak learners, the Bagged Trees
are two layers of hidden layer, with 27 nodes and 21 nodes, can reach 98.6% when input is at 300ms, which is far more
respectively. Scaled conjugate gradient backpropagation is higher than the fine tree (83.9%). In contrast, the Boosted
used for network training. The weight for each node is Trees’ accuracy is not only lower than that of the Bagged
adjusted by using a gradient descent strategy. Trees, but also much lower than that of the Fine Tree.
Therefore, we believe that ensemble learning can achieve
C. Hand Gesture Classification Result higher classification accuracy than a single classifier only
As discussed in the previous section, a variety of ML through a proper combination of multiple learners. In hand
classifiers are applied, so we can find the most suitable gesture recognition, a parallelization method can achieve a
classifiers for gesture recognition and glove control. The result better performance than the serialization method. Last but not
of hand gesture recognition is presented in Table I. least, the neural network also shows very high classification
accuracy (99.3%), which is comparable with SVM, kNN and
In this study, 3 types of input patterns are evaluated, the Bagged Trees.
difference is the length of the data window. Obviously, a
longer piece of data contains more sEMG signals, making it The training time and predict time is another important
easier for gesture recognition. Considering the constraint ofaspect of evaluating classifier performance. The Fine Tree and
LDA has quite short training time and predict time. Especially
real-time performance, the input pattern length is set from 200
ms to 300 ms. As mentioned in the previous section the time the LDA has the shortest training time around 3 seconds,
which shows its computing efficiency. However, due to their
increment is related to the total number of training sets, so it
is set to 50 ms. lower accuracy, they are not the most suitable classifiers.
SVM and kNN have quite high accuracies as discussed above.
Accuracy is the most important factor in evaluating ML The training time of kNN is obviously shorter than SVM’s
algorithms. In general, the accuracy increases with the except Cubic kNN. The real-time predict time of SVM with
increase of segmented data length. From Table I we can find fine Gaussian kernel function reaches 84.308 ms while input
that the decision tree’s accuracy and LDA’s accuracy are data length is 200ms, which hardly meets the real-time
lower than that of other classifiers, around 80%, and 65% performance requirement. As to ensemble learning, the
respectively. As to SVM, we conduct the experiment with a training time is much longer than that of the corresponding
variety of kernel functions. Compared with the decision tree single classifier due to the combination of multiple weak
and LDA, SVM reaches a relatively high accuracy. The learners. The Bagged Trees get a balance between accuracy
accuracy of SVM with Gaussian function kernel is up to 99.3% and predict time with a predicted time around 36 ms. Although
while the data length is 300ms. Linear kernel’s accuracy is the accuracy of Subspace kNN is the highest its real-time
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.
Fig. 8. The illustration of 10 hand gestures, the gestures are controlled by
the hand wearing MYO armband.
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.
provide power strength in the opposite direction, which [8] N. F. Roslan, A. M. M. Ali, and R. Suhairi, "Design of an artificial Arm
significantly increases the weight of the entire system. For the with hand gripper for telemedicine system," i Int. Conf. Eng. Technol.
Technopreneuship, ICET, pp. 284-288, 2014.
future work, it is worth keeping a balance between the system
[9] A. M. M. Ali and M. M. A. Jamil, "Biomechatronics design of a novel
functionality and complexity, fulfilling the requirement of artificial arm," BMEiCON - Biomed. Eng. Int. Conf., 2013.
portable and wearable in our proposed system. [10] H. S. Cao and D. G. Zhang, "Soft robotic glove with integrated sEMG
sensing for disabled Ppeople with hand paralysis," IEEE Int. Conf.
V. CONCLUSION Robot. Biomim., ROBIO, 2016, pp. 714-718.
This study presents a hand rehabilitation system consisting [11] L. Gerez, J. Chen and M. Liarokapis, "On the Development of Adaptive,
of a portable soft exoskeleton glove and a human-machine Tendon-Driven, Wearable Exo-Gloves for Grasping Capabilities
interface based on ML. The glove can provide external power Enhancement," IEEE Robot. Autom., vol. 4, no. 2, pp. 422–429, 2019.
strength to each finger to help the rehabilitation of the affected [12] M. Cortese, M. Cempini, P. R. D. Ribeiro, S. R. Soekadar, M. C.
Carrozza, and N. Vitiello, "A mechatronic system for robot-mediated
hand. The hand gesture recognition interface is implemented hand telerehabilitation," IEEE-ASME Trans. Mechatron, vol. 20, no. 4,
with accuracy in offline mode (99.4%) and real-time mode pp. 1753-1764, Aug 2015.
(82.2%), respectively. The result shows the feasibility of the [13] J. H. Bae, Y. M. Kim, and I. Moon, "Wearable hand rehabilitation robot
proposed system, and the limitations and future works are also capable of hand function assistance in stroke survivors," in Proc. IEEE
discussed. RAS EMBS Int. Conf. Biomed. Rob. Biomechatronics, BioRob, 2012,
pp. 1482–1487.
REFERENCES [14] M. Serpelloni, M. Tiboni, M. Lancini, S. Pasinetti, A. Vertuan, and M.
Gobbo, "Preliminary study of a robotic rehabilitation system driven by
[1] Z. Yue, X. Zhang, and J. Wang, "Hand rehabilitation robotics on
EMG for hand mirroring," in IEEE Int. Symp. Med. Meas. Appl.,
poststroke motor recovery," Behav Neurol, vol. 2017, p. 20, 2017
MeMeA - Proc., 2016, pp. 1–6.
[2] A. Basteris, S. M. Nijenhuis, A. H. A. Stienen, J. H. Buurke, G. B.
[15] X. Chen et al., "A wearable hand rehabilitation system with soft
Prange, and F. Amirabdollahian, "Training modalities in robot-
gloves," in IEEE Trans. Ind. Inform., in press.
mediated upper limb rehabilitation in stroke: a framework for
classification based on a systematic review," J Neuroeng Rehabil, vol. [16] M. Xiloyannis, L. Cappello, D. B. Khanh, S.-C. Yen, and L. Masia,
11, Jul 10 2014. “Modelling and design of a synergy-based actuator for a tendon-driven
soft robotic glove,” in Proc. IEEE RAS EMBS Int. Conf. Biomed. Rob.
[3] A. Chowdhury et al., "Active physical practice followed by mental
Biomechatronics, BioRob, 2016, pp. 1213–1219.
practice using BCI-Driven hand exoskeleton: A pilot trial for clinical
effectiveness and usability," IEEE J Biomed Health Inform, vol. 22, no. [17] N. Nilsson, B. Hakansson, and M. Ortiz-Catalan, "Classification
6, pp. 1786-1795, Nov 2018. complexity in myoelectric pattern recognition," J Neuroeng Rehabil,
vol. 14, Jul 10 2017.
[4] D. S. Nichols-Larsen, P. C. Clark, A. Zeringue, A. Greenspan, and S.
Blanton, "Factors influencing stroke survivors' quality of life during [18] B. Hudgins, P. Parker, and R. N. Scott, "A new strategy for
subacute recovery," Stroke, vol. 36, no. 7, pp. 1480-4, Jul 2005. multifunction myoelectric control," IEEE Trans. Biomed. Eng, vol. 40,
no. 1, pp. 82-94, Jan 1993.
[5] S. L. Wolf et al., "Effect of constraint-induced movement therapy on
upper extremity function 3 to 9 months after stroke - The EXCITE [19] K. Englehart and B. Hudgins, "A robust, real-time control scheme for
randomized clinical trial,", Jama-J Am Med Assoc, vol. 296, no. 17, pp. multifunction myoelectric control," IEEE Trans. Biomed. Eng, vol. 50,
2095-2104, Nov 1 2006. no. 7, pp. 848-854, Jul 2003.
[6] C. Y. Wu, P. C. Huang, Y. T. Chen, K. C. Lin, and H. W. Yang, [20] P. Kaufmann, K. Englehart, and M. Platzner, "Fluctuating EMG signals:
"Effects of mirror therapy on motor and sensory recovery in chronic investigating long-term effects of pattern matching algorithms," Annu.
stroke: a randomized controlled trial," Arch Phys Med Rehabil, vol. 94, Int. Conf. IEEE Eng. Med. Biol. Soc., EMBC, 2010, pp. 6357-6360
no. 6, pp. 1023-30, Jun 2013. [21] M. A. Oskoei and H. Hu, "Support vector machine-based classification
[7] A. Boos, Q. Y. Qiu, G. G. Fluet, and S. V. Adamovich, "Haptically scheme for myoelectric control applied to upper limb," IEEE Trans.
facilitated bimanual training combined with augmented visual Biomed. Eng, vol. 55, no. 8, pp. 1956-65, Aug 2008.
feedback in moderate to severe hemiplegia," in Proc. Annu. Int. Conf.
IEEE Eng. Med. Biol. Soc. EMBS, 2011, pp. 3111-3114.
Authorized licensed use limited to: San Jose State University. Downloaded on September 01,2021 at 21:28:19 UTC from IEEE Xplore. Restrictions apply.