Abstract

Many investigators are interested in improving the control strategies of hand prosthesis to make it functional and more convenient to use. The most used control approach is based on the forearm muscles activities, named ‘ElectroMyoGraphic’ (EMG) signal. However, these biological signals are very sensitive to many disturbances and are generally unpredictable in time, type, and level. This leads to inaccurate identification of user intent and threatens the prosthesis control reliability. This paper proposed a real-time fault detection and localization approach applied to handwriting device on the plane. This approach allows connecting inputs (IEMG signals)/outputs (pen tip coordinates) data as a parametric model for Multi-Inputs Multi-Outputs (MIMO) system. The proposed approach is considered as a model-independent abrupt or intermittent fault detection method and as an alternative solution to the unpredictable input observer based techniques, without any observability requirements. This approach allows detecting, in real time, several types of faults in one or two inputs signals and in the same or different instants. Our study is appropriate for many rapidly expanding fields and practices, including biomedical engineering, robotics, and biofeedback therapy or even military applications.

1. Introduction

In the last decades, robot control is considered as an important research field, especially for robots intervening in the tasks of everyday life (assistance robots, social robots, service robots, clinical application robots, etc.). Many investigators are interested in improving the quality of prosthesis to make it functional and more convenient to use. The increase in functionality is mainly based on the progression of the control strategies. The most used control approach is based on the amplifier electrical activity of the muscles, named electromyographic signal (EMG), which allows directly encoding the orders generated by the brain [14].

The wealth of information of these biological signals leads several researches to propose approaches based on the muscular activities control. For example, in [7] EMG signals of ten muscles were used to control an artificial hand with four fingers. Schulz et al. proposed, in [8], an artificial hand with hydraulically driven multifunction. In this context, different reviews of controlling by electrical muscles activities are proposed in [912].

However, EMGs are very sensitive to many disturbances in EMG recordings and are generally unpredictable in time, type, and level. The characteristics of the muscles activities are easily affected by many factors, such as recording over layers of muscles, fat, and tissue, abrupt changing of the electrodes positions, sweat of the prosthesis wearer at the recording site, changes of the impedance of the electrode, filtering method, noise of measure, disturbances, and user fatigue [1315]. All these conditions lead to inaccurate identification of user intent and threaten the prosthesis control reliability [1620].

In order to solve these problems, software integrated electromyographic (IEMG) sensor and intelligent techniques are used, in [21], to replace physical sensors. It was also used as a part of fault detection approaches, where the output is compared to the corresponding sensor.

In [22], Zhang et al. proposed a practical fault-tolerant module for robust EMG, based on Mahalanobis distance analysis [23]. Using the Zhang fault approach, Resnik included in [24] a fault detection module which detects faults in the input signals (EMG). In 2018, Huang proposed an EMG fault detector, taking into account only 3 kinds of signal faults (e.g., EMG signal drift and saturation, additional noise, and variation of EMG magnitude) [25].

In fact, nowadays, Fault Detection and Diagnosis (FDD) have been growing interest generated in several fields, especially in the emerging field of bio-robotics. For these reasons, this topic has been addressed in many previous works. In this sense, a bibliographic review on reconfigurable fault-tolerant control approach is presented in [26]. In [27], a data mapping fault detection approach was proposed to detect actuator faults of a manipulator robot. Indeed, FDD can be typically classified into two different classes: model-based and model-free approaches. For model-based class, different methods using mathematical models have been developed in [2732]. Furthermore, approaches that are not based on system modeling used generally intelligent methods like neural network or fuzzy logic concepts. In several works, deep learning have been developed to compute the residuals and to detect sensor or actuator faults for different types of systems (linear, nonlinear electrical, hydraulic, etc.), as presented in [33, 34].

The main contribution of this paper is the proposal of a real-time fault detection and localization approach applied to handwriting device allowing producing cursive letters and geometric shapes from electromyographic signals of only two forearm muscles. As already mentioned, these signals are subject to uncertainties and internal/external disturbances.

In this sense, a predictive model is particularly challenging for the estimation of faulty unpredictable inputs. This model allows connecting inputs (IEMG signals)/outputs (pen tip coordinates) data as a parametric model for Multi-Inputs Multi-Outputs (MIMO) system. Afterward, Recursive Least Squares (RLS) algorithm is used to identify the model’s parameters. Our focus, hereafter, will be the use of normalized residue to detect in real-time different kinds of actuator fault arising on the studied device.

Unlike previous fault detection studies, developed for myoelectric prosthesis and devices ensuring simple movement (opening, closing), the developed approach is proposed for handwriting system allowing us to generate cursive and complex pattern, especially cursive Arabic letters, ح, « HA» and س, «SIN », which are composed by combined movements, vertical, horizontal, oblique, etc.

On the other hand, our study is considered as a model-independent abrupt or intermittent fault detection method and as an alternative solution to the unpredictable input observer based techniques, without any observability requirements.

Indeed, the unpredictable characteristics and the variation of the biological parameters, related to the handwriting process, like muscle mass and muscle fatigue, which interfere with physical and psychical variation of scripters, make the task of modeling complex and tedious. Therefore, free models approaches are preferable in this case. The proposed fault detection method is part of this category of model.

In addition, the developed fault detection approach allows detecting several types of faults in one or two inputs signals and in the same or different instants. These faults may be due to technical problems (defective electrode), physical (sweat), or even damage to the components that constitute the system to be supervised.

The present paper is organized as follows: after presentation of the handwriting experimental approach in the second section, the third section focuses on the study and classification of different kinds of faults that can be detected during the handwriting act. The fourth section shows the development of actuator online fault modeling approach using damaged outputs to detect faulty inputs.

2. Experimental Approach

Handwriting movement, on the plane (x, y), is considered as a complex movement based on two electromyography signals, EMG1 and EMG2, of the most active forearm muscles, namely, the “Abductor Pollicis Longus” and the “Extensor Capri Ulnaris” [5, 35]. The first muscle is responsible for the vertical displacement and the second one for the horizontal motions.

In this sense, an experimental approach was proposed, in Hiroshima City University, to record at the same instant cursive Arabic letters or geometric forms and two forearm EMG signals [35].

In fact, the measuring data were synchronized by sending a step signal from the parallel interface port on the computer to the data recorder. This experimentation had required the following equipment:(i)Digital table of the brand “WACOM, KT-0405-RN”.(ii)Preamplifiers “TEAC, AR-C2EMG1”.(iii)Data recorder “TEAC, DR-C2”.(iv)Bipolar surface electrodes (MEDICOTEST, Blue Sensor N-00-S).(v)Computer.

Figure 1 indicates the positioning of electrodes on the writer’s arm. Electrodes, indicated by “ch1”, are relative to the first muscle and those relative to the second muscle are indicated by “ch2” [5].

In Figure 2, the recorded data for the Arabic letter, ح, “HA” are presented [35].

However, it is difficult to get the useful information from muscles activities. Therefore, a variety of signal processing techniques are used to make EMG waveforms easier to interpret. Indeed, the fluctuation of EMG’s magnitudes can be filtered to obtain new curves called integrated EMG (IEMG), represented by dotted red curves in Figure 2(b) [35].

We note IEMG1 and IEMG2, the integrated EMG1 and EMG2 signals, respectively.

The studied system is sensitive to different types of faults due to many problems: electronic, mathematic, biologic, etc. Electromyography signals are directly related to several morphological and technical properties that influence the quality of these signals (filtering method, noise and disturbances, distance between the muscles and the electrodes that are the sensors of the muscular activities, etc.).

An undesirable deviation could occur on the written shape if a fault appears on the handwriting process inputs. To conclude, the fault that damages EMG signals affects outputs of the handwriting model, and this will further make the studied system reconstructing faulty graphic traces.

Figure 3 shows the impact of faulty inputs on the quality of writing. Its presents some faulty responses of two cursives Arabic letters, ح, “HA” in Figure 3(a), س, “SIN” in Figure 3(b). The experimental recorded data, in fault free case, are represented by discontinuous lines. The model’s outputs, responses for faulty inputs, are presented by a continuous line [6].

However, electromyography signal is considered as a complex signal with the particularity of variation according to the writers, the nature of the produced form, and even the variation of physical and psychical state of an individual. In summary, electromyographic signals are very sensitive to several distortions, so a fault detection approach is necessary to ensure a good functioning of the studied process.

In this sense, we propose the new fault detection approach introduced in Figure 4, based on predictive model using parametric identification model for the reconstruction of not healthy inputs (IEMG signals) of the handwriting assistance system using the writing coordinates on (x, y) plane.

The proposed technique is applicable for several kinds of faults and can be useful in a lot of applications, such as hand prosthesis, smart electrodes maintenance, and military applications.

3. Faults Study and Classification

Before describing the proposed fault detection approach, we will present, in this section, the most frequent faults that can occur during the writing. Indeed, we can propose 3 main classes of faults [3649]: technical, neurogenic, and modeling class.

(i) In the case of the technical class, faulty IEMG signal presents decrease/increase in the IEMG amplitude from a nominal one. We can cite, for example, muscles-electrodes distance, distance between electrodes and alignment to the fiber direction, displacement of electrodes from their initial position, and transmission cable.

(ii) In the case of neurogenic or myogenic class, related to neuronal and muscular problems, the faults lead to record a poor muscle activity signal with an increase of the IEMG signal amplitude from a nominal one, such as variation of the muscular temperature, morphologic variation, weariness of the muscles, and moving of electrodes during the writing act.

(iii) The case of modeling class is related to modeling problems, as variation of model parameters.

4. Faulty Inputs Estimation from Outputs

In this study, we are interested in studying faults of technical class. Knowing that most of these faults are related to IEMG signals, we choose to consider all distortions as actuator faults.

Figure 4 describes the studied approach that is based on a mathematical model (Block 1), developed by Chihi et al. [5], allowing us to mimic the handwriting motion. Otherwise, this model estimates coordinates of a drawing shape on the plane (x, y) from IEMG signals. However, the performance of this model depends on the quality of IEMG signals, recorded by surface electrodes. As we already mentioned, these signals can present several faults and distortions.

The objective of the proposed fault detecting system is to detect and to localize a fault in the measured IEMG signals, inputs of the considered assistive handwriting system.

As it is presented in Figure 4, the main idea for detecting faults during this complex process is to propose a predicted model (Block 2) to estimate faulty inputs (IEMG signals) only from the coordinates of the written shape on the plane. The fault diagnostic algorithm is also based on the analysis of the absolute difference (Block 3) between the estimated em1 and em2, and the referential data, er1 and er2, of the forearm IEMG signals. These signals are recorded in good conditions without faults. A residue computing procedure is then proposed to define r1 and r2 (Block 4), which are used to decide and to generate fault warning (Block 5).

The residue, rsi, , is compared to a threshold value to detect and isolate the faults. Threshold is a parameter to be adjusted according the writer. In fact, each person presents his or her own reference IEMG signal, i.e., certain range of amplitude.

We note that rsi is the absolute difference between the faulty and the normal electromyography signal (5) and ri is the normalized residue (6).Figure 5 presents the flowchart of the proposed fault detection algorithm, based on the absolute residual computation which further allows defining a normalized residue to better present the instant of the fault appearance and its duration.

The proposed predicted model of faulty inputs (block 2 of Figure 4) is a MIMO fourth order model, whose inputs and outputs are interconnected. This model estimates both IEMG signals at the same time. It is based on velocities of the writing, Vx and Vy, according to x and y coordinates, respectively. In fact, velocity of writing has inspired many researchers to solve various problems related to handwriting, such as the study of the rapid human movement [50], pattern recognition [51, 52], and even the handwriting modeling and control [5355].where, are the velocities of faulty coordinates, x and y, respectively,, are the estimated IEMG signals,, , , and , , , are the estimated parameters, relative to em1 and em2, respectively.

The model parameters identification is based on classical Recursive Least Squares (RLS) algorithm with forgetting factor equal to 0,95 which performs the following operations to update the parameters of the researched model [5558]. where is the vector of estimated parameters, is the adaptation matrix, is the actual output of the system to be identified, is the observation matrix, is the estimated error.

The model structure used to identify the handwriting system dynamics for Multi-Inputs Multi-Outputs is also given as follows [5558]:where are the error vectors, relative to and signals, respectively, are the matrices whose elements are relative to and signals, respectively.

4.1. Simulation Results

In order to validate the proposed fault detection strategy, several simulations were performed in Matlab/Simulink with real measurement files of EMG signals and its corresponding pen tip movement coordinates.

We note that ed is the faulty input signal, which summarizes the frequent technical faults that could attempt the handwriting process and presents the model of each one [3649].

4.1.1. Case 1: One Faulty Input

In a first analysis, we consider that one fault can affect only one IEMG signal, measured using surface electrodes. Therefore, we propose studying these kinds of faults: step, decrease of amplitude, ramp, random, and intermittent.

For Figures 68, we note that the estimated IEMG signals are presented in red line with stars and real signals are in continuous blue line. Curves (a) present IEMG signal without fault (blue continuous line) and the faulty input (red line with stars). Curves (b) show the real (blue continuous line) and the estimated faulty inputs (red line with stars) using model (3).

(i) Step Fault. Step fault, expressed by (9), is mainly caused by the distortion of electrodes that can stay blocked by generating a constant amplitude, Figure 6. f is constant,tk is the discreet instant,tf is the instant of occurrence of the fault,e is the not healthy input.

Figure 6(a) shows the difference between the estimated signal (represented by red stars curves) and its corresponding referential data (represented by continuous blue curves). Figure 6(b) presents the ability of the proposed approach to predict IEMG signal, despite the abrupt changing of the studied signal.

(ii) Decrease of the Amplitude Fault. In this study, we take into consideration the disadvantages of most myoelectric prostheses, which are summed up in the variation of the distances between electrodes or between muscles-electrodes, the alignment to the fiber direction, etc.

The impact of these faults on IEMG signals can be represented by a decrease of the amplitude, Figure 7.

This type of distortion can cause problems in the control and precision of the prosthesis, which is very embarrassing for the user.

The normalized residue ri is equal to one during the fault occurrence. Figure 8 presents the residue relative to the cases step and decrease of the amplitude faults.

(iii) Ramp Fault. The distortion due to the aging of the electronic components can be represented by a ramp function (10). Figure 9 shows good estimation of this type of fault using the proposed approach.

(iv) Random Fault. The faulty input is also estimated with the presence of a random distortion which can be caused by a technical problem in the transmission cable, or an unknown external interference, etc. Generally random fault is difficult to estimate; however the proposed approach shows a good concordance between the real and the estimated faulty signal, Figure 10. The normalized residue is illustrated by Figure 11.

(v) Intermittent Fault. The estimation of inputs with an intermittent fault (11) is also taken into account. In effect, micro cuts, instantaneous displacement of electrodes, or even unpredictable changes in IEMG signals can be considered as an intermittent fault (ramp, step, random, etc.).tf1 and tf2 are instants of the beginning and the end of the fault.

Figure 12 shows good concordance between the real inputs and the estimated signals. The normalized residue is equal to one during the fault occurrence, Figure 13.

4.1.2. Case 2: Tow Faulty Inputs

In a second analysis, we suppose that both inputs are sensitive to two successive faults.

In order to show the relevance of the proposed approach, we start with the case where both IEMG signals have only one fault, Figure 10(a). Figure 10(b) presents the estimated and the real not healthy signals in the case of two faults in each electromyographic signal.

The normalized residues give information on the time of occurrence of the fault and its duration. Residues of each input are different to zero in the presence of any kind of fault, Figures 14(a)′ and 14(b)′.

To evaluate the proposed fault detection approach, one or more IEMG signals in the testing data set were artificially distorted. Therefore, step, ramp, intermittent, and random faults were adjusted to simulate the different disturbances commonly occurring at EMG signal. The proposed approach shows good concordance to estimate faulty recorded muscles activities from damaged shapes coordinates on the plane.

In order to make handwriting practical and available to people with motor deficits, real writing challenges resulting in deformation in EMG signals must be overcome. This work aims to address these challenges by proposing a fault detection approach to mimic the behavior of individual distorted EMG signals.

The results of this study demonstrate several benefits of the designed fault detection approach, such as the adequacy to correct for sudden signal disturbances simulated in the present study and the possibility of detecting one or multiple distorted EMG signals.

The proposed fault detection approach is considered as an alternative solution to the unknown input observer based techniques allowing model-independent fault detection without any observability requirements.

5. Conclusion

The present work developed a new model-free fault detection technique for unpredictable biological inputs. For this, we presented a classification study of the different kinds of faults that can affect the writing process. Then parametric identification model with RLS algorithm was proposed to reconstruct faulty integrated EMG signals. Thereafter, from the fault reconstruction results, a normalized residual is computed to facilitate the fault detection procedure from damaged system outputs.

The proposed fault detection method showed high performance in detecting faults of different types and for one or multiple distorted EMG signals.

These promising outcomes could inform the design of clinically viable EMG estimation and eventually benefit individuals with motor deficits.

Data Availability

The [EMG signals and the corresponding letters coordinates] data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.