Abstract
Maintainability is one important attribute of product quality. Maintainability evaluation is important to reduce maintenance cost and improve availability. With this research work, an effective maintainability evaluation method is proposed by mingling virtual environment and real world physical equipment. In order to reduce the cost of maintainability test scenario construction, the surrounding environment of physical equipment is replaced by virtual model for virtual-real fusion. The virtual environment is mainly used to simulate the impact on maintenance operation space. The difficulty of realizing the test concept deals with how to accurately identify the pose of the physical equipment and register the virtual environment around the physical equipment. In the proposed method, initially the ORB feature of the physical product is extracted through binocular vision. Secondly, the ICP method is used to match the physical product feature with the digital prototype feature, so as to identify the relative pose of the physical equipment. The virtual maintenance environment is then accurately superimposed. Thirdly, the experimental evaluation method of virtual reality fusion maintainability qualitative and quantitative indicators is formulated. Finally, taking an engine as an example, a virtual-real fusion maintainability test case study is carried out to verify the effectiveness and feasibility of maintainability evaluation based on virtual-real fusion test scenario. With 5 test groups, evaluation is carried out in three scenes-real environment, virtual-real fusion and without surrounding environment. Results of the evaluation revealed that error rate of the proposed virtual reality fusion maintainability method are somehow closer to maintainability in real environment. The average error rate of the three maintainability qualitative indexes in the virtual-real fusion scene is 9.90%, whereas the average error of the maintenance time is 3.25%. The proposed system offers an absorbing simulation environment where users are privileged to interact with the maintenance objects. The research work bears great significance in maintenance, product designing, visualization and related domains. As maintenance in real environment happens to be costlier and hazardous, the proposed method is suitable to ensure substantial cost reduction and safety in all stages of maintainability. Moreover, outcomes of the evaluation and the generated data of simulation can be generalized to improve maintainability of equipment.
1. Introduction
Maintainability evaluation is important to reflect whether product maintenance is convenient, fast and economical [1]. In order to ensure that a product has high availability and low life cycle cost, the product must have good maintainability. This is to reduce the maintenance requirements for manpower, time and resources [2,3]..Therefore, during the development process of industrial products, sufficient maintainability tests must be carried out to verify and evaluate their maintainability and to ensure that they meet the required maintainability requirements.
The traditional method of physical maintainability evaluation relies mainly on physical prototype. Such prototypes are not only expensive but are impractical in some scenarios [4]. The method of virtual maintainability simulation evaluation using digital prototypes is difficult to design. Due to the difficulties involved in human-machine force interaction, such methods often fail to accurately evaluate the maintenance force characteristics and maintenance time indicators. However, with virtual-real fusion, real world and virtual world can be represented in one single system at a time. With the fusion, a synthetic scene is augmented by realist objects and information extension is offered for real scenes. In the field of maintenance and assembly, the application of virtual-real fusion has made a considerable progress. Deshpande [5] designed an AR-assisted visual feature system with interactive modes for Ready-to-assemble (RTA) furniture. The application utilizes Microsoft Hololens™ headsets to enable users in timely conceiving the spatial relationship of components. The system also supports assembly tasks that otherwise require high spatial knowledge. For the first time in the literature, a number of users examined the system on RTA furniture. Vicomtech proposed the creation method using an AR workspace. The system offers interaction and visualization modes and provides more effective means for the assembly task of hybrid man-machine production [6]. It is proved that virtual-real fusion based maintainability test is advantageous in terms of accuracy and economy. With the nexus of virtual scene and real objects, a satisfactory reduction in hardware scale is promised. Hence, the virtual-real fusion based maintainability test offers a high application prospect. The key issue in such systems is the integration of physical equipment and virtual environment in accordance with the actual positional relationship. The three-dimensional pose of physical equipment must be accurately identified and need to be properly superimposed in the virtual environment. This paper focuses on accurately identifying positions of physical equipment and appropriate superimposing of real objects in a synthetic environment. Moreover, the paper presents applications of maintainability test in a systematic way. The key issues involved in the maintainability assessment are also discussed to pave the way for an optimize product design.
Rest of the paper is organized into 4 sections. In Section 2, related work from the literature is discussed. Overall solution of the proposed method is covered in Section 3. Section 4 deals with technological implementation while experimental verification is presented in Section 5. Conclusion and future strategy of the research is discussed in Section 6.
2. Literature Review
Maintainability refers to the possibility of restoring a failed item to operative condition within a given period of time [7]. Like proper planning, maintainability is important in product designing in general and in complex products particularly [8]. An appropriate maintainability scheme not only reduces costs but also reduces the chances of incidents and accidents in the product life cycle. It is why researchers are emphasizing to pay due consideration for planning and designing of maintainability testing [9]. Since last decade, the use of Virtual Reality (VR) technology is on the rise in the realm of maintainability. VR is of great significance in product designing and manufacturing sectors [10]. The technology enables users to interact with virtual objects while employing sensual perceptions. A system that utilizes VR for the enhancement of maintainability design is referred to as Virtual maintenance (VM). VM is a sort of virtual simulation-based engineering application that enables engineers to plan, analyze and evaluate the assembly of mechanical systems [11]. VM should satisfy several functional requirements [12]. The desktop based system, Virtual Maintenance Simulation of the Nanyang Technological University [13] is a novel sequence planning technology. The system uses an optimization algorithm for effective and feasible planning. The ergonomic based assessment method of Sanchez [14] aims to assess the ergonomics principals in a workplace. Different Software application is proposed by some researchers [15] for design, analysis and verification of effective maintainability. Integrating the technologies of Virtual and Augmented Reality, a system based on remote handling (RH) is proposed in [16]. It is claimed that the system helps the operator in maintenance of the Tokamak. It is added that the system ensures that all the processes of inspections and maintenance are performed in a safe environment. In [17] a 3D workforce training system is proposed for high-voltage power line maintenance. The study offers a risk-free environment for learning the practice and procedures related to line maintenance. The research also evaluated the tool findings of which suggest that the workforce training system has a positive effect in the overall process of maintenance. As suggested in [18], the mingling of virtual reality technology with the traditional approach of maintainability has a positive impact in product design optimization. However, the use of advance technology in maintainability evaluation is rarely focused, hence this research.
3. Overall Solution
In the process of a virtual-real fusion maintainability test, usually a set of digital prototypes of the product is provided. The set of prototypes is treated as the key input to the test. The digital prototypes reflect the relationship between the physical product and the surrounding environment. In order to superimpose the virtual maintenance environment model on the periphery of the physical product object it is necessary to identify the physical product. Moreover, to make it sure that an object is a part of the maintenance environment, the virtual world should be fully aligned with the physical world. Since the current AR glasses have the function of maintaining anchoring after superimposing virtual scenes, the focus of virtual registration is how to achieve initial recognition and registration. Once the registration is successful, they can be anchored so as to adapt to the posture changes of the person and the binocular glasses during the test. In this paper, the binocular camera is used to obtain the video stream of the real maintenance scene. The characteristics of the video image are extracted on the basis of calibrating the internal parameters of the camera. The transformation matrix is solved for pose estimation. Next, the virtual scene is registered to the real scene through coordinate transformation to complete the construction of virtual-real fusion maintainability test scene. The overall process is shown in Figure 1.
4. Technology Implementation
This section is divided into subheadings that provide a concise and precise description of the experimental results, their interpretation, as well as the experimental conclusions that can be drawn.
4.1. Image Feature Extraction of Maintainability Test Object Based on ORB
At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in the fields of image matching and object recognition [19]. Since the objects involved in the process of maintainability test are usually of a mechanical product, their surfaces sometimes lack rich texture features. Considering the stability and rapidity based on feature point extraction and matching, the ORB local feature is selected in the proposed approach. ORB local features use FAST as the feature point detector, the improved BRIEF as the feature descriptor, and use the BF pattern matching algorithm for feature descriptor matching [20].
FAST feature points are not directional, and the directional parameters are determined by obtaining the center of gravity of the feature point neighborhood. The neighborhood moment is given as:where is the gray value at point , , is the radius of the circle, and are non-negative integers. When is 1 and is 0, the value of in the direction can be obtained. Similarly, if is 0 and is 1, the value of in the direction can be obtained. being the image center of gravity can be obtained as:
The angle between the feature point and the center of gravity is defined as the direction of the FAST feature point, given as:
ORB extracts the BRIEF descriptor according to the direction parameters obtained in the above formula. However, due to environmental factors and the introduction of noise, the direction of feature points will change. Furthermore, the correlation of random pixel block pairs will be relatively large, thereby reducing the discrimination of the descriptor. ORB adopts a greedy algorithm to find random pixel block pairs with low correlation. Generally, 256 pixel block pairs with the lowest correlation are selected to form a 256 bit feature descriptor. The two descriptors are given as follows,
4.2. Matching of Physical Equipment Characteristics and Virtual Environment Registration
The ORB feature set is extracted from the real maintainability test object and the virtual maintenance environment model. The corresponding feature descriptors are obtained for onward processing. The similarity between two ORB feature descriptors is characterized by the sum of the exclusive ORB Hamming distances:
The smaller the , the higher the similarity, and the greater the probability that the two describe the same feature. Conversely, the lower the similarity, the more likely they are not describing the same feature.
The BF matcher is used to get all the possible matching feature pairs, assuming that the minimum Hamming distance of feature pairs is MIN_DIST. In order to select the best matching pair and improve the operating efficiency, an appropriate threshold is selected and the matching pair smaller than the threshold is selected for the next camera pose estimation. The threshold value cannot be too small, as it will affect the final effect. It is therefore necessary to select the best threshold value through experiments on the image frame.
Given the point in , find the point with the shortest Euclidean distance of from , and take and as the corresponding points to obtain the transformation matrix. Through continuous iteration, the following formula is minimized and the iteration is terminated. Finally the most Optimal transformation matrix is obtained to make them coincide.
In the formula, indicates the rotary transform matrix and indicates the translation form of the matrix.
The essence of the ICP algorithm is to calculate the transformation matrix between the feature sets, minimize the registration error between the two through rotation and translation and then achieve the best registration effect[21]. Assuming the two feature point sets and , the registration process using the ICP algorithm is introduced below:(1)Sample set , , represents a subset of the set ;(2)Search in set , find the closest point to each point in , and get the initial correspondence between and ;(3)Remove the wrong corresponding point pairs using algorithms or constraints;(4)Calculate the transformation relationship between the two according to the corresponding relationship in step (2). Minimize the value of the objective function and apply the calculated transformation matrix to to obtain the changed matrix ;(5)Determine whether the iteration is terminated according to . If is greater than the preset threshold, return to step (2) to continue the iteration. If is less than the preset threshold or reach the set number of iterations, the iteration stops.
By obtaining the transformation matrix through the above steps, the pose transformation relationship between the physical equipment and the virtual maintainability test environment can be obtained. Following that, virtual registration can be performed to complete the construction of the virtual-real fusion maintainability test environment.
4.3. Maintainability Index Evaluation Based on the Fusion of Virtual and Real Information
In the process of the virtual-real fusion maintainability test, the main maintenance test operation is carried out on physical products and the maintenance environment rendered by the virtual prototype can well simulate the spatial characteristics of the maintenance process. Therefore, it is reasonable to believe that the virtual-real fusion maintainability test scene is basically equivalent to the actual maintenance scene. This in turns prove that the propose test satisfies the standards of accurate maintainability evaluation. The maintainability qualitative indicators to be evaluated usually include visibility, accessibility, operating space and comfort, etc. which are considered to be the standard qualitative indicators [22]. In order to evaluate more finely, it is necessary to develop a reasonable evaluation grade method. The quantitative indicators of maintainability include Mean Time to Repair (MTTR), mean preventive maintenance time, maintenance ratio, maintenance degree and repair rate, etc. Next, we mainly introduce the evaluation methods of some of the main indicators.
4.3.1. Maintenance Visibility
Maintenance visibility refers to the visibility of the maintenance parts during maintenance. The good visibility of the maintenance parts is convenient for maintenance personnel to observe the service condition of the components and to complete maintenance tasks easily by reducing the difficulties involved. On the contrary, if the part to be repaired is invisible or difficult to see, it will greatly increase the difficulty of the maintenance work. Moreover, fatigue in maintenance will increase that will badly affect the completion of maintenance tasks. The values corresponding to the subjective evaluation of visibility index are shown in Table 1.
4.3.2. Maintenance Accessibility
Maintenance accessibility refers to the degree of difficulty a maintenance personnel faces while accessing the maintenance part during equipment maintenance. The accessibility can reflects the impact of work space, maintenance approach, and component layout on maintenance operations. Accessibility directly affects equipment inspection and maintenance. Therefore, full attention should be paid to maintenance accessibility. The values corresponding to the subjective evaluation of accessibility index are shown in Table 2.
4.3.3. Operating Space
The operating space is related to the feasibility and simplicity of the maintenance. An effective operating space is mainly guaranteed by the installation and layout of equipment or parts. The equipment must have a certain space to facilitate maintenance operation. Similarly, the values corresponding to the subjective evaluation of operating space index are shown in Table 3.
4.3.4. Mean Time to Repair
Mean Time to Repair is the average value of the actual repair time required for troubleshooting. It is a basic parameter of the equipment maintainability, also known as the average repair time. The parameter is usually measured by the cumulative time estimation method [23].
In the cumulative model, the basic maintenance operations are synthesized into maintenance activity time . The maintenance activity time is synthesized into the average repair time of each main replaceable unit under each fault detection and isolation output ( represents unit and represents output ), represented as:where is the number of activities for troubleshooting. The maintenance after the has a fault and is detected by the including various maintenance activities, i.e. preparation, isolation. It may include operations of other detected in the fault isolation result (example, determine the , with alternating replacement).
is the average time of the troubleshooting and maintenance activities for the detected by the , output. The mean value model to solve the average repair time is given as;
5. Experimental Verification
Taking the auxiliary engine room of a ship as a case study, the test verification is carried out to verify the correctness and applicability of the proposed virtual-real fusion maintainability test evaluation method. The auxiliary engine room is powered by a diesel engine, which is composed of a crank connecting rod mechanism, a gas distribution structure, a fuel system, a lubrication system, a cooling system and a starting system, etc. The engine needs to replace consumable parts such as fuel filter and air filter. The cylinder and starter motor have a certain failure rate. It needs to be well designed for maintenance to ensure rapid maintenance at the crew level.
In the ship cabin environment, the equipment maintenance process has a certain level of complexity. The other equipment around the peripheral pipelines and cables are easy to cause insufficient accessibility of the maintenance objects and inadequate operating space. Therefore, in the process of maintainability test of the engine, it is necessary to be able to simulate actual cabin maintenance scenes and maintenance space. Moreover, the impact of various operational obstacles on maintainability needs to be fully considered so as to obtain more accurate maintainability test results.
Since for the establishment of a 1 : 1 full-physical maintainability test condition is very costly and has a long cycle, therefore, the proposed virtual-real fusion maintainability test evaluation method is adopted. A small part of the physical equipment and an extended number of virtual scenes are used to realistically simulate a complete test scenario. During the evaluation, the test conditions used include the YN92 physical diesel engine and the complete digital model of the auxiliary engine compartment, as shown in Figures 2 and 3. To verify the outcomes, repairing and replacement of the starting motor were taken as examples.
(a)
(b)
5.1. Verification of the Establishment Method of Virtual-Real Fusion Test Scene
(1) In order to build a realistic virtual-real fusion maintenance scene, it is necessary to consider the impact of multiple factors on the registration accuracy of the virtual environment. The feature extraction method is an important factor affecting the registration accuracy. Therefore, firstly, the feature extraction and recognition of diesel engine were carried out. Different feature extraction methods have different feature extraction results. The feature extraction of the same object (diesel engine) is performed using the standard methods of SIFT, SURF, and ORB respectively. The results of the feature extraction of the diesel engine are shown in Figures 4–6.
The data results of the three methods for feature extraction are shown in Table 4.
Through experimental analysis and comparison, the feature points detected by SIFT, SURF and ORB are 502, 454 and 1023 respectively under the same experimental conditions. The feature points matched by SIFT, SURF and ORB are 112, 168 and 136 respectively. It can be found that although the number of feature points matched by the three methods is roughly the same, the time required for ORB matching is significantly shorter, guaranteeing a higher operation efficiency.
(2) To ensure the registration accuracy of the virtual environment, it is also necessary to consider the matching effect when the physical object and its corresponding model are not completely consistent. Therefore, the model of the physical object in the CAD environment is modified. The matching accuracy is calculated and analyzed on the basis of re-extracted feature points. Figure 7 is a comparison display of the equipment model after removing some parts.
(a)
(b)
(c)
(d)
(e)
(f)
The results of matching after removing some parts from the equipment model are shown in Table 5.
It can be seen from the above table that the more similar the model and the real object are, the higher the matching accuracy is. When registering the virtual environment, we should not only ensure the registration accuracy, but should also prevent the model and the physical object from any such inconsistency that may lead to registration failure. Therefore, an appropriate matching threshold must be selected. Through comprehensive analysis, it can be seen that the threshold 2 is more appropriate, which can guarantee a certain registration accuracy and avoid registration failure.
Incorporating the above two algorithms into AR glasses, an immersive three-dimensional visual information of the physical equipment through the binocular lens of the glasses is obtained. Feature extraction is then performed and matching is performed the virtual model one by one. The resulting virtual-real fusion ship cabin repair scene is shown in Figure 8.
5.2. Maintainability Test Operation and Result Analysis
Next, according to the established virtual-real fusion maintainability test scene of YN92 physical diesel engine, the maintainability operation test of the replacement of the starting motor is carried out. The tester wears the AR glasses to carry out maintainability test operation and to obtain basic test data. The operation steps of the maintenance process are shown in Figure 9.
A total of 5 groups of tests are carried out, and each group of tests is carried out in three scenes of real environment, virtual-real fusion and without surrounding environment respectively, as shown in Figure 10–12. The Maintainability (visibility, accessibility, operation space and operation time) results of the maintenance tests are shown in Table 6.
The computed average error of the three maintainability qualitative indexes in the virtual-real fusion scene is 9.90%, and the average error of the maintenance time is 3.25%. The error of the three maintainability qualitative indexes in the scene without surrounding environment is 22.29%, and the average error of the maintenance time is 9.45%. It can be found that the relative error of virtual reality fusion maintainability evaluation results is significantly reduced as compared to the results of the tests without surrounding environment. In the virtual-real integration maintenance test, the maintenance personnel perceive the existence of the surrounding cabin equipment through vision. During maintenance, in order to avoid collisions with the virtual cabin equipment, the bending angle and the movement range of arms were kept smaller. The posture of the maintenance personnel was also adjusted accordingly to be closer to the real maintenance situation. This was to ensure a reduced maintainability evaluation error. The main reason for the errors in the virtual-real fusion maintenance test is the psychological factors of the maintenance tester. It is also possible that the maintenance personnel passed through the virtual environment and did not correct the action in time. From the stated outcomes it is easy to deduce that the virtual-real fusion maintainability evaluation method takes into account the influence of the surrounding environment, and obviously enhances the accuracy and credibility of the test evaluation results.
6. Conclusions
This paper proposes a method of constructing a maintainability test scene based on the fusion of virtuality and reality for maintainability evaluation. The ORB feature of the equipment is extracted based on binocular vision. The ICP method is used for feature matching and recognition according to the feature extraction results, while the virtual environment is registered to complete the construction of virtual reality fusion maintainability test scene. Experiments show that the use of ORB features effectively extract equipment features with high speed and promising precision. The ICP method can be used to realize the registration of the physical object and the virtual environment, thereby completing the registration of the virtual environment. The maintainability test was carried out and evaluated in the built virtual-real fusion test scene. The results show that the surrounding virtual environment has a certain impact on the maintenance process, and the maintainability verification is closer to the maintenance process in the real maintenance environment.
The virtual-real fusion maintainability test method studied in this paper provides a novel and efficient method for simulating the real maintenance performance of the products under complex maintenance conditions. The method can be carried out for any operations on real objects and can effortlessly simulate the spatial characteristics at low cost. Moreover, the approach is suitable to perform accurate and timely index evaluation of visibility, accessibility and maintenance time. During the experimentation it was observed that psychological factors were involved in the maintenance process. Some of the reported errors were mainly due to the psychological factors. As our future strategy, we are planning to extend the research so that to trace out the psychological factors contributing in high error rate in the maintenance operations. This will help in avoiding the errors and tolerating the psychological.
Data Availability
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Authors’ Contributions
Conceptualization, Z. G. and Y. Z.; methodology, Z. G.; software, Y. Z. and Q. L.; validation, Y. Z. and Q. L.; formal analysis, Z. G.; investigation, Y. Z.; resources, Z. G.; data curation, Y. Z.; writing—original draft preparation, Y. Z.; writing—review and editing, Z. G. and Q. L.; visualization, Y. Z. and Q. L.; supervision, Z. G. and Q. L.; project administration, Z. G.; funding acquisition, Z. G. All authors have read and agreed to the published version of the manuscript.
Acknowledgments
This research was funded by 14th Five-Year Ministries-level Pre-research Project, grant no. 50904050201.