Abstract

In the current study, an accurate strategy was developed to detect the surface damage of the strawberries, because the existing indicators for evaluating surface damage are single, resulting in low accuracy of judgment for damage detection. Strawberry surface damage can be divided into three specific categories: outline depression, surface oxidation, and surface damage. The methodologies employed in this study are experimental analysis and comparative validation. The outline, area of surface damage, and surface reflectance of strawberries were analyzed using visual and hyperspectral data. Principal component analysis was used to evaluate the damage characteristics comprehensively. A four-finger rigid manipulator having one degree of freedom was selected, and 280 samples were analyzed (the manipulator grasped 85% of them, and 15% were not treated). The accuracy of damage detection based on the outlines, area of surface damage, and surface reflectance of strawberries was 88.75%, 91.25%, and 75%, respectively. The evaluation method proposed in this paper improved detection accuracy by 10.79%, 7.76%, and 31.1%, respectively. Therefore, this method could contribute to the design of manipulators to further improve fruit production efficiency.

1. Introduction

Manipulators are widely used in fruit picking and automatic production line to improve production efficiency [13]. Berry fruits are easily damaged when grasped by a manipulator, which limits automated picking [4, 5]. Researchers have investigated ways to make manipulators suitable for picking berry fruits, such as changing the material of the manipulator or designing a sensing system that can control the contact force between the manipulator and the surface of the fruit ([69]; W. [1012]). However, the inability to effectively assess complex surface damage is one of the main problems affecting the application of manipulators in fruit picking.

The cause of fruit surface damage can be divided into three categories: natural factors, external forces, and other external conditions. Typically, the natural factors include shriveling, cracking, and skin spots. The external force was generated from the contact force between the manipulator and the fruit [13, 14]. The other external conditions involve rapid depressurization [15], food-based attractants [16], vertical pressure [17], and so on. Physical characteristics (e.g., deformation, abrasion, cuts, and bruises) and chemical elements are used to assess fruit damage [18].

Sensors and cameras are commonly used to assess physical characteristics. The surface pressure and image defects are the primary evaluation indices. For the surface pressure, Piotr, Roman, Łukasz, & Daniel [19] used a Tekscan® measuring system to determine the bruise resistance; they measured the contact surface between tested fruit and fixed material. The results indicated that the relationship between bruise volume and surface pressure could ensure precise bruise resistance evaluation based on the verifying linear regression analysis. Komarnicki, Stopa, Szyjewicz, Kuta, & Klimza [20] analyzed the contact pressure at the point of contact of an apple falling on a fixed hardboard from various heights and explained how to reduce surface damage of fruits during transportation. However, the pressure-sensitive film technique requires a complex sensing system and needs to contact the fruit surface in the measurement process, which can cause secondary damage. Image processing is also commonly used to detect fruit surface damage. For example, Blasco, Aleixos, Gómez-Sanchís, & Moltó [21] developed a robot vision system to detect apple surface defects. The system used three different threshold segmentation routines and one based on artificial neural networks and principal components. This method is suitable for damage detection when there is a noticeable color change on the damaged parts caused by oxidation.

In addition to physical indices, based on the effects of contact force on the biochemistry and cell microstructure of the fruit surface, some chemical indicators are also used to investigate the degree of damage to the fruit surface. For example, several studies have investigated electrolyte leakage from the peel, polyphenol oxidase, and titratable acidity (TA) of pomegranate and mango [22]. Compared with physical indices, the accuracy of the chemical index is higher, but the testing process is more complex.

Different indicators have been used to analyze fruit surface damage under various external pressures to improve the accuracy of detection. Grasping with a manipulator could cause several types of surface damage on fruits, e.g., depression, surface oxidation, and skin bruising [2325]. When using a single index to evaluate damage; some aspects may be omitted. Therefore, several indices need to be assessed. There are limited studies in this area. Wang et al. [26] used multiple factors (e.g., contact area, average surface pressure, and compressed volume) to evaluate the damage. However, this method is inevitable to contact with fruit, which may cause surface damage.

In this paper, the shortcoming the study aims to close is that the existing indicators for evaluating strawberry surface damage are only one, leading to low judgmental precision for damage detection. We carried out the grasping experiments and suggested an extensive evaluation index of strawberry surface damage based on vision and hyperspectral data analysis. The method of evaluating surface damage on strawberries was proposed based on outline depression, surface oxidation, and skin bruising, and hyperspectral data were analyzed. The accuracy of the detection methods was evaluated by assessing strawberries with different maturity, and strawberries picked with other varying parameters of the manipulator. This paper verified the feasibility of the multi-index evaluation by comparing it with the single-index evaluation.

2. Materials and Methods

2.1. Preparation of Strawberries

Unblemished and regular-shaped strawberries, called Mingjing Strawberry were purchased in Beijing, China. These strawberries were stored in a thermostatic-humidistat cultivating box at a temperature of and humidity of 90%–95%. These strawberries were divided into two categories: S1 was bought from the supermarket and S2 was the same batch as S1 but left for three days to obtain a softer texture with higher maturity. The strawberries of S2 were more fully red, indicating a smoother texture after extended storage [2730]. The mass of strawberry samples needs to be 18 to 22 g; the length is 26 to 31 mm; the diameter is between 18 and 22 mm.

2.2. Grasping Experiment

The manipulator designed to pick strawberries is shown in Figure 1. It consists of a motor (model, N20; rated voltage, 12 V; rated speed, 100 rpm), a fixed bracket, and four symmetrically distributed fingers. For each finger, the links AB, BC, CD, and DA form a four-bar mechanism with one degree of freedom. The motor rotates the screw, thus driving the four fingers to move. Flexible protective sleeves made of a sponge, are installed on the fingertips to prevent scratches caused by direct contact between metal and strawberry. To change the structural stiffness, link 3, shown in Figure 1, is replaced by rigid link 3–1 and rigid link 3–2, which are connected by a torsion spring located in the middle of link 3, and four groups of torsion springs are selected. When the torque is zero, the angle between links 3–2 and the fixed support plate is about 65°. Strawberries were placed randomly on a flat surface without a static posture. The structural parameters of the manipulator are shown in Table 1. The fixed coordinate system is described for ease of explanation, as shown in Figure 1. Assume the origin of the fixed coordinate system o-xyz is located at the contact point between the fixed support plate and the screw. The -axis is parallel to the support plate and points to one finger. The -axis is parallel to the support plate and points to another adjacent finger. The right-hand rule can determine the direction of the -axis.

The experiment used a camera with no less than 12 million pixels (MV-CE120-10GC, Japan) to obtain image information. The shooting direction coincides with the three axes of the fixed coordinate system. A hyperspectral imaging spectrometer (SOC710, Japan) was used to obtain the reflectivity of the strawberry surface. The camera has its lens, and the built-in sensor model is IMX226. The resolution is , and the exposure time range is 34 μs–1 s. When shooting, the lens was located 0.4 m directly above the strawberry, and the lens plane was parallel to the desktop. The spectral range of the hyperspectral imaging spectrometer is 400–1000 nm. The spectral resolution is less than 4.8 nm. Data processing was performed with the Spectral Radiance Analysis Toolkit (V3.5-SOC710-E version, Japan). Arranged the strawberries in the o-xz and o-yz planes. Measured the image and surface reflectance were within 2 h after the strawberries were grasped.

To analyze the method’s feasibility, the experiments were carried out with different parameters, as shown in Table 2. No treatment refers to the strawberries that the manipulator did not grasp.

2.3. Integrity Indices

Integrity means that the shape and surface properties of the strawberries do not change after being grasped by the manipulator. It can be comprehensively evaluated by the outline depression rate (ODR), indentation proportion (IP), and surface reflectance (SR).

2.3.1. Outline Depression Rate

Squeezing strawberries by an external force may lead to an abrupt change in the curvature of the elliptic contour curve of strawberries from a stable state, forming visible dents (Figure 2). Visual information processing can obtain the outline of the strawberry. The background in the captured images is first removed to get a complete strawberry image for processing. Hole filling and noise removal were carried out on the images using the closed circle function to close the image. The closed operation is defined as expansion first and then corrosion. The formula for the closed operation of set B to set A is as follows:

Closed operations usually bridge narrow discontinuities and long and thin gullies and eliminate small holes and broken marks in the filling contour lines. Finally, the difference function was used to calculate the depression area of the strawberry, and then the ODR was calculated, defined as the ratio of the depression area to the original area.

2.3.2. Indentation Proportion

The color of the strawberry surface may change after extrusion. It is necessary to obtain the color features of the strawberry before the damaged area is accurately extracted. First, distinguish the strawberry color from the background according to different RGB values, and then the preliminary process of removing the background was performed. From the RGB gray distribution histogram, it was found that the -value and the -value of the intact and damaged parts of the strawberry were quite different. Therefore, these two parameters were selected as a criterion of the strawberry surface damage. Then, the RG values of () pixels from the better and damaged areas of the strawberries were substituted into the SVM (support vector machine) training algorithm to obtain the critical line equation for dividing RG values: where A1, B1, and C1 are the coefficients obtained by the training algorithm.

Traverse each image pixel with the background removed and substitute the RG value () of each pixel into the linear equation. When , the point is marked in the damaged area. Finally, remove some negligible single pixels, and multiple dead pixels are concentrated in a closed area through the opening operation of the image. Therefore, the IP is defined as the ratio of the indentation area to the strawberry view area.

2.3.3. Surface Reflectance

The change in reflectance indicates that the surface properties (such as roughness and absorbance) and texture of the strawberries have changed. First, measure of the hyperspectral data of the background plate was at 400–1000 nm. The noise ratio in other wavelength ranges is relatively large, which may interfere with the test results) [31]. Second, the background of the hyperspectral imager was set as a grayboard, and the software removed the interference caused by the background. Third, transformed the hyperspectral data of the strawberry samples into reflectance data. Lastly, defined the SR as the mean difference from the control group.

2.3.4. Accuracy of Judgment and the Comprehensive Index

After the manipulator grasped the strawberries, if the above three indicators are used to independently assess the damage characteristics of the strawberries, the results may be inaccurate. Therefore, the accuracy of judgment (AOJ) was proposed to quantify the results. AOJ can be defined as the proportion of the number of damaged strawberries based on the integrity indices to the total number of damaged strawberries, which can be expressed as follows: where Numsu is the number of damaged strawberries identified using the indices, if the values for the ODR and IP are more significant than the control group’s average (not grasped by the manipulator), this indicates that the strawberry has been damaged and detected. If the values of these two indices are smaller than the mean value of the control group, the strawberry is damaged and undetected. For the SR, the opposite circumstances apply. Numto is the total number of damaged strawberries, which can be regarded as the total number of strawberries used in the experiment because it will damage the strawberries to varying degrees after being grabbed by the manipulator.

The strawberries may show a variety of damage characteristics after being picked by the manipulator; therefore, the comprehensive index (CI) was proposed to improve the AOJ. Based on principal component analysis (PCA), the CI can be defined according to the following procedure [32].

Construction of the index matrix. Suppose that the manipulator grasps strawberries, and single indexes are used to illustrate the damage characteristics of each strawberry. The index matrix is expressed as

Normalization. Index matrix is normalized by score transformation, and the normalized formula is as follows:

Then the normalized matrix can be obtained as follows:

Determination of the correlation matrix. There is a specific correlation between single indicators, resulting in information overlap in the data. Therefore, the correlation matrix is used to fully reflect the correlation between indicators, which is also the primary condition for dimensionality reduction. The correlation matrix can be written as follows:

Solution of the eigenvalues and eigenvectors of the correlation matrix. According to the equations and the eigenvalues and eigenvectors of matrix can be obtained as follows:

Determination of the number of principal components. The contribution rate of the -th principal component and the cumulative contribution rate of the first principal components are calculated using the following equations:

Generally, the first principal components with a cumulative contribution rate greater than 80% or the first principal components with an eigenvalue greater than 1 are selected to form a comprehensive performance index.

Calculation of the principal components. The expression of principle can be written as

Acquisition of CI. CI can be expressed as follows:

3. Results and Discussion

3.1. Comparative Analysis of the Grasping Results

The integrity of the strawberries grasped was compared by three different rigid-flexible coupling manipulators. The torsion spring coefficients of the three rigid-flexible coupling manipulators were 10 Nm/rad (manipulator 1), 1 Nm/rad (manipulator 2), and 0.1 Nm/rad (manipulator 3), respectively, and the strawberry maturity was from S1. The strawberries not grasped by manipulators were used as controls to illustrate the better effect of grasping.

For the outlines of the strawberry, the view plane was parallel to the xy plane. The outlines of strawberries grasped by manipulators 1 and 3 are shown in Figures 2(a) and 2(b), respectively. The outline of the strawberry grasped by manipulator 1 (the torsion spring coefficient was significant) was depressed, and manipulator 3 (the torsion spring coefficient was small) had little effect on the strawberry outline. Each group of 60 strawberries was analyzed, and the results are shown in Figures 3(b) and 3(d). The average value and variance of the ODR strawberries grasped by manipulator 3 were 0.039 and 0.0002, respectively. The average value and variance of the ODR of the strawberries grasped by manipulator 1 were 0.096 and 0.0012, respectively, much larger than the values of strawberries grasped by manipulator 3. In particular, the ODRs of the images shown in Figures 2(a) and 2(b) were 0.1431 and 0.03, which corresponds to the third point in Figure 3(b) and the fifth point in Figure 3(d), respectively. The ODR of the strawberry grasped by manipulator 3 (the torsion spring coefficient is very small) was not zero (Figure 3(d)), which is caused by a shooting error (the relative distance and posture between the strawberry and the camera).

The surface indentation is mainly reflected in the color change of the indentation caused by the extrusion by the manipulators. For the strawberries grasped by manipulators 1 and 3, the surface indentation extraction process is shown in Figures 2(c) and 2(d), respectively. The IPs of strawberries grasped by manipulators 1 and 3 are shown in Figures 3(b) and 3(c). For strawberries grasped by manipulator 3, the IP was smaller, and the surface of the strawberry was complete (Figure 2(d)). These results indicated that the smaller the torsion spring stiffness of the rigid-flexible coupling manipulator, the less effect it had on the integrity of the strawberry. The surface of the strawberry grasped by manipulator 1 had obvious indentation; the mean and variance of the IP were 0.108 and 0.0009, respectively.

The normal and damaged areas of the strawberries were analyzed using hyperspectral imaging. The mean value of the SR of each group with 60 samples grasped by the rigid-flexible coupling manipulators 1 to 3 is shown in Figures 3(b) and 3(d). The control group was the strawberries from “S1” that were not grasped by manipulators. The mean values of SR for manipulators 1 and 3 minus the control group were 0.306 and 0.376, respectively; indicating that manipulator 1 caused damage to the strawberry, resulting in a change in the surface properties.

It is worth noting that for the strawberries grasped by manipulator 2, the average values of ODR, IR, and SR are 0.066, 0.082, and 0.338, respectively, which is between the average value of ODR/IP/SDR of the strawberries grasped by manipulator 1 and manipulator 3. The same law appears in the variance value of ODR, IR, and SR of the strawberries grasped by manipulator 2, and the variance value of ODR, IR, and SR are 0.0005, 0.0002, and 0.0001, respectively.

Thus, the comparative analysis of the grasping results indicated that the smaller the stiffness of the torsion spring of the rigid-flexible coupling manipulator, the less effect it had on integrity in the strawberry grasping process.

3.2. Comparative Analysis of AOJ
3.2.1. Influence of the Torsion Spring Coefficient

The AOJs assessed by the three single indices and the CI are shown in Figure 4 after the three different rigid-flexible coupling manipulators grasped the strawberries. As shown in Figure 4(b) and Table 3, corresponding to the strawberries grasped by manipulator 1, the AOJ assessed by the CI was 98.33%, and the AOJs assessed by the three single indices (ODR, IP, and SR) were 88.75%, 91.25%, and 75%, respectively. The AOJs assessed by the CI for the strawberries grasped by manipulators 2 and 3 were 95% and 87%, respectively, both larger than those assessed by the single indices, which indicates that the CI is better than the single indices in assessing damage characteristics in strawberries. In addition, the AOJs assessed by SR was the same for the three cases (75% for each), which illustrates that the stiffness of the manipulator had no effect on the AOJ assessed by SR. This is because the SR of the strawberries grasped by the three manipulators was not significantly different from that of the control group (Figure 3).

As shown in Figure 4(c), the AOJ of the strawberries grasped by manipulator 1 was the largest, and the AOJ of the strawberries grasped by manipulator 3 was the smallest. For example, the AOJs of strawberries grasped by manipulators 1, 2, and 3 were 88.75%, 85%, and 80%, respectively, when assessed by the ODR. Similarly, when assessed by the CI, the AOJs of the strawberries grasped by manipulators 1, 2, and 3 were 98.33%, 95%, and 87%. This is because the damage characteristics of strawberries grasped by the manipulator with a low spring coefficient were not obvious, and there was no significant difference from the control group (Figure 3(d)), which makes it difficult to identify the damage characteristics of the strawberries.

The AOJ assessed by the CI increased compared to the three single indices. In addition, the smaller the stiffness of the manipulator grasping the strawberries, the smaller the AOJ assessed by the three single indices or the CI.

3.2.2. Influence of Maturity

To analyze the effect of strawberry maturity on AOJ, based on the strawberries from S1, another two groups of strawberries from S2 were selected. One group of samples was picked by manipulator 1, and the other group was not grasped by manipulators and was used as a control group. The integrity indices and assessment of the damage characteristics of the strawberries from 2 are shown in Figure 5.

For the strawberries from 2 grasped by manipulator 1, the AOJ assessed by CI was 100%, whereas the AOJs assessed by the three single indices were 91.25%, 91.25%, and 75%, respectively (Table 4). These observations illustrate that using the CI to assess the damage characteristics of strawberries was more accurate, which is consistent with the results shown in Section 3.2.1. In addition, combining Figure 4(a) and Figure 5(g), the AOJs assessed by the three single indices or the CI for the strawberries from S2 was more significant than that assessed by the corresponding indices for the strawberries from S1. This can be explained by comparing Figure 3(b) and Figures 5(d)5(f). The mean values of the outline depression rate and indentation proportion of the strawberries from S2 grasped by manipulator 1 were 0.1156 and 0.1275, respectively, which were larger than those of the strawberries from S1(Figure 3(b)). This shows that the higher the maturity of strawberries, the easier they are to be damaged, and the more obvious the damage characteristics are. Therefore, the AOJ of strawberries with high maturity has increased.

3.3. Feasibility of Integrity Indices and the Comprehensive Index

To further illustrate the rationality of the CI proposed in this paper, three special cases for integrity evaluation were set, as shown in Table 5, and in each case, there are still 60 strawberries being grabbed. Case 1S refers to the strawberry grasped by manipulator 1. The IP was 0.0722, and the mean value of SR minus the control group was 0.0489. According to these two indices, the strawberry was judged to be undamaged after being grasped by the manipulator. However, the outline depression rate was 0.1533, and the surface depression was obvious. This is because the color of the damaged part showed no obvious change before and after being grasped, therefore, the IP cannot be accurately extracted by visual image processing. In addition, the surface of the strawberry was not damaged, and the surface roughness and absorbance did not change significantly, resulting in a small change in SR. Combining the three indices, still undermined the integrity of the strawberries. Case 2S refers to the strawberry grasped by manipulator 2. The ODR was 0.0264, which is low. If the damage characteristics of the strawberry are judged only by the depression rate, the strawberry was not damaged. This may be because there is no depression on the surface of the strawberry, or the visual angle may cause it. The IP was 0.1242, and the mean value of SR minus the control group was 0.1348. The strawberry was damaged after being grasped by manipulator 1 through comprehensive judgment. Case 3S refers to the strawberry grasped by manipulator 3. The depression rate and indentation ratio were small, and the mean value of SR subtracted from the control group was large. This is because the color of the damaged part did not change significantly, but the absorbance changed because the surface of the strawberry was damaged.

From the above analysis, it is difficult to accurately assess strawberries’ damage characteristics with a single index because of the complex surface characteristics after being grasped. The image shooting angle, color change in the damaged part, and the degree of damage to the fruit surface can lead to inaccurate assessments of the damage characteristics. Therefore, the CI was proposed in this paper based on the three single indices. Using the CI to judge the damage characteristics of strawberries, the maximum and minimum AOJs were 98.33% and 87%, respectively. When using a single index to judge the integrity of strawberries, the lowest accuracy was only 80% (using the depression rate). This proves the feasibility of the CI proposed in this paper.

4. Conclusion

This study used a comprehensive method of evaluation based on the ODR, IP, and SR to improve the AOJ of the damage characteristics for strawberries grasped by manipulators.

Based on the three single indices, ODR, IP, and SR, a CI was put forward. The AOJs judged by the three indices were 88.75%, 91.25%, and 75%, respectively. This method improved detection accuracy by 10.79%, 7.76%, and 31.1%, respectively.

For the strawberries grasped by manipulators with different stiffness coefficients, the greater the stiffness of the manipulator, the larger the AOJ assessed by the three single indices or the CI. This is because the more significant the stiffness of the manipulator, the more serious its damage to strawberries, resulting in a larger AOJ. The AOJ assessed by the three single indices was still smaller than that assessed by the CI, which directly shows the rationality of the comprehensive index. For the strawberries of different maturity, the higher the maturity, the larger the AOJ assessed by the three single indices or the CI. This is because the higher the maturity, the easier it is to be damaged, which increases judgment accuracy. Therefore, the appropriate torsion spring coefficient can be selected according to the maturity of the fruit to ensure AOJ.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Additional Points

Novelty Impact Statement. Visual and hyperspectral data were used to examine the strawberry’s contour, area of surface damage, and surface reflectance. Based on the evaluation approach suggested, the accuracy of strawberry damage detection increased by 10.79%, 7.76%, and 31.1%, respectively. This approach might aid in developing manipulators that increase the productivity of the fruit manufacturing process.

Conflicts of Interest

The authors declare no conflict of interest.

Acknowledgments

This work was supported by the National Key R&D Program of China (2022YFD2100304), the National Natural Science Foundation of China (Grant No. 52275001), and the National Science Foundation for Young Scientists of China (Grant No. 51705201).