Abstract
Research on rice (Oryza sativa) roots demands the automatic analysis of root architecture during image processing. It is challenging for a digital filter to identify the roots from the obscure and cluttered background. The original Frangi algorithm, presented by Alejandro F. Frangi in 1998, is a successful low-pass filter dedicated to blood vessel image enhancement. Considering the similarity between vessels and roots, the Frangi filter algorithm is applied to outline the roots. However, the original Frangi only enhances the tube-like primary roots but erases the lateral roots during filtering. In this paper, an improved Frangi filtering algorithm (IFFA), designed for plant roots, is proposed. Firstly, an automatic root phenotyping system is designed to fulfill the high-throughput root image acquisition. Secondly, multilevel image thresholding, connected components labeling, and width correction are used to optimize the output binary image. Thirdly, to enhance the local structure, the Gaussian filtering operator in the original Frangi is redesigned with a truncated Gaussian kernel, resulting in more discernible lateral roots. Compared to the original Frangi filter and commercially available software, IFFA is faster and more accurate, achieving a pixel accuracy of 97.48%. IFFA is an effective morphological filtering approach to enhance the roots of rice for segmentation and further biological research. It is convincing that IFFA is suitable for different 2-D plant root image processing and morphological analysis.
1. Introduction
In the field of biology and genetic breeding, plant phenotypic refers to the external traits of organisms determined by the genotype and environment of the plant, such as shape, structure, size, and color. Researches on aboveground parts of the plants have made great progress, while the study of underground parts, especially the roots of plants, encounters difficulties in observing and analyzing a large number of plants. Root system architecture shows great importance in studying interspecific interactions and genetic improvement of crops [1, 2]. To acquire a high-quality root-system image and analyze the root message, much research had been done. In Gernot Bodner’s research, a data processing pipeline was developed for automatic root segmentation [3]. Johannes Pfeifer provides a study approach to develop a comprehensive segmentation method suitable for comparatively large columns sampled in situ [4]. Kaspar established a platform to image the roots automatically and nondestructively [5, 6]. Morphological filters possess effective noise suppression with reduced geometrical feature blurring [7]. [8] extracts the main features using a morphological filter algorithm to reconstruct fine roots with high fidelity. WinRHIZO, EZ-RHIZO, and ROOTEDGE are software tools designed for the measurement of root system architecture [9–11]. In addition to 2D image processing and X-ray microtomography [12], Fang et al. focus on 3D reconstruction and dynamic modeling [13]. Compared to the former methods, a cost-effective and efficient way to automatically extract precise root system architecture from high-resolution root images is proposed in this article. Since rice is the main food source, this article would take rice roots as a research object.
Commonly, plant roots cannot be observed in a nondestructive manner. Dragging roots out would lead to unrecoverable injury to the phenotypic root system. To solve these problems, Frangi et al. use transparent media to observe roots architecture [6, 14]. In this paper, to establish a high-throughput root phenotyping environment, transparent plastic bags with nutrient solutions are used to cultivate the plants. CCD cameras are used to gather the pictures of roots, as shown in Figure 1.
(a)
(b)
(c)
Difficulties arise from the fact that some of the lateral roots are transparent to a certain extent, and they occupy only a few pixels in the image (measurement shows that their diameter is only 0.1 mm, and the pixel width of the image is only about 3 pixels). This also means that lateral roots are as thin as the noise, making it harder for an algorithm to differentiate them. Frangi et al. proposed an effective algorithm to enhance vessel structures with the eventual goal of vessel segmentation [6]. The Frangi filtering method can be concluded as smoothing the image using Gaussian filtering and finding the Hessian matrix that describes the image curvatures. And in such morphological filtering means, the tubular object would be enhanced. Gaussian filtering is essentially a low-pass filter. If the window width is the scale factor , the filter output is the largest, and the effect is best when it matches the actual width of the root. As shown in Figure 2(a), the original image has a lot of noise, which is close to the lateral roots on a scale. In the background, some disturbing indentations are similar to the color of the root. When using the classic Frangi filter, if the noise filtering is to be ensured, a large number of side roots are filtered out (Figure 2(b)); when the side roots are retained, there will be more noise and adhesion and blurring between different roots (Figure 2(c)). In general, the output is not satisfying or suitable for research purposes. Some countermeasures should be taken during applying the Frangi filtering algorithm on roots enhancement, which we refer to as an Improved Frangi Filtering Algorithm (IFFA) described in the following sections in detail.
(a)
(b)
(c)
2. Materials and Methods
2.1. Automated Platform for Image Capturing
A multiangle camera layout is adopted to improve the efficiency of image capturing of the bag seedlings. To avoid the interference of ambient light, the imaging equipment is mounted in a dark room. As shown in Figure 3, four cameras are installed at different angles to capture four root images each time. The darkroom is the aluminum-profile frame wrapped with metal skin to cut off the ambient light. This procedure is controlled by a PLC, which receives the signals generated by the sensors mounted on the equipment. We used the CCD industrial cameras with a resolution of 4608 × 3288 pixels to ensure the detection of the lateral root.
(a)
(b)
(c)
(d)
2.2. Original Frangi-Filtering Method
Frangi 2D filtering is an edge detection and enhancement filtering algorithm based on the Hessian matrix. Hessian matrix is the square matrix of second-order partial derivatives of scalar functions. It describes the Local curvature of multivariate functions. Hessian matrix is a binary matrix for a 2D image. The calculation of the Hessian matrix is as follows:
The cross product of stands for the convolution with the image, and stands for the Hessian matrix.
Before calculating the Hessian matrix, Gaussian smoothing is applied to remove the noises, reducing the deviation of the second-order partial derivative. Gaussian smoothing is described as follows:
is a symmetric matrix, so its eigenvalues could be used to construct the enhancement filtering. The two eigenvalues and can be calculated by
When the scale factor matches the actual width of the vessel, the output of the filter is maximized. So as a spatial scale factor, iterations can get different scales of output. The half-width of the window rectangle of the local characteristic analysis is generally . When the diameter of the blood vessel is smaller than the width and height of the corresponding window rectangle of the current scale, the eigenvalues of the Hessian matrix of the tubular vessel satisfy
Define and with and .
The responsivity of in vessel areaswhere the background of an image would be cut off and is a parameter controlling the difference between thread objects and block objects. are parameters controlling the smoothness of thread objects.
The half-width of the rectangular window of each pixel is generally taken to be . Note that when the window corresponding to different scales matches the diameter of the blood vessel or tubular-shaped objects, the maximum response occurs. Changing the value of , recording the maximum response of each pixel, and finally reconstructing the image, the trace of blood vessels would be enhanced.
2.3. Target Improving Aspects and Processing Procedure
Drawn from engineering experience, a successful image filtering method should meet such properties as noise removing while preserving the original signals and fast computation and is robust against different signal characteristics. To achieve the goal, the processing procedure is designed as in Figure 4. In the following sections, each of the novel parts will be discussed.
2.4. Truncated Gaussian Kernel
In the original Frangi filtering procedure, Gaussian filtering and calculation of Hessian matrix are compiled on the same step, and the width of Gaussian kernel is given as follows:
A large kernel enhances the ability to remove noise; however, the blurring effect may cause the loss of needed information. In this case, when ranges from 1 to 10, h ranges from 7 to 70. The Gaussian kernel is significantly too large to reserve the lateral roots. Thus, the truncated Gaussian kernel is proposed. Its width is given as follows:
In this way, even though the ability of noise removal is weakened, it is still used, as the lateral roots would be reserved. When ranges from 1 to 10, h ranges from 2 to 20. Figure 5 shows the comparison of original Frangi filtering and Frangi with truncated Gaussian kernel. Figures 5(b) and 5(d) preserve more lateral roots than Figures 5(a) and 5(c).
(a)
(b)
(c)
(d)
2.5. Adjustment of Variance
In the original Frangi filtering algorithm, variance σ varies from 1 to 10 with the step size of 0.5, and correspondingly, we need 19 times scanning for processing one image. If the Frangi algorithm is used to filter a large number of images with high resolution, it is time-consuming. Finding a way to minimize the range of variance would benefit the whole procedure. It is also a self-adaptive way to handle images of different scales.
In this program, the intersection-over-union (IOU) method is introduced. It is a measurement of the similarity between two binary regions, widely used in machine vision programs. For grey-scale (0–255) images, we define that if the grey-scale difference of corresponding pixels in two grey-scale images is less than 10, then the two pixels are similar. The following pseudocode (Algorithm 1) explains the specific operation process. During one loop of Frangi, if the IOU of the original grayscale image and current output is more than 0.9, the circulation will end. Variance σ ranges from 1 to 5 in this program, saving over half of the time.
|
2.6. Multilevel Image Thresholding
The cultivation of rice roots requires the use of a transparent bag. There will be water droplets and bubbles in the transparent bag, and there will be highlights on the surface of the transparent bag when images are taken; in addition, the kraft paper in the transparent bag has many colors. These factors have affected the segmentation effect of the root image. To ensure that the main roots or lateral roots of different thicknesses and transparency can be segmented as accurately as possible, we introduce an improved Frangi filter algorithm to enhance the image and use multiple Otsu threshold segmentation. From Figure 6, we can see the necessity of more threshold segmentation.
(a)
(b)
The multithreshold Otsu method is an alternative formulation for Otsu’s method. Regardless of the number of classes being considered during the thresholding process, the sum of the cumulative probability functions of M classes equals one, and the mean of the image is equal to the sum of the means of M classes weighted by their cumulative probabilities [10]. The number of classes could be controlled, usually 3 or 4. Figure 7 shows some different output images of the multithreshold Otsu method. means dividing the point set by m thresholds and removing n part nearest to black (0/255). Taking into consideration of the image details, sharpness, and processing time, is chosen to be applied to the whole dataset.
(a)
(b)
(c)
(d)
2.7. Label Connected Components
This step can be regarded as the final step to remove noise and unnecessary parts in images. Morphological computations, namely, erosion and dilation, are used to augment the connecting component, making the image clearer for observation. Figure 8 shows the original and final output to visualize the effect of IFFA.
(a)
(b)
(c)
(d)
2.8. Width Correction
Width correction is proposed to meet different research acquirements. By using IFFA, the width of roots would be magnified. If the width of roots is most important for algorithm users, the real form should be reserved. We have compromised between the two types of errors (rejection of the true, failure to reject a false), and a decrease of IOU from 0.9 to 0.7 makes the root shrink to a more reasonable level. Statistically, the comparison between the filtered roots and the manually labeled truth in more than 30 images shows that the acceptable IOU limit is 0.65–0.75. Set the IOU limit to , and the true width of roots would be reserved. Figure 9 shows an example of width correction.
(a)
(b)
(c)
2.9. Analysis of Binary Image
As long as the binary image is extracted, many indexes of the root system could be calculated. The width-depth ratio (WDR) is defined in the algorithm as follows:
WDR is an initial point to study the drought resistance ability of plants.
If every pixel stands for identical mass, the center of mass (CoM) is defined as follows:
By switching the value of σ, the number of main roots and lateral roots can be calculated. Thus, the ratio of lateral roots to main roots and the density of lateral roots are obtained. The other indexes, such as primary root length and total root length, are easy to be figured out by an algorithm.
3. Results and Discussion
3.1. Comparison with the Original Frangi Method
A data set label by a human is introduced. Suppose that IMG stands for the output binary image, and MASK (ground-truth) stands for the labeled unfiltered image. Intersection over Union (IOU) is defined as follows:
The higher the IOU value is, the more accurate the object is reserved. Pixel accuracy (PA) is a standard measuring of the ratio of correct pixels to all pixels. The dice accuracy (DSC) is defined as follows:
Similar to the IOU standard, DSC indicates the overlap measurement between the ground truth root regions and the segmentation results [15].
To verify the performance of the algorithm, we first cultivated 30 rice seedlings in a transparent bag and took 100 close-up pictures of the root system. Then, through manual annotation, we got the root mask of each picture. Then, we input 50 original images into the IFFA and Frangi algorithms to get the output of IFFA and Frangi and finally combine the artificially annotated images to obtain the IOU, Pixel Accuracy, and DSC of the two algorithms. Table 1 shows the average processing time and IOU of Frangi and IFFA to the labeled image. Figure 10 gives a labeled image, Frangi output, and IFFA output of the same basic image.
(a)
(b)
(c)
3.2. Comparison with U-Net Backbone Network
U-Net network performs extraordinarily in medical image segmentation. Because of the similarities between plant images and medical images in many aspects, we use U-Net-based image segmentation neural network as the baseline of this research.
Liang Gong et al. proposed a deep neural network algorithm for plant image segmentation based on the U-Net backbone network [16]. We reproduced the algorithm and obtained the root image segmentation results through training. The training of the U-Net network uses 90 pictures as the training set and the remaining 10 pictures as the test set. Figure 11 shows the output results of U-Net and IFFA on the same original input image.
(a)
(b)
(c)
Comparing the output image of U-Net and the output image of the IFFA algorithm, we can easily find that the deep neural network with U-Net as the backbone network can effectively identify most of the main root system, but there is still a phenomenon of mis-segmentation, for example, the part in the small red frame circle in the output image. In addition, the U-Net backbone neural network algorithm cannot correctly segment the small and transparent lateral roots in the root system like the IFFA algorithm. Table 2 shows the average processing time and IOU of U-Net and IFFA to the labeled image.
3.3. Comparison with Software: ARIA
ARIA is software created by Jordon Pace at Iowa State University [17]. It is powerful in analyzing data of roots, while, after image filtering, some main roots and most of the lateral roots would be removed. The start point of roots should be labeled using the mouse before calculation, and it could handle an RGB image.
In comparison with ARIA, our focus is on the computing time. We input 100 different original images and calculate the average computing time. Table 3 shows the time comparison between ARIA and IFFA. Compared with ARIA, IFFA has a shorter calculation time. Figure 12 shows the interface of ARIA.
4. Discussion
We have shown that the IFFA method can achieve outstanding performance in the enhancement of images of rice roots that grow in transparent plastic bags. Replacing Gaussian filtering with truncated Gaussian filtering, the lateral roots will be protected from being erased. Applying Otsu multithreshold segmentation and width correction to the images, we enhance the root structure, which is pure and sharp. The results show an effective way to extract rice root structure.
Compared to naïve Frangi method, IFFA is faster in the loop of σ. IOU standard helps judge when σ becomes too large and the image quality is impossible to attribute to the enhancement. Otsu method removes all the grey noise points, and it leaves a parameter to us, by which we could choose to keep more lateral roots (and more noise) or less. ARIA seems to remove nearly all the lateral roots and cause distortion, so we could not use it to make valuable comparisons.
However, there are still some shortcomings in our algorithm. The database is not large enough to make this procedure automatic and precise, such as the determination of the IOU threshold in section Width Correction. One in ten lateral roots will be removed during the process, and a large area of overlapping will cause an intolerant mess. In further research, we should make effort at the very beginning, the cultivation of rice seedling and the collection of images for the investigation and a better understanding of rice root biological characteristics.
5. Conclusions
A high-performance and well-defined automatic root image processing tool is important for the agronomic study and plant physiological investigation. Based on the original Frangi filter, a 2D image processing algorithm for plant root structure analysis is improved, which effectively filters out the in-bag root background and enhances the primary roots while keeping the lateral root almost intact. Experiments show that the IOU score of the IFFA algorithm reaches 63.8%, the segmentation pixel accuracy is 97.5%, the DSC is 77.9%, and a single image takes about 3 seconds. The proposed IFFA outperforms other root image enhancement algorithms in both pixel resolution and image signal integrity, and it can be used for automatic rice root phenotyping. [17]
Data Availability
All additional files, containing the algorithm, original images, and processed images, are provided in the repository https://github.com/gitDux/IFFA.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors thank the School of Life Science, Shanghai Jiao Tong University, for providing experimental sites and crops. The research was funded by the National Natural Science Foundation of China under Grant no. 51775333, the Royal Society under CHL\R1\180496, and Shanghai Municipal Agricultural Commission, under 2019-2-2.