Abstract
With the development of modern science and technology and more and more image processing systems, related technologies are becoming more and more complex. The application of image processing technology can be seen in various fields of society, such as medical field, aerospace field and life, and entertainment field. Due to the increasing amount of information on the picture, the requirements for the speed and clarity of image processing are also increasing. The existence of various external factors will lead to the production of image products and objects between the error and distortion problems. In order to make the process product design more authentic and reliable, this paper studies the process product design based on image processing multimode interaction. It uses radiometric correction and geometric correction to process distorted images and uses GPU parallel computing technology to accelerate the correction process. In this paper, this technology is applied to the visual recognition of welding robot, and the experiment shows that the product produced by the image processed by this module can obviously reduce the error.
1. Introduction
Image refers to the general information carrier existing on paper, photo, TV, or computer screen. Image processing technology refers to the process of image collection, processing, processing, and storage. Most images are stored in computer in digital mode, and image processing is mainly aimed at digital image processing. With the rapid development of science and technology, the amount of information accepted by human beings has expanded rapidly, and the research and application of image processing have also increased. In the 1960s, the United States used image processing technology to process nearly 100,000 photos of spacecraft returning to Earth, which laid the foundation for the lunar landing plan of the United States. In the 1970s, Britain applied image processing technology to human medicine and obtained sectional images of various parts of human body. Now, in all fields of social production, such as space remote sensing, agricultural production, scientific research, and other fields, we can see the application of image processing technology, and image processing technology is also concerned as a whole. With the increase of image information, the requirements of image processing speed and resolution are getting higher and higher. In many fields, image imaging is affected by many factors. External factors are weather factors, illumination angle, exposure, and so on. There are internal factors, such as low precision of camera sensors and image transmission errors. These are errors and distortions between the produced image products and the actual target objects. These errors and distortions are not conducive to users’ feelings and evaluation of image products. In order to eliminate the image distortion, this paper uses efficient image correction technology to study the method of dealing with the image distortion and reducing the error between the image and the real target and devotes itself to displaying practical and reliable image data to users [1].
Multimodal interaction refers to the comprehensive interaction between products and users [2]. However, the innovation of products has changed from technology type to user perception and human-computer interaction. Exactly, multimodal interaction can reflect this perception and experience. At present, we propose a design method based on human-computer interaction and TRIZ cooperation. It includes the whole process from requirement analysis to scheme formation and provides a user experience view. After verification, it shows its effectiveness and helps product innovation. Internet of things devices have been integrated into the public physical environment. It is subject to the application and voice assistant as the interface [3], which makes its expansion performance poor. On this basis, the researchers decompose the user's IOT commands into two components, selection and interaction in the design space, and then clarify the possibility that they go beyond the current way. The design is transformed into a perception platform, showing a novel interactive scene of the Internet of things and solving the fuzzy direction of messy devices. Nowadays, a wide range of computing and communication resources can support us to think about sound in a positive way [4]. Acoustic interaction design belongs to a new field and is in a cross field. Sid is the abbreviation of acoustic interactive design. Its application has the artistic significance from alarm to music creation. It emphasizes the interactive system of auditory mode and is realized by the integration of computing, communication, and interactive technology. Background augmented reality classroom has become meaningful [5], and its limitations are mainly in two aspects: first, the inconvenience caused by using cards to operate experiments and second, single-mode interaction lacking the sense of real interaction. The proposed multimodal interaction algorithm based on augmented reality (argev) and the virtual reality fusion interaction tool kit (vrfits) enhance the visual and tactile feedback. By comparing the availability of single-mode and multimode interaction and existing TV remote control applications [6], we find out the appropriate interaction mode. Through their tests, they prefer to adopt unimodal interaction design in TV remote control applications, but due to interface problems, favorable interaction design involving brush gestures will replace this design. The handicraft design in the traditional design is manual training to achieve a certain technical height, but there are skilled operators; handicraft design defective rate is high, resulting in the cost of handicraft design. It affects the large-scale production of products, and the process of the same batch of products is different. This paper puts forward a design method of handicrafts based on image processing technology, thus improving the design accuracy of handicrafts.
2. Introduction of Related Theories and Technologies
In digital image processing, generally, the image information collected by image sensors is transmitted to computers or other devices, and a series of processing is carried out on this image by means of software or hardware, so as to highlight the interesting feature information in the image. Digital image processing technology can be seen everywhere in our lives, such as aerospace, consumer payment, artificial intelligence, intelligent transportation, and medical imaging, which plays a great role in promoting the progress of human civilization.
This section introduces digital image processing and FPGA technology in detail before using image processing algorithm to identify PCB defects.
2.1. Representation Method of Related Images
F(X,J) is often used to represent a gray image, where F(X,Y) is a two-dimensional mathematical matrix, (X,Y) represents the pixels in this gray image, and the value of a single F (X, Y) represents the gray value at the position (X,Y). Most signals in nature are analog quantities, and digital image processing is a process of processing digital quantities, so it is generally necessary to transform analog quantities into digital quantities, and a gray image is usually represented by matrix and binary values.
2.1.1. Array Representation of Gray Image
Let the analog image become a digital image F(X,Y) after a series of processing, which is usually expressed in the matrix form of MN size and can also be expressed in the matrix size of NN), which is expressed by
In a digital image, a pixel is the value of each element in the matrix f(x, y).
2.1.2. Binary Image Representation
In digital images, the values of matrix F(X, Y) generally have only two values, such as logic 0 and logic 1 in logic circuits, and there are only two possibilities. For digital image processing, gray images are often changed into binary images for image processing, which can greatly reduce the amount of computation. The characteristic of binary image is that it does not occupy too much memory and is convenient to store. The disadvantage is that when describing complex things, it can only outline its rough outline and cannot describe it in detail [7].
2.2. Basic Algorithms of Digital Image Processing
In the external natural environment, when the image sensor is used to collect the image to be detected, due to external interference, making the images we collected, more or less, it will not be so clear. There may be certain noise, low image contrast, and other characteristics. These phenomena will cause great interference to our image recognition. In order to get the feature information of the image to be detected better, it is necessary to carry out a series of image preprocessing on these collected images first, so as to enhance the feature information of the image to be detected to detect the related defects of the image to be detected more quickly and accurately [8].
2.2.1. Image Enhancement Algorithm
In digital image processing, image contrast enhancement is usually carried out in spatial domain and frequency domain. Histogram equalization is a digital image processing algorithm in spatial domain. Its main implementation process is to change the overall gray value of the image by changing the distribution of the gray histogram. Histogram equalization algorithm is a common digital image processing algorithm [9], and its basic principle is as follows.
Let an image have L gray levels and n pixels, and nk is the number of pixels of gray value f in image f (x, y); then the probability of gray level k is
The realization of histogram equalization is to change the gray value of the original image through mapping function. If T represents its mapping function, histogram equalization can be expressed as follows:where fk is the gray level of the original image, Sk is the equalized gray level, N is the total number of pixels in the image, and L is the total gray level in the image.
2.2.2. Median Filtering Algorithm
In the external natural environment, when the image sensor is used to collect the image to be detected, because of the external interference, the images we collected will be more or less clear and may have certain noise, low image contrast, and other characteristics. These phenomena will cause great interference to our image recognition, so we must first filter the collected images. Median filtering is an image processing algorithm used to remove noise. It can carry out a series of processing procedures on images containing noise to suppress noise. Generally, the median value in the neighborhood space of the template is used to replace the gray value at the center of the template [10], and its mathematical formula is shown in where g(x, y) is the gray value of the image before processing (x, y) and f(x, y) is the gray value of the image after processing (x, y). For the specific implementation of median filtering, sliding window is very important for median filtering. The size of sliding window is usually 3 × 3 or 5 × 5. As shown in Figure 1, it is a specific filtering process of 3 × 3 sliding window.
2.2.3. Edge Detection Algorithm
In digital image processing, the types of image edges mainly include three types as shown in Figure 2.
The types of image edges and the variation law of derivatives are shown in Figure 2. In digital image processing, in order to describe the edges of images, derivatives in mathematics are often used to express them, which are generally obtained by calculating the first and second derivatives of image matrix functions. In digital image processing, edge detection is equivalent to derivation of two-dimensional function. In mathematics, derivation of discrete function is equivalent to derivation of differential operator. Therefore, the derivative gradient vector in digital image processing can be defined as follows:
The direction and amplitude of the gradient are expressed as
In general, for digital image processing, in order to facilitate calculation, the formula of gradient amplitude is simplified:
However, in the actual image processing, some directional template operators are usually constructed, and it is easy to obtain derivatives through these directional template operators. Generally, Sx and Sy are used to represent horizontal and vertical gradient operators. According to the size of template and the value of element, Sobel operator is the most commonly used edge detection operator. In digital image processing, Sobel edge detection operator often uses convolution operation in mathematics to realize edge detection algorithm. Generally, it includes template operators in horizontal and vertical directions. The gradient operator templates in these two directions are shown in Figure 3.
G and G are used to represent the edge detection gradient in horizontal direction and vertical direction. In mathematics, the formula can be expressed as in Figure 4:
2.2.4. Algorithm of Target Recognition
There are three commonly used target defect detection methods, one is reference method, the other is nonreference method, and the third is mixed method. The most commonly used method is the reference method [11]. The reference method generally compares the pixel values of the image to be tested with those of the standard image, so as to detect the defects of the image to be tested. This method is relatively simple, but the detection speed is fast and easy to complete; however, the nonreference method is to detect interesting things through the feature size of objects or some unique design criteria, so it is difficult to realize this method, and it can only identify objects through some feature information, which requires more complex detection algorithms. Mixed method is a combination of reference method and nonreference method, which can achieve twice the result with half the effort. But for the specific actual situation, the most commonly used method is the reference method.
Background subtraction algorithm is a commonly used target recognition algorithm to realize reference method, which is widely used in the related fields of image detection and recognition. Background subtraction method is to subtract the current image and the background image point by point based on pixels, take their absolute values, and binarize them with thresholds, so that the interested objects in the current image can be determined. It can also be used for defect detection in practice in Figure 5.
The concrete implementation of the background subtraction algorithm is shown in Figure 5. B (X, Y) is a two-dimensional matrix function processed in advance, FN (X, Y) is a two-dimensional matrix function of the current image, and a two-dimensional matrix function DN can be obtained by processing the two-dimensional matrix function of the background image and the two-dimensional matrix function of the current image by
A two-dimensional matrix function R can be obtained by an appropriate threshold T and corresponding operation according to the following formula:where 255 stands for the target and 0 stands for the background, the interesting target area in the image can be obtained finally. There are four processes in target recognition using background subtraction: obtaining background image, updating background image, obtaining target area, and extracting feature information.
3. Welding Image Processing
3.1. Welding Image Preprocessing
In the process of generation and transmission, the pretreated weld image will be affected by imaging system and transmission medium, which will lead to the degradation of image quality, resulting in image blurring, distortion, noise, and so on. At this time, it is necessary to introduce image processing technology to preprocess the weld image, so as to make the weld image clearer.
3.1.1. Fuzzy Welding Image Processing
When the camera takes pictures, the front end of the welding gun moves faster than the weldment, which is easy to cause motion blur, which brings a lot of inconvenience to the acquisition of weld image. Using motion blur image restoration technology, recognizable weld image can be obtained.
3.1.2. Improving the Contrast of Weld Image
After obtaining the weld image, the weld profile may not be obvious, which is not conducive to extracting the important information of the weld. In order to make the contour of weld image obvious and discriminate, it is necessary to improve the image contrast. Commonly used methods are gamma value adjustment and stretchlim function method. For this experimental environment, gamma value adjustment method has better effect.
3.1.3. Gray Scale Processing
The weld image obtained by industrial camera is a RGB three-channel color image, which requires a long calculation time. Gray processing can change the color image of RGB three-channel data into a single-channel data image with only brightness information but no color information. After gray-scale processing, the main contour information of the image is not affected, which can improve the speed of image processing and reduce the amount of data processed.
3.1.4. Image Noise
In gray image processing, image noise should be considered firstly, which is caused by spatter, smoke, and welding electrical noise in the welding process. In addition, noise will also be generated when the weld image is stored, processed and transmitted. Noise is the pixel value of the obtained image, which cannot reflect the real scene brightness and will affect the accurate understanding of image information. There are two typical noises in digital image processing: Gaussian noise and salt and pepper noise [12].
(1) Gaussian Noise. Gaussian noise is based on the normal distribution of probability density function caused by poor illumination and high temperature. Its amplitude follows Gaussian distribution, but its power spectral density is also evenly distributed.
(2) Salt and Pepper Noise. Salt and pepper noise is caused by interference of image signal, which is random white dot or black dot and may be white dot noise or white image on black image.
3.1.5. Image Denoising
The existence of noise has an important feature that makes the image flooded. Usually, the image obtained by using image emphasis technology and image filtering technology is processed by noise reduction, which reduces the influence of multiple interference signals and improves the image quality after processing. Image noise removal mainly refers to suppressing or removing the noise of the target image while using image filtering technology to keep the details of the image as much as possible. General image noise removal methods include frequency domain filtering and spatial domain filtering. Frequency domain filtering enhancement processing does not process the input image directly, but processes the converted image, which has a large amount of computation and poor real-time performance and is rarely used in beacon tracking system using machine vision system. Therefore, spatial region filtering is usually selected to process images, and the most common processing methods are average filtering and median filtering.
(1) Mean Filtering. Typical random noise is caused by sharp changes in image gray level. Mean filtering can reduce sharp changes in image gray level by replacing pixel values with neighborhood mean value, thus reducing image noise. 33 template smoothing filter, 55 template smoothing filter, 77 template smoothing filter, and 99 template smoothing filter are used for average filtering.
It is found that 55 template smoothing filter has a good effect on removing Gaussian noise and can retain more details. For salt and pepper noise, its amplitude is basically the same, but the position of noise points is random, and the average value of noise in statistical sense is not 0, so the filtering effect of average filter on salt and pepper noise is not ideal [13].
(2) Median Filtering. Median filter is a nonlinear signal processing method, which is insensitive to outliers and can reduce the influence of outliers without reducing image contrast. The template of median filtering is usually a square adjacent area, and the median point is the target pixel point. When calculating, the gray value of the surrounding points is sorted, and the median value obtained becomes the filtering result of the central point. By this method, the median filtering of the whole image is realized for each pixel point, and better smoothing processing is obtained.
The research shows that if the value filter of 77 template is selected, the filtering of pepper noise is the most effective, and the important information of image edge can be kept to the maximum extent.
3.1.6. Gray Scale Transformation Enhancement
The contrast of the image becomes worse, and the distribution of the histogram of the image becomes uneven. The main pixels are concentrated in the vicinity of several pixel values. Gray scale linear transformation every pixel stretches linearly, gray scale value is transformed into new gray scale value by certain mathematical transformation formula, and the dynamic range of gray scale changes. If the contrast of gray image is enlarged, the display effect of the image will be improved, and the image quality will be improved. If the gray scale range of the original image f (x, y) is set to [a, b] and the gray scale range of the converted image is linearly expanded to [c, d], the gray scale value f (x, y) of any point in the image is converted to . The formula is as follows:
If the gray levels of most pixels in the image are distributed in the interval [a, b], the maximum gray level of the original image is maxf, and only a small part of the gray levels exceed this interval. In order to improve the image enhancement effect, you can make
There are images after direct gray processing. The gray scale in [0, 1] is stretched to [0.1, 0.5] by local stretching of the image. The image is stretched locally, and the gray scale in [0, 1] is stretched to [0.3, 0.5]. There are images after direct gray processing and images after stretching gray range.
Compared with the images with linear gray-scale changes, it is found that the images with direct gray-scale processing are not obvious in light and shade, and the contrast is not high. After the gray level changes linearly, the contrast between light and shade of the image is enhanced, which makes the weld clear and the contour obvious, effectively improves the visual effect of the image, and brings convenience for the subsequent processing.
3.2. Image Binary Processing and Scatter Removal
3.2.1. Image Binarization Processing
The image after gray-scale processing is binary, which can make the weld image only black and white, so that it is convenient to identify the edge features of the weld. Image binarization processing mainly sets the gray value of the image and divides the image through a certain gray value. The expression of binarization processing is
The algorithm of this expression is as follows: given a threshold T, when the gray value is greater than T, it is white, and when it is less than T, it is black. After processing, the two-dimensional matrix is only composed of 0 and 1, and the gray-scale image will become a special gray-scale image with only black and white.
In the image after binary processing, the contour of weld edge is discontinuous and the image feature is not obvious, so the bimodal method is used for image processing. This method considers that the image is composed of foreground and background, and the gray distribution of the image can be considered as the superposition of two normal distribution functions, so there will be two separated peaks in the histogram of the image, and the trough between the two peaks is the image reading value.
3.2.2. Scattered Removal
After image denoising, gray transformation enhancement, and binarization, the weld image basically meets the requirements of edge detection, but there are some problems in the welding process, such as rough welding surface, welding slag splashing, and so on, which will make the collected weld image appear. The light spots with large area are not related and distributed irregularly, which are usually called scatter spots. The area of scatter points is much larger than that of noise points. But it is much smaller than the area of continuous weldments on both sides of the weld. At this time, it is necessary to use image algorithm to remove these large scatter points, that is, calculating the area of each bright spot area in the image and then judging whether the area is large enough. If the area is small, it can be considered as scatter points and removed directly, so that the final output image is only the weld and weldment.
After the scatter points are removed, the image can be reversed, and the black part can be changed into white, while the white part can be changed into black, which does not affect the effect of the weld image and is convenient to compare with the original weld image and extract the edge contour of the weld.
After the image is inverted, it can be seen that the white part in the middle is the weld and the black part is the workpiece to be welded on both sides. Compared with the original weld image, we can see that the weld area is basically the same as the original weld area, which proves that the image processing flow is feasible and the method is correct.
3.3. Edge Detection and Centerline Extraction
In the image, the region boundary with sharp change of gray value is called edge, and the local features of edge image are discontinuous, and there are sudden variation of gray level and sudden change of texture structure. Edge detection is an important research direction in the field of machine vision and image processing, and it is an important feature extraction method. The commonly used edge detection is Roberts operator detection, Sobel operator detection, Prewitt operator detection, Cany operator detection, and Logo operator detection.
Comparing the results of edge detection, it is found that Prewitt operator edge detection and Sobel operator edge detection cannot meet the requirements, the edge contour is not clear, and the complete weld cannot be extracted. It is found that the contour of the weld seam is consistent between the image processed by edge detection and the original image collected, which proves that the flow of weld seam image processing can be carried out.
The ultimate goal of the seam tracking system is to control the welding gun to move along the seam centerline, so that the welding has real time and high accuracy, so it is particularly important to accurately extract the seam centerline. In this paper, the image skeleton method is used to extract the center line, which takes the center of the target as the benchmark, refines the target, and extracts the center line of the part of the figure, thus removing unnecessary information and facilitating recognition.
4. Experiment
4.1. Trajectory Extraction Experiment
Through the seam edge detection algorithm based on directional wavelet transform in Section 3, the seam edge image is obtained, and the seam trajectory image is extracted by morphological image processing method. Seam trajectory extraction experiments are carried out on different shapes of welds such as straight lines, broken lines, and curves, and the experimental results are as follows:
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(1)The weld trajectory extraction experiment of straight weld is carried out on the weld trajectory extraction experiment of straight weld, and the experimental results are shown in Figure 6.(2)Experiment on extraction of weld trajectory of 90 broken line weld: the weld trajectory extraction experiment is carried out on the 90 broken line weld, and the experimental results are shown in Figure 7.(3)Experiment of extracting weld trajectory of broken line weld greater than 90: the weld trajectory extraction experiment is carried out on the broken line welds greater than 90, and the experimental results are shown in Figure 8.(4)Experiment of extracting weld trajectory of broken line weld less than 90: the weld trajectory extraction experiment is carried out on the broken line welds less than 90, and the experimental results are shown in Figure 9.(5)Experiment of weld trajectory extraction of regular arc weld: the weld trajectory extraction experiment is carried out on regular arc weld, and the experimental results are shown in Figure 10.(6)Experiment of weld trajectory extraction of irregular arc curve weld: the weld trajectory extraction experiment is carried out on the irregular arc curve weld, and the experimental results are shown in Figure 11.
4.2. Experiment and Error Analysis of Center Line Extraction
Through the above experiments, the weld trajectory image is obtained, and the weld trajectory image is further processed. The weld trajectory centerline is extracted by thinning method, and the weld trajectory centerline image and coordinates are obtained. The weld trajectory centerline extraction experiments are carried out for different shapes of welds, such as straight line, broken line, and curve. In order to verify the accuracy of the algorithm for extracting the centerline of weld trajectory, the error analysis of the extracted centerline of weld trajectory with different shapes is carried out. The actual weld width measured by microscope is about 0.25 mm. The industrial camera selected by the platform is 2 million pixels, its resolution is 1624 × 1240 pixels, and the corresponding actual working area is 80 mm x 60 mm. Therefore, the actual distance between every two pixels can be calculated to be about 0.05 mm, which is the design accuracy of extracting the center line of weld trajectory, which is far less than the actual weld width of 0.25 mm, and can meet the design requirements. From the above analysis, it can be concluded that the extraction accuracy of weld trajectory centerline is related to the resolution of the selected industrial camera. Improving the resolution of industrial camera can improve the extraction accuracy of weld trajectory centerline. Take 20 coordinate points on the center line of each weld track. Error measurement is carried out with the weld center of the original weld image. The extraction results of the weld trajectory center line and the measured error curves are shown in Figures 12–17. The red line in the error curve indicates that the maximum error value range is 0.12 mm, and the blue curve indicates the error curves of the weld trajectory center lines with different shapes.
It can be seen from the error graphs of the center line of weld trajectory with different shapes.
For the weld image with cusp, such as right angle, more than 90° broken line and less than 90°broken line, there is a large error in the center line of the weld track at the cusp. The maximum error value is 0.105 mm, which is less than the maximum allowable error value of 0.12 mm, which is generated from the center line of the weld track of less than 90°broken line. For weld images without cusps, such as straight and curved welds, the error fluctuation range is small, which can be controlled within the range of +0.07 mm. Through the error analysis of weld trajectory centerline with different shapes, it can be verified that the errors produced by the extraction algorithm of weld trajectory centerline studied in this paper are all less than the maximum allowable error, which meets the system accuracy requirements.
Finally, in order to prove that the welding robot using this image enhancement technology has a lower error rate, we have done two groups of experiments, which are the center line error in straight and curved welds in Figures 18 and 19.
Through comparative experiments, it is found that the accuracy of process products based on image processing has been greatly improved, especially the weld error before enhancement has exceeded the standard value in several points, which is not allowed for process products, which further shows that our technology has brought a breakthrough in the design of process products. At the end of the article, some conclusions are given without in-depth analysis. Image enhancement technology can improve the accuracy of products in process product design. The improvement of accuracy brings a new direction to the development of process technology, continuously integrates into high-precision equipment, does not rely on imports, and realizes technological breakthroughs and opens up new ideas.
5. Conclusion
Due to the increasing amount of information on images, the requirements for the speed and clarity of image processing are also increasing. The existence of various external factors will lead to errors and distortions between the produced image products and the target objects. In this paper, the process product design based on multimode interaction of image processing is studied. It uses radiometric correction and geometric correction to process distorted images, and GPU parallel computing technology is used to accelerate the correction process. In order to validate the effect of image enhancement technology in this paper on improving process product design, it is applied to the visual recognition of welding robot. As the most important visual recognition of welding robot in welding is the first process, the image quality of recognition is directly related to the follow-up process. Finally, through experiments, it is found that the error value of the center line extracted by welding robot through image enhancement is less than the industry requirements. It shows that the image enhancement technology based on image processing studied in this paper can improve the design of process products and make the design of process products more authentic and reliable.
Data Availability
The experimental data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.