Abstract
The metropolis of the future demands an efficient Automatic Number Plate Recognition (ANPR) system. Since every region has a distinct number plate format and style, an unconstrained ANPR system is still not available. There is not much work done on Pakistani number plates because of the unavailability of the data and heterogeneous plate formations. Addressing this issue, we have collected a Pakistani vehicle dataset having various plate configurations and developed a novel ANPR framework using the dataset. The proposed framework localizes the number plate region using the YOLO (You Only Look Once) object detection model, applies robust preprocessing techniques on the extracted plate region, and finally recognizes the plate label using OCR (optical character recognition) Tesseract. The obtained mAP score of the YOLOv3 is 94.3% and the YOLOv4 model is 99.5% on the 0.50 threshold, whereas the average accuracy score of our framework is found to be 73%. For comparison and validation, we implemented a LeNet Convolutional Neural Network (CNN) architecture which uses the segmented image as an input. The comparative analysis shows that the proposed ANPR framework comprising the YOLOv4 and OCR Tesseract has good accuracy and inference time for a wide variation of illumination and style of Pakistani number plates and can be used to develop a real-time system. The proposed ANPR framework will be helpful for researchers developing ANPR for countries having similar challenging vehicle number plate formats and styles.
1. Introduction
Vehicle ownership is increasing proportionally with the economic growth that makes the management and governance of the transportation system complicated. Violation of traffic rules, overspeeding, and car theft are common practices. The detection and retrieval of number plates from fast-moving vehicles make it hard to catch and penalize the culprit. The traffic congestion and unavailability of parking slots lead to the problem of time, fuel consumption, and air pollution. Till today, the vehicle number plate is often noted manually, and human errors in record keeping are unavoidable. Indeed, there is a need to have an automatic and efficient device for detecting, collecting, and managing car information.
In the era of the fourth industrial revolution, an Intelligent Transport System (ITS) is a necessity in which the ability to share information without any human intervention is possible by the use of Artificial Intelligence (AI) and the Internet of Things (IoT). The most important subsystem of an ITS is Automatic Number Plate Recognition (ANPR). The ANPR system reads the image, preprocesses it, and recognizes the vehicle number plate characters independent of human involvement. It helps to identify potential risks, prevent crime, improve reliability, develop barrier-free infrastructure, and provide location information. The Global Automatic Number Plate Recognition System Market is forecasted to increase with a ratio of 9.63% from 2017 to 2025 [1].
The studies show that existing ANPR methods are not viable, and it is hard to find a single efficient approach for different regions due to the unique format and style of each region. Several parameters, namely, vehicle pace, illumination, image dimension, and resolution affect the rate and accuracy of recognition. Considering the case of Pakistani vehicle number plates, another challenge is that no standard formulation of plates is followed in Pakistan. Unlike other region’s plates, plates in Pakistan are both double and single rows and have no fixed number of characters. Moreover, there is no standard Pakistani vehicle number plate dataset available. To address these issues, we collected our dataset having heterogeneous plate formations, i.e., double or single row, varying font styles, namely, Charles’s wright, Barcelona, indigo, nonconforming background having stickers, and orientation, which is followed in various areas of Pakistan. We reviewed the existing methods applied in different parts of the world and implemented few methods on Pakistani number plates. Our analysis showed that the factors affecting the ANPR performance are ghastly plates, low illumination, plate orientation, diverse fonts and languages, and imperfect hardware vision. We developed a novel ANPR framework that works efficiently for various plate configurations and addresses the challenges to a large extent. It has been examined on several images with low resolution (416 × 416). The performance of our proposed OCR method is discussed and further compared with existing methods to help develop an efficient ANPR for Pakistani number plates.
The remaining paper is structured as follows. A detailed review of previously applied techniques is presented in Section 2, whereas in Section 3 the proposed Framework of ANPR is discussed. In Section 4, the proposed methodology using OCR Tesseract for recognition is implemented. In Section 5, results and a comparative study of the proposed methods in tabular form are discussed. Finally, Section 6 concludes the paper and presents the future work.
2. Review of Previous Work
Recently, Automatic Number Plate Recognition (ANPR) has turned out a primary mechanism for security and traffic administrations that spans from congestion supervision to vehicle parking leading to data maintenance for surveillance causes. ANPR offers a simple description of vehicles and a guide for more vehicle tracking and behavior analysis. The big range of number plates, different in color, form, size, and pattern, is one of the main LPR challenges. Such challenges include adverse weather, bad lighting, and a limited camera resolution, as they affect image clarity in real time as a camera captures the plate. In the field of ANPR, a detailed review/comparison of formerly existing systems, their performance, their inferiorities, and their limitations are discussed in the following.
In 1998, PC-based number plate recognition systems were common. C. Coetzee proposed the system [2] that uses the Niblack thresholding algorithm over digital gray level car images that transcend all binarization methods used in comparable systems. The position and size of the number plates are detected by an efficient rule-based algorithm. The segmentation of character is done with blob-coloring and is transferred to the neural network optical character recognition (OCR) method as a 15/spl times/15-pixel bitmap. The proposed feature reduction method lessens the input features from 255 to 50. Six characters are recognized using six small networks running in parallel. The system is performed with an accuracy of 86.1 percent for single- and double-line plates under variable brightness and annotations.
As time passed, many improved ANPR systems were developed to identify the vehicle number plates, but recognition remained a challenge due to the diverse formation of plates in different regions. The work [3] focuses on Nepali vehicle number plates for noting the performance of automobiles license plate identification system based on template matching. The digital camera captures the images that are processed to obtain the vehicle plate information. Different algorithms are applicable for plate detection, including morphological operations, edge detection, smoothing, and filtering. Characters are segregated for identification by using a template matching standardized interrelation and phases correlation, and the result is differentiated into 70 × 70 blocks and correspondence with the repository templet. For consistency, the characters are separated. Ninety patterns under serval conditions are assessed using the following approach. After application on serval images, the study exhibits cross-correlation with greater precision than phase correlation. The exactness of normalized cross-correlation and phase correlation was 67.98% and 63.46%, respectively.
Real-time identification of number plates using Raspberry Pi is also considered as detection and classification of characters that are additionally utilized in surveillance, congestion, and access control applications. The common ALPR systems that come under image processing uses MATLAB as an exclusive tool. As vehicle recognition systems have complex features, a substitute and more systematic method for automated number plate identification is Open Computer Vision (OpenCV). OpenCV is used by the OCR method in [4]. The significant deviation between the MATLAB and OpenCV is a mode of the procedure: online and offline. For online processes, MATLAB cannot be done, while OpenCV is ideal for real time. Another quicker way of using Machine Language python and the Open Vision Library is provided in the paper.
All formerly means of ALPR systems work under constraints like frontal view, standard font, and good illumination, but the introduced novel Convolutional Neural Network (CNN) is efficient for skewed and disoriented depictions in [5]. YOLOv2 is used as a black box for car detection with a precision of 76.8 percent. The suggested Warped Planner Object Detection Network (WPOD-NET) explores the number plate (NP), resized to keep the NP identifiable, and creates rectangular bounding boxes. The CNN architecture comprises 21 convolution layers with 14 inside residual blocks, 3 × 3 ReLU activation, and 4 max-pooling layers sized 2 × 2. Annotated NPs are accomplished by 100k iterations and batch-size 32, where the learning rate is set to 0.001 for training. OCR is utilized for character segmentation and recognition giving an accuracy of up to 75 percent. The unhackneyed methodology is best for oblique images but more affine transformations are yet to explore.
In [6], Aishwarya considered vehicle images with low resolution for automated detection and recognition of NP using a support vector machine (SVM). The model was evaluated for 150 car number plates under particular circumstances. The images are manipulated into grayscale, binarized, and filtered to remove noise. The masking technique assists in extracting the required input image. Lastly, recognition is done by distinguishing definite numbers and characters. The system is deployed and synthesized in MATLAB 2010a. The accuracy of identification is 92.0%. The false results arise because of overlapped, blurred, and slang vehicle or variable style and font plates.
The system in [7] initially captures the license plate’s image, pretreat, and scan every single character available on the number plate separately for their absolute identification. The OCR usage is the most significant phase; here, the letterings on the plate’s portrayal are converted to the textual format which is lately decoded. The specified paper encapsulates an overall computation and organized stream for ANPR and its effective applications. The following method established an exactness of 75 to 85 percent for Indian Number Plate after the concept of template matching for ANPR systems.
Researchers proposed four algorithms for the OCR phase of real-time high definition (HD) ALPR models, in [8]. HD images are preferred for the proposed complex over standard definition (SD) as the precision of recognition improves. HD images take more computing time which is countered by executing the system on heterogeneous Systems on Chip (SoC) and Field Programmable Gate Array (FPGA). The input HD image undergoes resizing and morphological operations. With several promising solutions, Automatic Number Plate Recognition (ANPR) is a reasonably well-explored problem. However, owing to the differences in the characteristics of number plates around the world, these solutions are usually tuned to a specific area. Such characteristics are based on algorithms written for number plate identification, but it will be impossible to realize a universal solution since the image processing methods used to develop these algorithms cannot themselves boast a hundred percent accuracy. An algorithm proposed by Andrew et al. [9] that is optimized to work with Ghanaian vehicle number plates is the subject of this article. The algorithm, written with the OpenCV library in C++, uses edge detection and feature detection techniques for locating the plate combined with mathematical morphology. For recognition, they used OCR Tesseract.
Fakhar et al. proposed an affordable mobile ANPR system using Raspberry Pi [10]. The model utilizes a real-time image taken from the camera. The captured image is desaturated, filtered, and segmented, and the plate’s characters are recognized. All computation complexity is handled on the Raspberry Pi, and a noticeable 3-second delay is observed before achieving the final output on the LCD. The resultant label is also stored in the database.
In [11], a plate recognition framework employing deep learning is designed. The image dataset consists of 6500 Indian car number plates that are split into 90 and 10 percent for training and testing, respectively. The images have diverse qualities because these are captured from 3 cameras having different specifications, like bit rate and focal length. For character recognition 37-class CNN model is trained. The model comprises 126 filters. YOLO v3 is utilized for the detection and identification of car plates. An accuracy of 100% and 91% is achieved for plate detection and number plate recognition accordingly.
Moreover, in [12], an embedded system based on KNN Machine Learning for number plates following Polri regulations is developed. The attained model exactness using KNN for character recognition is 84%. The images are taken by webcam with a resolution of 640 × 480 pixels. They undergo cropping, gray scaling, inversion, thresholding, edge detection, and morphological operations before contouring in segmentation. Character improvement increased the accuracy of the recognized number plate model to 98%.
Ariff et al. in [13] explore variant segmentation techniques for processing a noisy number plate image. Hundred Malaysian car plates of 1932 × 2576 pixels are passed to threshold techniques, namely, Savoula and Niblack method for removal of undesired pixels. In this, Savoula segmentation gives an average accuracy of 83.17% which compromises the accuracy of recognition. The classification of characters is done by the template matching technique.
In [14], a speedy and more accurate vehicle nameplate recognition system is proposed based on optical character recognition (OCR) engine “Tesseract”. Raw images go undersampling and quantization. Then, region of interest (ROI) is obtained by a binary mask. The input plate image goes through grayscale, global image thresholding by the Otsu method, and noise removal. The processed image is passed to OCR where character segmentation takes place. The segmented decision is a string that is stored in a text file. The overall precision of the model lies between 90 and 100%.
In [15], using the ANPR idea for smart check-in and check-out is narrated to minimize waiting time and keep records of vehicle entries. Images are captured from CCTV cameras, so ANPR instantly identifies the plate number via image processing and stores vehicle registration and security information in a database to access it electronically through a web application. The protection and approved staff may monitor the progress of vehicles with the details of their owners through this device. The proposed solution would minimize check-in time while bringing additional advantages in terms of parking and traffic surveillance.
A state of art image processing technique for dealing with noisy and substandard Indian number plates, in [16], as previously presented models, performs well under controlled conditions such as clear text, fine illustration, and perfect angles. The shoddy images undergo morphological operations and Gaussian rectification as preprocessing. Plate detection is done by applying contours formulated on character cartography and proportionality. Then, the extracted region of interest is filtered and deskewed to pass to the KNN model for character identification. The following work gives an accuracy of 96.22%.
Virakwan and Nui Din in [17] suggested Automatic Vehicle Number Plate Recognition System in POLIMAS to verify that only permitted automobiles set foot on the campus. A webcam is installed considering four orientations: front side, rear side, front top, and rear top. The seized image is converted to grayscale; then histogram equalization technique helps to adjust the intensity and contrast. Two combinations which are Canny and Laplacian edge detector and Sobel edge and Laplacian edge detectors are introduced. The best one will be contemplated in the process. Succeeding plate discrimination, the bounding box method is used for identifying and cropping characters. Each character is compared in OCR using eigenvector and correlation. The outcomes are accumulated in string configuration and differentiated with the reserved entries in the database. Barrier unbolts if the plate matches the database; else, entrance is opposed.
A 2020 study published by Shivani et al. [18] reviews ANPR systems for vehicular applications deployed to track the traffic jam, intelligent parking complex, toll fee collection and protection of the drivers, etc. Numerous propositions are used to build mechanized vehicle plates, namely, Cloud-Computing based, Artificial Neural Network (NN) based, morphological based, composite based, template matching based, etc. A study shows 97.7% of precision where OCR, NN, Support Vector Machine (SVM), and Hierarchical Temporal Memory (HTM) are applied. Another research indicates that the histogram approach with Machine Learning (ML) fails to detect vehicle number plates under diverse illumination. Further, genetic NN and forward backpropagation NN are capable of variable segmentation and recognition. OCR has the lion’s share when evaluating techniques for ANPR, but more coherent systems are still to be presented.
Agrawals in [19] portrayed a Cognition Number Plate Recognition System centered on Machine Learning (ML) and Data Visualization techniques. Loaded images passed through the minimum filter and changed to grayscale for getting a subimage of the localized number plate. Further characters are circumscribed and parsed for character identification using ML methods. 20 images for a total of 36 characters are converted to a one-dimensional array. These arrays are considered a feature for model training. The trained model recognizes segmented characters and passes across template matching. The outcome is exhibited to the user and saved in the database. K means clustering on a dataset is performed for number plates considering car brands, namely Nissan, Ford, Honda, etc. The plate localization accuracy is 97.2% while it is 73.4% for character identification. The average precision of the proposed model that efficiently tackles adverse conditions is 88.38%.
Although there are a lot of vehicle recognition systems in Europe, the USA, China, Korea, India, Singapore, and several other well-developed countries, hence there is no such ANPR system for Pakistani number plates. Nevertheless, little work is available for Pakistani number plates recognition, due to the variety of nonstandard number plates. An attempt to develop a usable and efficient ANPR system for the multiscript Pakistani number plate is done in [20]. The method used distinct feature matching acceptable for Pakistani plate conditions. The introduced method was tested on 50 images and had a margin of improvement for further implementation in a real-time environment. A rational methodology especially for Pakistani number plates recognition is designed in [21]. A dataset consists of 900 images collected from six different regions of Pakistan. 80% standard and 20% nonstandard number plates following some constraints of distance, brightness, and pixels. The captured image is transfigured to grayscale, and for noise removal, it is passed to the Gaussian filter. As to approach plate localization Canny edge detector and contouring are carried out. Subsequently, every distinct character is fragmented into distinct images. The KNN technique is utilized for the recognition of characters. The average performance is 93 percent.
Table 1 depicts the summary of the comparative study of the previously presented methods for ANPR. After analyzing the performance metrics of the proposed methods, we inferred that their localization techniques for number plates would not generate the best results. The performance rate column defines the existing method’s performance rates in terms of accuracy and mean average precision score. After this, the discussion and remarks column shows some major points which are discussed in previously proposed papers and are affecting their performance rate.
3. Proposed ANPR Framework
After reviewing the previous papers, we observed that everyone followed the typical ANPR framework [22–24] in which they usually used character segmentation or template matching having average results.
In the United Kingdom, one of the cop stations invented the primary Automatic Vehicle plate recognition system [25]. Moreover, many hardware and software-related ANPRs have been developed using evolving technology to improve accuracy. But unfortunately, corresponding to traffic concerns, adequate precision is not attained [4]. Vehicle plate localization and identification become a daring assignment [26]. An ANPR [27] is a framework that scans the alphabets and digits on the vehicle number plate.
We proposed a novel ANPR framework in Figure 1 that gives better results in less computation time. The five phases that comprise our ANPR framework are as follows: number plate image capture, plate extraction, image preprocessing, character recognition, and number plate label management.
3.1. Image Acquisition
Image acquisition is the fundamental stage of any vision system. The following stage, known as image acquisition, is to transform an optical image (real-world features) into a numerical data array for future manipulation. Pakistani vehicles images are considered for further preprocessing. We used low-resolution images of Pakistani number plates having various formats in terms of illumination, orientation, brightness, etc.
3.2. Number Plate Localization
The number plate can be detected anywhere in the vehicle. The image is devised for robust plate detection. ANPR performed under constraints as brightness, angle of the plate, and resolution can use basic image processing techniques discussed in [23, 28–30]. Contouring is a method used for shape scanning and entity localization and identification but it would not give us accurate results in all situations; for example, the image is tilted. For the identification of number plates, a deep neural network-based YOLO (You Only Look Once) model is recommended. To analyze the performance of the YOLO family on proposed ANPR methods, we have trained YOLOv3 and YOLOv4 to localize the number plate region.
3.3. Image Preprocessing
Image preprocessing [31] is substituted for functionalities performed on nonfigurative images. The objective is the augmentation of image information that subdues the undesirable deformation or amplifies features for further process. The methods involved in the paper for image preprocessing comprise gray scaling, binarization, thresholding, and histogram equalization, etc. During this stage, such images are obtained which are passed to algorithms for further training and predictions.
3.4. Character Recognition and Identification
In the recognition stage, the characters from the image are recognized by using different approaches. A commonly used technique behind character recognition is the Artificial Neural Network (ANN). ANN is considered a script since the neural network comprises interconnecting artificial neurons and a mathematical definition by using feedforward backpropagation (BP). In [25], BP is proposed as a neural network-based method. The most important neural network for character recognition is the Convolutional Neural Network (CNN) which is used in Section 4 as an implemented method of the previous ANPR system. Our proposed ANPR method for recognition involves OCR Tesseract, which is the time-efficient method.
3.5. Number Plate Label Management
At this stage, the number plate’s labels are stored in a file to keep a record. We have reconstructed the recognized characters into plate label strings and stored them in the excel file which is further utilized for scoring.
4. Implementation of ANPR Methods
According to the above-proposed ANPR framework, we implemented two pipelines having different recognition techniques; the first one includes the deep learning method (CNN) and the second one is OCR Tesseract. These methods being used are applied in different phases from image number plate localization to recognition of plate’s characters.
4.1. Dataset Acquisition
Due to commercialization and privacy concerns, there was no publicly available dataset of Pakistani number plates. We, hence, prepared our real-time dataset by capturing 1000 images of vehicles having various formats of number plates in terms of size, fonts, plate color, and orientation practiced in different regions of Pakistan; see Figure 2 which depicts some examples of our vehicle dataset. From two different cameras, i.e., 900 images were captured by the Samsung J series cell phone, and 100 images were captured by Oppo mobile phone. All images were taken from different angles considering environmental conditions like illumination, dust, and fog, etc. The vehicle number plates are comprised of a subset of 36 characters set in which 26 characters are alphabets and 10 are numbers. We resized all high-resolution images into low-resolution fixed-size (416 × 416) images using an online web service given by Roboflow [32]. This helps to make the dataset realistic as the images received from CCTV cameras mostly have low resolution. Further, reduced frame size decreases inference time as the YOLO network runs faster on small-size images. For training the Darknet object detection model, without using any augmentation technique, we split our dataset into 900 training images or 100 validation images. Further, the performance of the model has been examined on 200 images out of these resized 1000 images.
4.2. Number Plate Localization on Pakistani Vehicle Dataset Using Object Detection
The inspiration to prefer YOLO (You Only Look Once) [33] model can be found from the literature review where the number plate localization using YOLO models exhibits greater precision. YOLO is a quick and efficient real-time open-source system trained to object detection. To localize the object, the basic working of the YOLO model is applied to an image at multiple scale and location, puts some scores on multiple regions, and then selects only prioritized high scoring of the region as a detection region. We have used two versions of the YOLO family; the first is YOLOv3 and the second is YOLOv4.
Table 2 depicts the parameters which we have set for training the YOLOv3 and YOLOv4 models on images. The learning rate for both models is 0.001. All parameters are the same as for the YOLO family except the maximum batches because it depends on the number of classes.
Figures 3 and 4 depict the average loss and mAP score of YOLOv3 and YOLOv4. The dotted line shows the average loss rate throughout all iterations and batches. While training, the graph shows the average loss under 2.0 which makes the model efficient. The solid line shows the mean average precision score. We achieved 94.3% mAP using YOLOv3 and 99.5% mAP using YOLOv4 on the 0.5 thresholds. The best mAP score is achieved by YOLO v4. The results of localization using both versions are discussed in Section 5.
4.3. ANPR Method Using CNN
The CNN is a deep learning algorithm which is a special kind of multilayer perceptron using Gradient Descent Backpropagation learning algorithm. It is proved to be the best neural network in recognition and prediction. To compare the performance criteria of previous methods, we have also trained a CNN-based model by using TensorFlow and Keras. TensorFlow and Keras are python libraries that are mostly used to train models based on large datasets. The dataset which we are using for training purposes is collected from 36 classes of characters in which 864 are training images and 216 are validation images. We gathered a collection of images of characters and then performed rotational and brightness techniques. The dataset which we are using for testing purposes is collected after training the YOLO Number Plate Detection model in which all image resolution is 416 × 416. We trained 6 layers’ sequential CNN model. The modified LeNet architecture of our CNN model is discussed as follows. The model comprises a convolution layer with 32 filters, kernel size (5, 5), and ReLU (Rectified Linear Unit) activation function with an additional max pooling layer with pool size (2, 2). To avoid overfitting, a drop-out of 0.4 is utilized which drops 40% of neurons, while to flatten the node, the information flatten layer is added. Then, a dense layer with 128 outputs and ReLU activation function is added. And the last layer comprises a dense layer with 36 outputs and Softmax (probabilistic final decision) activation function. The 36 neurons are for the total number of outputs as 26 alphabets and 10 numbers. The learning rate we used for training is 0.001 with 50 epochs, categorical_crossentropy loss function, and Adam optimization function. The CNN model is making predictions with 97.89% accuracy, but after reconstructing the number plate label, the accuracy decreases.
Figure 5 illustrates the overall pipeline of the CNN model, while Figure 6 portrays the resultants of the CNN method series. First, the segmentation of the cropped number plate is done using the CNN model. The CNN generates a string of recognized characters which are ordered before saving into records. The reason behind not considering the CNN for further processing is mainly related to time resources.
4.4. Proposed ANPR Method Using OCR
Character recognition from the processed ROIs is done using Tesseract. Tesseract is an open-source API used by the optical character recognition engine with the ability to identify and recognize more than 100 languages. We used pytesseract (Python-Tesseract) as an open-source module for python which recognizes the read the text. It is a wrapper of Google Tesseract OCR Engine which can read all image types and returned the recognized text instead of writing it to a file. Pytesseract is easily used as we simply install the Tesseract OCR engine into our system and then install this library and used it. This library used page segmentation and OCR engine mode. To run OCR on a small region of an image with a different orientation, the page segmentation mode is used. It has 13 engines. The OCR engine mode has a 2 OCR engine which is used for LSTM and legacy engines.
As we know, the OCR gives precise results when a standard image (Perfect Resolution and White background with black characters) is passed. We have taken preprocessing steps for this as it offers an outstanding input option for the OCR stage. Figure 7 portrays the proposed OCR Tesseract-based ANPR pipeline in which the following steps applied on each frame are described in detail. To start with, we have done gray scaling of the extracted image for a better result. It removes all color information leaving the brightness of each pixel. Bilateral filtering is selected which is a denoising technique for smoothing the image. This reduces the redundancy while preserving edges through a nonlinear grouping of adjacent pixel values. The intensities are adjusted to improve the global contrast of the image using histogram equalization. It also enhances the edges of each object region of an image. The conditions of lighting differ widely over a plate for which thresholding is important. The binarization process is used to convert grayscale images into black and white pixels (0 bits for black and 1 bit for white). The pixels which have less value than a threshold are converted to 0, and the above threshold image is converted into 1. We used morphology to exclude small objects from the image. The opening function of morphology not only excludes the small objects from the image but also retains the shape and scale of the larger object in the image. The performance of the OCR engine is further enhanced by sharpening. This increases the visibility level of characters and makes it readable for the OCR engine. The results of our proposed ANPR framework are also discussed in Section 5.
5. Results and Discussion
In this section, the results of the proposed framework have been presented. We have verified and validated our proposed framework by experimenting on a real dataset and comparing it with the previously proposed approaches using the same accuracy metric.
5.1. Results of Proposed Pipeline on Single- or Double-Row Vehicle Number Plates
The raw image is passed to the YOLO model for plate detection. YOLO Darknet neural network framework is applied on a single neural network to image, divides the image into the region, and predicts bounding boxes and scoring probabilities for each region. The highest probability is selected as an object detected area. It can easily localize every type of number plates of vehicles including cars, busses, trucks, and bikes.
To localize the number plate area, we have trained YOLOv3 and YOLOv4. After observing their mAP score, we prefer the YOLOv4 as it gives us the best mAP score. Figure 8 depicts the localization results of YOLOv3 on a double-row number plate. Figure 9 represents the OCR-based ANPR pipeline using YOLO v3.
Figure 10 and Figure 11 exhibit the localization results of the YOLOv4 Darknet model on single-row and double-row number plates, respectively, by creating the boundary around it. The vehicle plate is cropped according to the drawn outline. Extraction of cropped images is done by using the coordinates of bounding boxes.
The localized cropped image is passed to the chain of preprocessing techniques like gray scaling, noise removal, morphology, and binarization before proceeding to the OCR engine for best recognition of the car’s plate label in Figure 12.
Table 3 summarizes the performance of implemented pipelines involving CNN and OCR Tesseract. To compare uniformly, we used the same metric for measuring the performance of the OCR Tesseract and CNN model. We have calculated the average score using fuzzywuzzy python library in which every predicted string is compared with the actual string and generate a score on the base of correct occurrences of string character. For the evaluation of the proposed method, we used a 64-bit system with a Core i5 CPU having a 3.00 × 6 Gigahertz processing speed. After analyzing our pipeline computation time on CPU, our proposed method works well with less computation time of 0.8 seconds per image. We prefer the YOLOv4 for localization as it is efficient and has more precision rate than the YOLOv3.
5.2. Comparative Analysis with Previous Papers
In spite of the fact that there are a lot of vehicle number plate recognition systems for Europe, the USA, China, Turkey, Korea, India, Singapore, Malaysia, and many other well-developed countries, for Pakistani number plate identification, little work is present and it still needs more consideration due to the diversity of nonstandard number plates. The proposed research work is targeting the identification and visualization of Pakistani number plates by using the OCR Tesseract library. We compared our ANPR framework with an existing ANPR method involving the KNN approach proposed for Pakistani number plates [21].
In Figure 13, we inferred by using the preprocessing methods of [21] on our images that their pipeline did not recognize the characters well due to the poor localization. Further, their preprocessing techniques are only applicable to car vehicle types.
In the view of Table 4, when we applied their [21] ANPR framework on our dataset, the processing computation time taken per image is approximately 44 seconds with an average score of 9% using the fuzzywuzzy library. It did not recognize the plate characters well, due to poor localization done by edge detection. Moreover, after applying our ANPR approach on their given dataset of resolution (1140 × 641), we achieved a better outcome with a 71% average accuracy score in less computation time as the localization is improved by the use of YOLOv4.
From the comparative analysis in Table 5, we can easily observe that our proposed ANPR system achieved the best results. The previous ANPR system used different techniques, namely, edge detection, histogram equalization, and CCA for localization which is not an efficient way to handle every type of vehicle number plate. Their preprocessing techniques did not work well on double-row number plates due to the poor localization approach. Our employed YOLO-based ANPR framework performs fast and more efficiently for detecting such challenging plates. They used different metrics to evaluate their performance rate; hence, we used the average score metric for our proposed method by usingthe fuzzywuzzy python library.
Therefore, considering the comparative study of our framework with the existing methodologies, we concluded that, by using our proposed pipeline on the Pakistani dataset, a better result would be achieved. Our proposed pipeline strategies are universal and applicable on every type of Pakistani vehicle datasets such as cars, bikes, and busses having double and/or single-row number plates.
6. Conclusions
This paper presents a novel ANPR framework using OCR Tesseract to address the vast heterogeneity and assortment of number plate genres across the provinces in Pakistan. The proposed OCR pipeline is tested on low-resolution images which are collected by considering the variations in the environment illumination and image orientation. The obtained mAP score for the plate extraction using YOLOv3 is 94.3% and YOLOv4 is 99.5% on 0.50 thresholds. The robust preprocessing techniques are applied to the localized plate image, and finally, the processed image is passed to the OCR Tesseract to recognize the number plate. Our introduced pipeline with the YOLOv4 model takes a 0.80-second average computation time per image and gives an average accuracy score of 73% for recognition. We have also processed our dataset using a previously proposed APNR framework based on KNN and got an average score of 9% due to the poor plate localization technique. Moreover, we found that LeNet CNN architecture gives 97.89% accuracy for character recognition on our dataset but an average score of 67.76% for complete plate recognition. Furthermore, we experimented with an available constrained high-resolution dataset on our proposed framework and gained an accuracy of 71%. The comparison between our proposed method and the existing ANPR approaches shows that our proposed framework using YOLOv4, preprocessing steps, and OCR Tesseract is applicable in real time. It can be concluded that, to the best of our knowledge, it is the first robust and low complexity ANPR framework for Pakistani number plates that gives reasonable accuracy for unconstrained plates. Finally, this paper can be useful for researchers to develop an ANPR framework for the countries having similar challenging vehicle number plate formats and styles. A promising direction for future research is to test our proposed framework on real-time scenarios that can be installed commercially and improving the pipeline to counter unconstrained scenarios like the distance between camera and vehicle.
Data Availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported in part by the National University of Computer and Emerging Science, Chiniot Campus Pakistan. The research work was funded by an Internal National University of Computer and Emerging Science (NUCES) University research grant for faculty.