[Retracted] Face Security Authentication System Based on Deep Learning and Homomorphic Encryption

Abstract

The development of deep learning technology has promoted the wide application of face recognition in many scenarios such as mobile payment and social media, but the security of user data is facing great challenges. To protect the privacy of users, face authentication cannot be operated in plaintext. To solve this problem, a face feature ciphertext authentication scheme based on homomorphic encryption is proposed. First, the face image feature extraction is completed based on a deep learning model. Second, the face features are packaged into ciphertext by using homomorphic encryption and batch processing technology, and the face feature ciphertext is saved in the database of the cloud server. Third, combined with automorphism mapping and Hamming distance, a face feature ciphertext recognition method is designed, which can complete face recognition in the case of ciphertext. Finally, the integrity and consistency of face feature ciphertext recognition results before and after decryption are guaranteed by the one-time MAC authentication method. The whole framework can finish identity recognition without decrypting face feature coding, and the homomorphic ciphertext of face feature coding is saved in the database, so there is no risk of face feature coding leakage. Experiments show that the system has met the requirements of real application scenarios.

1. Introduction

In recent years, with the continuous development of artificial intelligence technology with deep learning as the core, the face recognition system has been widely used in mobile payment, social media, and many other scenes. The wide application of this technology also makes it easy to become the target of malicious attacks. If facial features are directly stored in the database in plaintext, the risk of disclosure of registered users’ biometric privacy will be greatly increased, which will seriously affect the security of the authentication system. Therefore, as an authentication system, it is particularly important to develop a solution with stronger protections for biometric data.

The privacy protection of biometric data has always been a research hotspot in academic research. To solve this problem, researchers have proposed many solutions based on different technologies. Belguechi et al. [1, 2] proposed to convert characteristic data into random data by using a hash function or password. This method is practical in performance, but if the user password is broken, it is no longer secure. Fuzzy vault-based approaches [3, 4] bind the user’s biometrics with secret information to generate real points and produce the vault by adding a large number of hash points. It can encrypt the biometric template while protecting the biometric information, so it has been widely used [5]. However, due to the invariance of biometrics, it is easy for attackers to obtain real points from the biometric-based fuzzy vault, resulting in the permanent loss of biometric templates.

Fontaine and Galand [6] proposed a homomorphic encryption scheme that can compare and calculate on the ciphertext. This scheme greatly improves the security of data, but due to the use of multiparty computing it needs interactive computing between multiple parties, which reduces the efficiency of computing. Another scheme uses the Paillier homomorphic encryption system, but the scheme requires that the participants must be honest and credible, and the scheme is limited to a face recognition system [7].

A fully homomorphic encryption (FHE) system supports the arbitrary operation of ciphertext without decryption [8]. This special property makes FHE have a wide range of theoretical and practical applications. An IBM researcher Craig Gentry [9] proposed the first FHE scheme based on bootstrapping technology on an ideal lattice. Although this scheme cannot meet the practical feasibility, it opens a new chapter in the research of homomorphic cryptography. Dijk et al. [10] proposed DGHV algorithm based on integer ring. This algorithm constructs a homomorphic encryption scheme according to the difficulty of approximating GCD (great common divisor) [11], which is transformed into a homomorphic public-key encryption algorithm through simple transformation and then transformed into a fully homomorphic encryption scheme by bootstrap technology. The scheme is simpler than Gentry’s ideal lattice scheme, but the operation efficiency is still not high and the storage space of the key still needs to be large. On this basis, Brakerski and others proposed a homomorphic BGV encryption scheme based on integer ring module switching technology, which greatly reduces the storage space of the key and significantly speeds up the operation efficiency [12]. Ducas and Micciancio proposed a new method of homomorphic computing bit [13] operation, which improved the efficiency of calculation to a certain extent. Xiang et al. [14] proposed the privacy protection online face authentication scheme in an outsourcing scenario based on the FHE scheme, which avoids the decryption process with large computational consumption in the homomorphic encryption algorithm. Although there is a great improvement in efficiency, there is still much room for improvement.

FHE-based schemes often require high computational overhead, which is not applicable in some scenarios with high real-time requirements or resource constraints. To address the limitation of the computational complexity, Vishnu et al. [15] proposed a scheme based on FFE, which uses batch processing and dimension reduction methods to decrease the computational complexity, and this achieved good performance. However, in this scheme, the ciphertext authentication result is sent to the client for decryption and is not returned to the server. Therefore, it lacks the verification of the calculation result and cannot be applied to the cloud server scenario.

Therefore, to prevent the client data from being tampered with and further improve the computational efficiency of the whole system, a fully homomorphic encrypted face recognition scheme based on Fan–Vercauteren (FV) scheme is proposed. It does not use trusted hardware and adopts one-time MAC authentication, which well protects the user’s face feature template and completes the corresponding face authentication.

In summary, the following contributions are made in this paper: (1) a face recognition security system is designed based on the FHE scheme, batching technology, and Hamming distance (HD) calculation, which greatly improves the efficiency and flexibility of calculation; (2) the one-time MAC authentication method is directly utilized on the server, removing the trusted center for authentication. This scheme ensures the integrity and consistency of face feature ciphertext recognition results before and after decryption; and (3) improved face recognition technology and dimension reduction methods are used to further decrease the computational complexity.

2. Materials and Methods

2.1. FV Fully Homomorphic Encryption Algorithm

The FV scheme in this study is based on the ring R = Z[x]/(xⁿ + 1). The elements in R are polynomials with integral coefficients of degree less than n, and n is always a power of 2. Let λ be the security level, q the ciphertext module, and t the plaintext module. ω is the base for decomposing the integer coefficients, and means decomposing the integer d into parts [12]. The algorithm is as follows:(1)GenKey (λ) An element s ← R₂ is randomly and evenly selected as the private key in R₂, and then a₁ ← R_q is randomly and evenly selected in R_q. Meanwhile, an error e ← χ is randomly selected from Gaussian distribution χ, and a₀ = −(a₁s + e) mod q is calculated. The output is a private key and public key (s_k, p_k) = (s, (a₀, a₁)).(2)EvKeyGen (s_k, ω) Let randomly and evenly select the element a_i ← R_q in R_q, and randomly select the error e_i ← χ with Gaussian distribution χ, and output the calculated public key ev_k = ((−(a_is + e_i) + ωⁱs₂) mod q, a_i).(3)Encrypt (pk, m) To encrypt the message m ∈ R_t, an element u ← R₂ is randomly and evenly selected from R₂, and the error e₁, e₂ ← χ is randomly selected from Gaussian distribution χ. According to the public key, pk = (a₀, a₁), c₀ = (Δm + a₀u + e₁) mod q, and c₁ = (a₁u + e₂) mod q are calculated, and the ciphertext c_t = (c₀, c₁) is output.(4)Decrypt (s_k, c_t) According to the ciphertext t = (c₀, c₁), using the private key s_k = s, m′ = ((t/q × (c₀ + c₁s)) mod q) mod t is calculated.(5)Add (c_t0, c_t1) Input the two ciphertexts c_t0, c_t1, and output the sum of c_t0 + c_t1 by calculating (c_t0[0] + c_t1[0], c_t0[1] + c_t1[1]).(6)Mul (c_t0, c_t1) Input c_t0, c_t1, and output the product c_t0 × c_t1 of the two ciphertexts by calculating as follows:  Then calculate:  Finally, is the product of the two ciphertexts c_t0 × c_t1.

2.2. Batch Processing and Automorphism Mapping

The main bottleneck of encrypted face matching is the number of homomorphic multiplications needed to calculate face similarity. To improve the processing efficiency, batch processing technology is used in this study, which utilizes Chinese remainder theorem (CRT) and single instruction multiple data (SIMD) [9, 16], n numbers can be packed into a plaintext polynomial, and the operation on this polynomial is the same as on n numbers in plaintext slot. It is conditional to use batch processing: the plaintext module t is prime, t = 1(mod 2n). Under this condition: ζ ∈ Z_t makes ζ²ⁿ = 1(mod t), and ∀m, 0 < m < 2n, there is ζ^m ≠ 1(mod t). It is called the 2n-th primitive unit root of the module t. So we have

According to the Chinese remainder theorem, a ring can be decomposed into two parts:

All above isomorphisms are over rings, which means that both sides of the equation keep the structure of addition and multiplication. The rightmost can be expressed as Z_t × Z_t ×  …  × Z_t. As a result, the addition of the two vectors on the right is actually to perform the same operation of n corresponding elements. Based on additive homomorphism, the corresponding left is only one addition of two polynomials on R_t. Similarly, multiplication is homomorphic. Let  = ζ^2i + 1, we can get the unpacking:

In the same way, the opposite operation is also called packing. Automorphism is a method that can replace the plaintext corresponding to each plaintext slot. If the plaintext is q(k), the corresponding plaintext with each plaintext slot is q(k₀), q(k₁), …, q(k_n − 1). When Frobenius automorphism mapping is used, we can make q(k) ⟶  move i plaintext slots circularly. When i = 2, for instance, the plaintext slot of m(α) circularly moves two steps, and the corresponding plaintext becomes q(k₂), q(k₃), …, q(k_n − 1), q(k₀), q(k₁) [17]. Therefore, we can use batch processing technology and automorphism mapping to make the plaintext move circularly in the ciphertext environment.

2.3. Facial Feature Coding

Facial feature representation is an important part of homomorphic face security authentication. The face recognition algorithm based on deep learning has achieved very high recognition accuracy with the support of the powerful computing and storage capacity of the server. However, due to the limitation of hardware resources and the lack of computing and storage capacity, these excellent models cannot achieve good results when transplanted to the mobile terminal. To apply the face security recognition model based on deep learning to the mobile terminal and make it more widely used in real-life scenes, this paper proposes a method of combining the lightweight network MobileNet and the high-precision face recognition model FaceNet [18, 19] and uses the lightweight network as the basic network of FaceNet model as well as softmax loss and center loss as comprehensive loss functions for training [20].

2.3.1. Face Feature Extraction Model

FaceNet [18] is one of the most excellent algorithms for face recognition at present. It does not need face alignment and other preprocessing operations on the image and directly learns the feature representation from the original pixel value. Its model structure is shown in Figure 1, and FaceNet uses the inception model as the basic network model and achieves very good results. However, this network model has a deep network level, many parameters, and a large model, so it cannot achieve ideal results when transplanted to the mobile device. MobileNet is a lightweight network using deep separable convolution. Depth separable convolution decomposes the standard convolution into depth convolution and point convolution, which play the role of filtering and linear combination, respectively, and reduce the number of parameters and calculations. To reduce the model parameters, this paper uses MobileNet instead of the inception model as the basic network of FaceNet.

Figure 1 

Network structure of FaceNet.

2.3.2. Loss Function

The innovation of FaceNet is to remove softmax, the last classification layer of the network structure, and uses the triple loss as the loss function, which can achieve very good results. However, the choice of tuples has a great impact on the model. A good choice of tuples can converge quickly. On the contrary, it is difficult to converge and cannot achieve the ideal effect. Therefore, it is often difficult to use the triple loss for training. In this paper, softmax loss function-weighted and center loss function-weighted training are used to make the feature distance between similar classes closer and the feature distance between different classes longer, to learn more distinguishing and generalization features.

The formula of center loss (LC), where x_i represents the feature, before the full connection layer, and y_i represents the center of the category, is as follows:

The gradient of LC and the update formula of the category center are as follows:

When using softmax loss (LS) and center loss as the total loss for training, the parameter is used to control the ratio of two. The total loss function is shown in the following equation:

2.4. MAC Authentication Research

After computing the HD of the ciphertext, the cloud server sends the result to the client that will decrypt the plaintext and return the result to the cloud server. There is a security problem, which is how to ensure that the result received by the cloud server is the decryption result of the ciphertext transmitted to the user. To solve this security problem, message authentication code (MAC) is used.

The MAC generally uses cryptographic hash functions such as MD5 and SHA-1 to confirm that the message comes from the specified sender and has not been tampered with [21, 22]. However, this paper needs to verify the binary data decrypted by the front end on the cloud server. Therefore, we develop a one-time MAC authentication algorithm, that is, the cipher generated by the message authentication code can only be used once. The specific scheme description is given below: MkGen (Z_J): let the message key m_k = (r₀, r₁) and r₀ and r₁ be randomly selected from Z_J, where Z_J is composed of J-bit integers MacGen (m_k, m): authentication code of the message m be calculated through m_c = m × r₀ + r₁ Verification (m_k, m, m_c): verify whether m is equal to (m_c − r₀)/r₁ by inputting a key m_k, message m, and message authentication code m_c, and output authentication result . If b is 1, the authentication succeeds, and message m has not tampered with; otherwise, the authentication fails and the message x has tampered with.

2.5. Ciphertext Recognition Method

The face recognition method in this study compares the encoded face feature templates by calculating HD. It takes the number of different corresponding bits on two feature codes as the distance between them [23]. The smaller the distance, the better the matching of the two templates.

Suppose A = (a₁, a₂, …, a_n) and B = (b₁, b₂, …, b_n) denote two binary vectors of length n, as the initial template. HD can be obtained by calculating the sum of XORs of two vectors, that is:

To prevent the user’s biometric information from being leaked in the identity authentication service, we use the characteristics of homomorphic encryption technology to design a recognition method based on the facial ciphertext. First, this paper aims to test the homomorphism performance by converting XORs into a combination of multiplication and subtraction while calculating HD. Second, because the FHE method is based on ring R, it is necessary to encode the facial feature template into integer polynomial. In this paper, the feature extracted from the face image based on the deep learning method is a binary vector with length n calculating the HD between two face image features requires at least N times of multiplication. However, the multiplication time between face feature ciphertexts after homomorphic encryption is very long, which will increase the computational complexity of the system.

Therefore, we develop batch processing technology to package the binary vector with length n into a polynomial, only one subtraction and one multiplication can complete the XOR calculation of the vector. At the same time, using the characteristics of automorphism mapping, the sum of elements in the homomorphic ciphertext slot can be calculated by only log2ⁿ shifts and log2ⁿ additions, that is, HD of ciphertext can be calculated. Assume the vector I = (2, 6, 3, 7), and its corresponding homomorphic ciphertext is I′ = (I₁, I₂, I₃, I₄). Because the length of the slot is four, operations of log2⁴ shifts and log2⁴ additions are needed, here (I₁, I₂) = (2⁰, 2¹). Figure 2 shows the illustration.

(a)

(b)

Figure 2 

Illustration for homomorphic ciphertext slot element summation operation. (a) First shift and addition operation in homomorphic encryption slot. (b) Second shift and addition operation in homomorphic encryption slot.

To sum up, the face recognition method mainly includes the following steps:

Step 1. The binary features A and B of two face images are packaged into plaintext polynomials: BP_A and BP_B ∈ R_t, (BP_A, BP_B) ← (Compose (A), Compose (B)).

Step 2. The plaintext polynomials BP_A and BP_B are encrypted by homomorphism, and the ciphertext polynomials are output: ct_A and ct_B ∈ R_q × R_q, (ct_A, ct_B) ← (Encry (A), Encry (B))

Step 3. ct_A and ct_B are sent to the cloud server S: (ct_A, ct_B) ⟶ [S]:

Step 4. Calculate the HD between ct_A and ct_B: .

2.6. Ciphertext Feature Authentication Protocol

In this study, we use homomorphic encryption to encrypt the biometric template and store the ciphertext, then measure the similarity by calculating HD between the two ciphertext features, and finally authenticate by one-time MAC authentication. The overall authentication protocol of the system is shown in Figure 3. The overall protocol includes two parts: registration and authentication.

Figure 3 

Ciphertext feature authentication protocol diagram.

2.6.1. Registration

At this stage, the user extracts feature vectors from many face images and encrypts them. The specific processing process is as follows: (1) the private key and public key (sk, pk) are generated using the GenKey algorithm; (2) n face images of a registered user are acquired, and the face feature vector Fea is extracted with our method based on deep learning, and n represents the number of samples of registered users; (3) the Fea is packaged into polynomial BP_Fea through batch processing technology; (4) BP_Fea is encrypted and ciphertext ct_Fea is generated using pk, and (5) the ciphertext ct_Fea and identity label U_lab of the registered user are sent to the server. The public key pk, the ciphertext ct_Fea, and identity label U_lab are stored in the database.

2.6.2. Authentication

During this stage, user face authentication is completed through the following process: (1) the current user’s facial image is captured, and the facial feature is extracted and represented as y; (2) y is packaged into polynomial BP_y through batch processing technology; (3) BP_y is encrypted, and the ciphertext ct_y is generated; (4) then, the authentication request (U_id, ct_y) is transmitted to the server; (5) HD Ct_d between ciphertext ct_Fea and ct_y is calculated (equation (7)); (6) the server randomly selects (r₀, r₁) from the Z_I, outputs the message key mk = (r₀, r₁), and calculates the message authentication code ct_T of the HD by ct_T = ct_d × r₀ + r₁, (7) the server sends (ct_d, ct_T) to the client; (8) on the client, (ct_d, ct_T) is decrypted using the private key sk; (9) the decryption result is unpacked to generate the plaintext (d, T); (10) (d, T) is sent to the server; (11) on the server, the authentication result b is output and it is sent to the client by verifying whether d is equal to (T − r₀)/r₁; and (12) when the received data is equal to one, the authentication result is not tampered; otherwise, the result is modified.

3. Experiments

To verify the effectiveness of the face encryption scheme based on FHE in this paper, our scheme adopts browser/server mode. The front end mainly uses our improved FaceNet-Mobile deep learning model to extract users’ facial features and provide users with registration service. The server has rich processing resources and sufficient storage capacity and can calculate the distance of the face feature vector under ciphertext to provide homomorphic operation and authentication services.

3.1. Development Environment

This system uses Python Flash Web framework to implement B/S architecture, Intel Core i7-6700HQ processor, and the Python Tensorflow module to realize face detection, face alignment, and face feature vector extraction under deep learning. The homomorphic encryption algorithm library uses the SEAL library, which does not need external dependencies and is easily compiled under many different development tools. Presently, the encryption operations supported in SEAL library include negation, addition, accumulation, subtraction, multiplication, cumulative multiplication, power square, ciphertext plus plaintext, and ciphertext plus plaintext. The front end and back end of the web are realized by the Python web module and MySQL database. It mainly realizes the functions of user file upload, calling camera to take photos in real time, user ciphertext feature vector database management, face comparison result, ciphertext decryption display, and so on.

3.2. Computing Performance Analysis

In this scheme, FHE needs to encode the face features into integers before operation. Therefore, three different quantization schemes are designed for the coding of face eigenvalues, with coding accuracies of 0.1, 0.01, and 0.025. Two models—our FaceNet-Mobile and origin FaceNet [18]—are used for face matching tests on benchmark data sets (LFW [24], IJB-A [25], IJB-B [26], and CASIA [27]). The experimental result evaluation takes the unencrypted face feature matching performance as the benchmark. Table 1 provides a list of the correct acceptance rate when the false acceptance rates are 0.01%, 0.1%, and 1.0%. We can observe that when the coding accuracy is 0.0025, its accuracy can reach the level of unencrypted face features. At the same time, it can be seen from Table 1 that the accuracy of our model trained with softmax loss and center loss using the lightweight network as the basic network is slightly lower than that of the original network, but the complexity is greatly reduced.

Table 1 

Face recognition accuracy.

The complexity of the model is analyzed from the aspects of calculation amount and model size. Table 2 shows the experimental results. The model size refers to the size after saving the model as a PB file. According to Table 2, the model based on MobileNet proposed in this paper reduces the number of parameters by three times compared with the original FaceNet model and the improved model based on conception ResNet v1. Similarly, the model size is greatly reduced to meet the operation requirements of the mobile terminal. Therefore, while providing face template protection, preventing information leakage, and protecting user privacy, the matching based on homomorphic face can achieve the performance of matching with the original facial features. Finally, the experimental results also show that, even after using the dimension reduction method of classical PCA, its performance is the same as the original high-dimensional face features, but the matching efficiency of the homomorphic faces is improved.

Table 2 

Comparison of model complexity.

3.3. Parameter Optimization

Using the SEAL library for homomorphic encryption will produce some noise, and with the improvement of security level, the noise of ciphertext will also increase. If the total noise is greater than the threshold, the system cannot decrypt correctly. So, we must first ensure that the ciphertext can be decrypted successfully and then consider improving the security level of authentication.

For encrypting binary data with a length of 1024 bits, according to the homomorphic encryption principle, the transformed polynomial degree m must be greater than 1024. Yet if the value of m is too large, the calculation time of ciphertext will be very long. To solve this, this paper studies the partition of a 1024-bit binary vector. The minimum is given in Table 3 under the completion of ciphertext HD calculation at different intervals.

Table 3 

Comparison of (m, ) values under a different number of segments.

Table 3 shows the reduction of ciphertext module after segmentation is small; therefore, this method cannot remarkably improve the system’s efficiency and security. At the same time, Table 4 gives the maximum value of for n = 1024, 2048, 4096, 8192, 16384 in the case of 80-bit security level [22]. Based on the data in Table 3 and Table 4, the parameters m = 2048 and q = 2⁷⁶ − 2²² + 1 are selected. At this time, the noise growth of the ciphertext after completing the HD calculation does not exceed the upper limit of noise while the safety level is above 80 bits.

Table 4 

Comparison of (n, ) values under 80 bit security level.

3.4. Safety Analysis

The system mainly includes three possible attack sources: (1) front end, (2) communication channel, and (3) cloud server. Our B/S architecture ensures the higher security of the front end because the location of its facial feature template and private key cannot be fixed, which makes it more difficult for attackers to obtain these data; if the attacker wants to edit the front-end authentication results, the one-time MAC authentication used in this paper can well avoid this problem. In the communication channel, the attacker can only get the facial feature data based on FHE. So, network attackers could not utilize the intercepted data to decode the facial feature code before encryption. The database of the cloud server stores the user’s fully homomorphic encrypted feature template data and user label data. If the attacker cannot get the private key, the server is also secure.

4. Conclusion

Aiming at the problem that sensitive data are easy to be leaked in the face authentication system, this paper proposes a safe and efficient privacy protection face authentication scheme. The system combines homomorphic encryption technology with improved face recognition technology, which ensures the security and integrity of the user face feature template and keeps the accuracy of the feature comparison of ciphertext. From the performance, we can see that the fully homomorphic encryption does not have a great influence on the matching of face feature templates. After our optimization, the computation time of the ciphertext feature vector is greatly reduced. This efficiency can be used in practice, which provides a good guide for the practice of homomorphism. However, in the complex application scenario, further research and optimization are needed.

Data Availability

Readers can access our data on the findings by sending an email to the corresponding author.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work was supported in part by the Project of the Science and Plan for Zhejiang Province (Nos. LGF19F020008, LGF21F020022, and LGF21F020023), Ningbo Science and Technology Plan Project (Nos. 2021Z050, 2019C50008, 202003N4320, 202003N4324, 202003N4321, and 202003N4325), and the Humanities and Social Science Foundation of the Ministry of Education of China (Grant no. 17YJCZH178).