Abstract
Due to the long design cycle of watercolor cultural and creative products, a design method based on style transfer algorithm is proposed. Here, the design of Shanghai style watercolor painting cultural and creative products is taken as the research object, VGG-19 network is used as the style transfer model, and the loss function of style transfer algorithm is designed. The simulation results show that the proposed method can effectively extract and reconstruct the color, texture, brushstroke, and other styles of Shanghai style watercolor painting. The generated new images are added as design elements, which makes the cultural and creative products have both appearance and personalization. In addition, the newly generated cultural and creative products have also obtained high satisfaction from experts and users, which meet the aesthetic changes. It can be seen that proposed method provides a new idea for the design of watercolor cultural and creative products, which shortens the product design cycle, reduces the design cost, and has a certain practicality.
1. Related Work
Cultural and creative product design integrates cultural elements and creative thinking into products in a modern way. In recent years, the rich and colorful forms are more and more in line with modern people’s aesthetics and are gradually accepted, sought after and loved by the public. However, the design is a creative work, with the characteristics of a long design cycle, and the time cost is high for designers and users. Thus, relevant practitioners have carried out in-depth research. Liu Yuan proposed an improved generative adversarial network based on gradient penalty. The migration and reconstruction of image oil painting style are studied by constructing a total variance loss function, which could provide good edge and texture details for the migration process of image oil painting style. In addition, the generation rate and image quality of oil painting style images are improved [1]. Cai et al. proposed a method of image color style transfer. Dichotomy is utilized to extract color features of template images. The color style of the source image can be transferred to a new image, which can be used as a design element in product design to improve the speed of product design [2]. Bai et al. proposed a multiscale and multilayer feature fusion network, which uses a local binary pattern to generate three LBP-RGB feature maps. Furthermore, a neural network model is constructed to better express the rotation features of images, which can realize automatic directional detection and feature extraction of abstract paintings. By applying the extracted features to product design, the design speed is improved to a certain extent [3]. Wei proposed a painting image style feature extraction algorithm based on intelligent vision [4]. Here, similarity analysis, pixel smooth transfer, and semisupervised learning methods are adopted. On this basis, the similarity rule of painting image style is established, and all style features are quantified. Using intelligent vision technology to extract the style features of painting images can effectively reduce the average running time and improve the success rate of feature extraction. Thus, the design speed is improved. de Manincor et al. proposed to use the principal component analysis method to analyze the image sample data, and the data to be analyzed are made as a separate contribution, and thus, the analysis of carpaccio large-scale canvas paintings is realized, which lays a theoretical foundation for the product’s design [5]. Based on the style transfer algorithm, Guo et al. determined three main factors affecting human visual complexity perception, by collecting subjective complexity labels of painting images [6]. On the basis of psychology and art theory, there are 29 regional feature painting images representing the above three factors designed. The experimental results show that the proposed method can predict the visual complexity perception of paintings, and the prediction accuracy is 86.78%. It has a high correlation coefficient between subjective complexity and objective complexity, which is superior to other image complexity measurement methods. Wang et al. used the style transfer algorithm to propose a subgraph exchange method [7, 8]. The wall painting works selected by users are fused with the simulated environment image, and thus, the wall painting simulation renderings are generated. As can be seen, the design is highly innovative. Jeremiah and others put forward the use of the neural network to detect artistic style, which provides a reference for the neural network in the extraction of artistic style [9]. Kang and others proposed to improve the quality of artistic style pictures through texture transmission, which provides a set of image processing technology for the improvement of artistic style [10]. Gardini et al. applied a convolutional neural network to high-dimensional art image processing, thus realizing the transformation of music and visual art [11]. Yang and others applied deep learning to the classification of art images [12].
Based on the above research results, the style transfer algorithm has certain advantages in product design. Therefore, based on the style transfer algorithm and taking the Shanghai-style watercolor paintings transfer as the research object, a new design method is proposed.
2. Introduction to Style Transfer Algorithm
Style transfer algorithm means that the computer will specify content samples and style samples through operation to generate new samples with both content characteristics and artistic style characteristics [13]. The new sample has both the shape and outline of the content sample and the texture and color of the style sample. Gatys’ transfer algorithm is the most widely used style transfer algorithm at present. Moreover, it is divided into three parts: content reconstruction, style reconstruction, and style transfer.
2.1. Content Reconstruction
Assuming that there are content image and a trained convolutional neural network, the number of filters at each layer is Nl, and thus, multiple feature maps can be obtained at each layer. Vectorizing feature map can obtain Ml. Save Nl to , specify a layer of feature representation, and generate a new image such that the layer of feature representation Pl is equal to the original feature representation Fl. The loss function is defined as [14] follows:
Element is the corresponding activation of ith filter at position j at layer l.
2.2. Style Reconstruction
Gram matrix is adopted as the style image style, which is defined as follows [15]:
is the style representation. In order to make the style feature consistent with the original style feature, back propagation algorithm is used to optimize the white noise image. Assuming that the style image is , the image expected to be generated is , and the corresponding ram matrices of l layer are Al and Gl, and then, the loss function of this layer is [16]
The total loss function iswhere is the weight of each layer.
2.3. Style Transfer
To generate a new image integrating the style of image and the content of image , the gap between a layer content representation of and and the gap between the multiple layers style representation of and are minimized at the same time. The loss function is [17]
3. Design of Watercolor Cultural and Creative Products Based on Style Transfer Algorithm
3.1. Selection of Style Transfer Model
VGG network is a deep convolutional neural network with convolution operation, which is composed of a convolutional layer, pooling layer, full connection layer, and Softmax layer. It has various variations and is often used in object classification and other fields [18]. At present, the commonly used VGG model is VGG-19 (VGG-E) model. The structure of this model is shown in Figure 1. There is a 3∗3 convolution kernel used for convolution operation to extract more significant features of input images, and classification is achieved through activation function and Softmax layer [19].
The convolution operation formula of the VGG-19 model is as follows [20]:where is output result; is activation function; is grayscale matrix; is convolution kernel; is convolution operation; and is bias value.
The convolution operation is included in the activation function, as shown in formula (7) [21], to reduce the amount of computation of the network and avoid the over-fitting phenomenon.where x is output value of the convolution operation.
After entering the pooling layer, in order to reduce parameters, the maximum pooling with step size of 2 and box size of 2∗2 is adopted for pooling operation.where M and N are the length and width of two-dimensional vector of image, respectively.
The regression multiclassification data label of Softmax layer , and the training set is k-labeled samples.where is classification label, and is sample set. Assume that j is a different category and the probability value of j is estimated. For each sample, the probability of the kth label is [22]
Transform the regression sample into a k-dimensional probability vector, and then, there is
The learning parameters of model are
The sum of all probabilities in formula (11) is 1. For sample training, the loss function can be expressed in the following formula [23]:
In the back propagation of the VGG-19 network, the error function cannot be used in the hidden layer of the network, so the residual is adopted for propagation. The back propagation expressions arewhere is the output value of the node at the output layer , and is the residual of the node whose output is .
According to formulas (13)∼(15), the updated amount of weight and bias can be obtained as follows:where indicates that when , its value is 1. When , its value is 0. As can be seen from the formula, adjusting parameters can be realized by adjusting parameter .
VGG-19 network has a strong ability for feature presentation and detail description and can extract more features and details [24, 25]. Therefore, based on the VGG-19 network model, the Shanghai-style watercolor painting style is transferred.
3.2. Design of Style Transfer Loss Function
Gatys’ migration algorithm is adopted to minimize the objective function through iteration and transfer the style of content image S to style image I [26].where is the total loss function, is the content image loss function, and is the style image loss function. I is the i-layer model of depth model, and is the weight distinguishing influence of formulas (15) and (16) on the total loss function. F is the feature matrix, and G is the Gram matrix.
3.3. Scheme Design of Style Transfer
Based on the above analysis, the Shanghai-style watercolor painting style is adopted as a transfer style to design cultural and creative products. The specific scheme is as follows.
Firstly, the style transfer algorithm is used to extract and reconstruct style characteristics of Shanghai-style watercolor paintings, including brushstroke, color, and texture, to generate a new image example. In the design process, besides the decoration of the product, the practicality of products should meet the needs of users. Furthermore, the design should not be limited to a two-dimensional plane, so as to enrich the product style. The color collocation can choose the color close to the original Shanghai-style watercolor style or boldly choose a contrasting color. The materials can choose high-quality and inexpensive acrylic, paper, and other materials to reduce the selling price of products and improve the cost performance of products.
According to the above design scheme, the watercolor cultural and creative products are designed, and Figures 2 and 3 are the sketches of the product design.
4. Simulation Experiment
4.1. Construction of Experimental Environment
This experiment is conducted on tensorflow-gpu1.9.0, deep learning framework, and torch7 environment. It is run in Linux operating system. The GPU is GTX P102-100, which is a 4-core 24G E5-2660 V2, and the hard disk is 500 GB SSD. And Python, OpenCV, and other support libraries are installed.
4.2. Data Sources and Processing
The sample data of this experiment include style sample data and content sample data. Among them, style sample data are divided into realistic style data, freehand style data, and abstract style data, and the three kinds of data are, respectively, selected from the works of Shanghai-style cultural watercolor painters Pan Sitong, Ran Xi, and Ping Long [27]. The content samples are selected from Shanghai landmark Hudecs buildings, including Wu Tongwen Residence, Hudecs Private Residence, Joint Saving Society Bank, Park Hotel, and Wukang Building.
There are certain errors in the iterative process of the model resulting in problems such as noise or blurry images in the final style transfer image [28]. To solve this problem, Gaussian filtering, median filtering, and other methods are adopted for processing, and it finds that the image processed by Gaussian filtering is closer to the reality, and the details are richer. Therefore, Gauss filtering is used to process style transfer watercolor paintings. Figure 3 shows the calculation process of style migration.
Considering that the model has some errors during iteration, and the images obtained through style transfer contain noise or blurred image [24], this paper tries to optimize the number of iterations, content weight, and style weight, and filtering smoothing to solve these problems. The specific operations are as follows.
4.2.1. Number of Iterations
Due to the different difficulties of learning different styles of watercolor painting, adjusting the number of iterations properly to learn different levels of watercolor paintings to a varying degree, which is conducive to improving the transfer effect of watercolor painting styles. Table 1 shows the change process of the watercolor style transfer effect under different iterations. According to the figure, when the number of iterations is 1000, the watercolor style transfer effect is the best.
4.2.2. Content Weight and Style Weight
The content weight and style weight of watercolor have a great influence on the transfer effect, too high content weight will lead to full style transfer, and too high style weight will lead to the loss of details. Therefore, in order to achieve the ideal transfer effect, different content weights and style weights are tested, and the results are shown in Table 2. According to the figure, when the content weight is 5 and the style weight is 200, the migration effect is the best.
4.2.3. Filtering Smoothing
Filtering smoothing usually includes Gaussian filtering, median filtering, and other processing methods. In order to select the best filtering smoothing method, different filtering smoothing methods are used in this paper to process watercolor paintings, and the results are shown in Table 3. As can be seen from the figure, compared with mean filtering, median filtering, and bilateral filtering, the image processed by Gaussian filtering is closer to reality and has richer details. Therefore, a Gaussian filter is used to process the style transfer of watercolor paintings.
Figure 4 shows the comparison of results of the Wukang Building before and after Gaussian filtering after style transfer. In the picture, the style pictures from top to bottom are Pan Sitong style, Ran Xi style, and Ping Long style. As can be seen, compared with the image before processing, the image processed by Gaussian filtering has more clear stylized features, stronger color features, more natural brushstroke edges, and more realistic light and shade of the overall composition, which indicates that Gaussian filtering has obvious treatment effect and certain effectiveness. In addition, by comparing the transferred images of Pan Sitong, Ran Xi, and Ping Long, it can be seen that Pan Sitong's works are more realistic in color and composition, and the overall treatment of light and shade and details of virtual and real are more balanced and unified than those of Ran Xi and Ping Long. Therefore, the Pan Sitong style is finally transferred to the cultural and creative products.
4.3. Parameter Settings
Initial model training parameters are set as follows: epoch = 500, batch_size = 1, learning rate η = 1e0, style image weight ɑ = 1e5, and content loss weight β = 1e2. Through repeated experiments, the optimal training parameters of the model are determined as epoch = 1000, batch_size = 1, η = 1e0, ɑ = 1e200, and β = 1e5.
4.4. Experimental Results
4.4.1. Experimental Results of Style Transfer
(1) Style Transfer Results. To verify the transfer effect of the proposed method, Pan Sitong style transfer is applied to Residence, Building successively. The results are shown in Figure 5. As can be seen, the proposed method realizes the transfer and lays a foundation for the secondary creation of Shanghai-style watercolor paintings.
(2) Evaluation of Style Transfer Effect. In this experiment, user satisfaction is selected to evaluate the effect of the proposed style transfer method. Three art-related practitioners, three design professionals, and four ordinary people are selected to score the generated style transfer effect images on an integer scale of 1∼5 and calculate the average of final evaluation scores. The evaluation results are shown in Table 4. The proposed style migration algorithm based on VGG-19 can generate relatively satisfactory transfer images for users, and there are 4 samples with satisfaction scores higher than 3, accounting for 80%. This shows that the proposed style transfer algorithm can extract and reconstruct the color, texture, brushstroke, and other styles of Shanghai-style watercolor paintings and generate new images with high user satisfaction.
4.4.2. Design Results of Cultural and Creative Products
(1) Design Effect Display. Figure 6 shows the design effect diagram of products. The style transfer algorithm can quickly generate design materials, reduce design time, improve design efficiency, and reduce the design cost. At the same time, cultural and creative products can be customized for different users, which meets the aesthetic trend of the era.
(2) Design Evaluation. There are 10 professionals, including art museum experts, students majoring in design, and painters, invited to use a scale of 1 to 5 to score the products from appearance [29], cultural connotation, practicality, and personalization, so as to realize the evaluation of designed watercolor cultural and creative products. The results are as shown in Table 5. Here, content images can reflect the characteristics of Shanghai. Moreover, the newly generated images are added to cultural and creative products as design elements, which can make products have both cultural connotation and personalization. Therefore, the products have obtained a high expert satisfaction score, and the overall satisfaction is more than 3.4 points.
To evaluate the market demand for designed watercolor cultural and creative products, a questionnaire survey is conducted on random passers-by, and 100 questionnaires are sent and collected, among which 61 questionnaires are valid. In the valid questionnaire, there are 31 male and 30 female users. Among them, 37 users are aged between 18 and 25, and 24 users are aged between 26 and 55. The scores of users on the designed cultural and creative products are counted, and the average scores are calculated, which are shown in Figure 7. Here, users have high overall satisfaction with the designed postcards, with an average score close to 4, indicating that the designed watercolor cultural products are popular with the public and can simultaneously meet users’ needs for cultural connotation, appearance, personalization, and practicality.
5. Conclusion
To sum up, the proposed watercolor product design method based on the style transfer algorithm adopts the VGG-19 network as the basic model to migrate the Shanghai-style watercolor painting style. In addition, the loss function of the style transfer algorithm is designed. Thus, the rapid extraction and reconstruction of the color, texture, brushstroke, and other styles are achieved. Moreover, the design cost is reduced, and the design process is shortened. Adding newly generated images as design elements to cultural and creative products can make products meet the needs of appearance, personalization, cultural connotation, and practicality. There is no doubt that these products can obtain high expert and user satisfaction and meet the aesthetic changes in the market, which is conducive to the spread of watercolor painting. The innovation of this study is to apply the deep learning algorithm to the specific artistic style extraction and apply the extracted style to the actual design, so as to provide a more practical case for the information design of artistic style.
Although certain results have been achieved, due to the limitations of conditions, there are still some deficiencies to be improved and perfected. On the one hand, the proposed style transfer algorithm has a good style transfer effect on Pan Sitong's realistic style watercolor paintings, but the transfer effect of freehand style and abstract style still needs to be improved. On the other hand, the proposed method has been verified using Hudec architecture as an example, but the themes such as landscapes and people have not been selected in the content sample. Therefore, in future research, improvements should be considered from the above deficiencies to optimize the algorithm and improve the generalization of the algorithm.
Data Availability
The experimental data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest regarding this work.