Conditional generative adversarial network based data augmentation for fault diagnosis of diesel engines applied with infrared thermography and deep convolutional neural network

▪ Utilize infrared thermography and deep convolutional neural network (DCNN) for fault diagnosis of diesel engines. ▪ Conditional generative adversarial network is deployed for data augmentation of the diesel engine infrared images. ▪ DCNN-based fault diagnosis method has better classification effect and algorithm stability compared with stacked auto-encoder, long short-term memory network and multi-layer perceptron. This paper tries to introduce a new intelligent method for the early fault diagnosis of diesel engines. Firstly, infrared thermography (IRT) is introduced into diesel engine condition monitoring, then infrared images of diesel engines in four health states, such as normal condition, single-cylinder misfire, multi-cylinder misfire and air filter blockage, are collected and the region of interest (ROI) of infrared images are extracted. Next, conditional generative adversarial network (CGAN) is deployed to perform data augmentation on infrared image datasets. Then, deep convolutional neural network (DCNN) and Softmax regression (SR) classifier are used for automatically extracting infrared image fault features and pattern recognition, respectively. Finally, a comparison with three deep learning (DL) models is performed. The validation results show that the data augmentation method proposed in the paper can significantly improve the early fault diagnosis accuracy, and DCNN has the best fault diagnosis effect and resistance to temperature fluctuation interference among the four DL models.


Introduction
Diesel engine is the core power unit of many production systems, including construction machinery, agricultural vehicles, electric generators, ships, etc.Its working condition determines the reliability and overall efficiency of the system [12].Due to the complex working environment and certain inherent manufacturing process defects, diesel engines are prone to various early faults during high temperature and pressure, high speed and heavy load operation, which can seriously affect productivity or even cause major safety accidents [14].Therefore, to maintain the safety and stable operation of diesel engines, it is crucial to implement effective and reliable monitoring and diagnosis of diesel engine early faults and to take necessary maintenance measures.torque signals [53].Among the above signal analysis methods, vibration signals are easy to measure and contain important dynamic information of related mechanical components, such as the burst combustion of gases in the engine cylinder, the impact of valves during opening and seating, the reciprocating inertial impact of pistons, and crankshaft rotation, etc. [54,5], which have been widely used in fault diagnosis studies of diesel engines [2,8,43].However, vibration signal detection and acquisition has some prominent limitations, including noise pollution, local contact measurement, and high computing costs, etc.Furthermore, the acquisition environment of diesel engine vibration signals is usually a high temperature and violently vibrating cylinder surface, which can easily lead to distorted or even damaged sensor measurements [51].Besides, in many industrial production activities, vibration signal acceleration sensors are not allowed to be deployed [55].Due to the above reasons, the development of alternative sensor technologies for monitoring the health state of diesel engines has become a hot research topic.
In recent years, using contact sensors such as thermistors and thermocouples to collect temperature signals to carry out condition monitoring of mechanical equipment has attracted wide attention of researchers [26].Common fault patterns of diesel engines such as misfire, air filter blockage, and insufficient oil supply are usually accompanied by significant temperature changes on the cylinder surface.However, contact temperature measurement usually only collects temperature changes at a few points on the equipment surface, which cannot fully characterize the real-time status of diesel engines.On this basis, infrared thermography (IRT) has gradually become a new non-contact nondestructive testing technique for measuring the surface temperature of equipment to overcome the negative effects of complex transmission paths and local detection caused by the contact temperature measurement.Compared with traditional methods based on vibration signal analysis, IRT-based fault diagnosis methods have the following outstanding advantages: 1) As the core device of the IRT system, infrared thermal camera can monitor the surface temperature of the equipment under test without contacting it and visualize it through infrared images; 2) IRT can detect the equipment remotely and in a wide range, which is suitable for health state monitoring of large mechanical equipment.3) Unlike the vibration signal analysis method, IRT-based method is not affected by the vibration frequency of mechanical equipment.
In summary, infrared images reflect the temperature changes of the equipment and the surrounding environment, and contain rich information about the equipment state, which can be well applied to the subsequent fault diagnosis studies.Other noncontact detection techniques, such as sound signal detection, have some shortcomings such as low signal-to-noise ratio and complex noise pollution, and traditional signal processing methods are difficult to separate and extract the fault signals.
Therefore, IRT-based fault diagnosis methods for mechanical equipment have been widely studied and applied by scholars at home and abroad [42,16,39].However, the existing IRT-based fault diagnosis methods have the following main challenges in the practical industrial applications of mechanical equipment: (1) Currently, IRT-based fault diagnosis methods of mechanical equipment usually require manual extraction of infrared image feature parameters, and the process of feature selection requires a large amount of a priori knowledge, but in actual industrial applications, such a priori knowledge is not always available in time, which largely affects the accuracy of fault diagnosis; (2) During the operation of mechanical equipment, the surface temperature usually changes according to the rule of first increasing and then remaining stable, but the fault may occur at any time after the power is turned on.The current IRTbased methods mostly collect infrared images of the equipment in the stable state and carry out fault diagnosis afterwards, without considering the influence of temperature changes on the fault diagnosis results at the early stage of equipment operation; (3) To date, IRT has been widely used for fault pattern identification of rotating machinery such as rolling bearings, rotor systems, and shafts in steady state, but there is a lack of diagnostic research for early faults of mechanical equipment, especially a lack of research on early fault identification of reciprocating machinery such as diesel engines based on IRT.
For early faults of diesel engines, especially minor faults, there is often a lack of supervision signals with obvious labels and sufficient training samples.In this case, the method of unsupervised learning [32,46,50]

Related literature review
IRT-based fault diagnosis of mechanical equipment mainly includes three steps: infrared image acquisition, feature parameter extraction and fault pattern recognition.Among them, feature parameter extraction is the most critical step among the above steps [24], which directly affects the accuracy of subsequent pattern recognition.In recent years, deep learning (DL) has received key attention from all walks of life for its excellent feature extraction ability, and has been widely applied in speech recognition, image processing, machine vision and other fields [21,33,28,35].Compared with the traditional method of manually extracting feature parameters, DL can directly process a large amount of raw data, dig deep into the intrinsic laws of data, and carry out adaptive learning of mapping relationships between data without artificially establishing complex mathematical models, which effectively reduces the information loss brought by manual processing.In the research of IRT-based fault diagnosis of diesel engines, some scholars have started their preliminary exploration.Baranitharan et al. [1] proposed an intelligent life monitoring method for diesel engines based on infrared image processing technique, which quantifies the infrared images of engine operation using image processing techniques to assess the engine life in terms of pixel values.Li et al. [36] designed a test rig for monitoring diesel engine exhaust faults using IRT to measure the radiated temperature of the outer surface of the exhaust pipe under different operating conditions, and analyzed the relationship between diesel engine exhaust temperature and faults according to the principle of infrared thermometry.Using ultrasonic IRT, Su et al. [37] carried out ultrasonic IRT principle and series of experimental studies for the limitations of traditional NDT technology in detecting crack defects in aeroengine blades, and built an ultrasonic IRT detection test platform to achieve the detection of fine cracks in aero-engine guide blades and working blades.
Although some achievements have been made in the research of diesel engine fault diagnosis, there are still many deficiencies in the application of traditional ML, including the cumbersome feature engineering, the need for manual intervention, the inability to automatically learn, and the limited ability to process large-scale nonlinear data.CNN has been proven to be an effective method in the field of feature extraction [29,10].Compared with traditional ML methods, CNN has two significant advantages: 1) It can automatically extract deep level features of data and has good generalization ability; 2) By using local connections and weight sharing, the network parameters are easy to train and optimize [47].DCNN has a more complex network structure and better feature extraction ability compared to general CNN models.Therefore, in this study, DCNN was used to extract infrared image feature parameters of diesel engines under different fault patterns.
From the above literature, it can be seen that the methods based on IRT and DL are widely used in the field of rotating machinery fault diagnosis, however, the acquisition process of infrared images is usually carried out in the stable state after the equipment has been running for a period of time, and the research of early fault diagnosis for rotating machinery under

CGAN-based data augmentation
In the application of DL methods, only limited data can be acquired for model training in many cases, so data augmentation operation is an indispensable part.Data augmentation methods mainly include two categories of supervised data augmentation and unsupervised data augmentation methods, which are classified as shown in Figure 1.
In recent years, with the continuous development of DL techniques, unsupervised data augmentation methods have received more and more attention.For example, generative adversarial network (GAN) [11] has been widely used in many fields such as image processing, speech processing, computer vision (CV), and natural language processing (NLP), including image generation [40], style migration [15], super-resolution [6], and image restoration [7].As an unsupervised generative DL model, GAN adopts the idea of confrontation optimization in its structural design.Among the many functions of GAN, image generation is the most widely used, aiming to generate the desired images using the generator.CGAN has made some improvements on the basis of GAN.
For the generator of the original GAN, the generated images are random and unpredictable, so the output of the network cannot be controlled, and the actual operability is not strong.In response to the issue of the inability of the original GAN to generate images with specific attributes mentioned above, Mehdi Mirza et al. [30] proposed CGAN in 2014, which generates images that meet specific conditions by adding constraints to the generator and discriminator in the original GAN.The method flow of CGAN is shown in Figure 3.As can be seen in Figure 4, the constraint c is introduced as additional input information to CGAN, which is combined with the random noise t as the input to the generator G; while in the discriminator D, the constraint c and the output of the generator are combined with the original data x as the input to the discriminator.This improvement has been shown to be very effective in many studies [31,3,45].

The loss function of CGAN
During the iterative training of GAN and its modified forms, the adversarial optimization between the generator and discriminator can be viewed as a maximal and minimal game, where the generator tries to maximize the error rate of the discriminator and the discriminator tries to minimize its error rate.This competition drives the training of the model and motivates the generator to generate more realistic data and the discriminator to become more accurate.The loss function of a classical GAN is as follows [11]: where, x denotes the original data, t denotes the random noise input to the generator, G(t) is the data generated by the generator, D(x) is the probability that the discriminator judges the original data to be true, D(G(t)) denotes the probability that the discriminator judges the data generated by the generator to be true, and E represents the mathematical expectation under the specified data distribution of the solution.In summary, in the actual training process, GAN achieves the optimization of two models by alternating iterations.First, the generator is kept constant and the parameters of the discriminator are optimized so that it can better distinguish the real image from the generated image.Then, the discriminator is fixed and parameters of the generator are optimized so that it can generate more realistic images to deceive the discriminator.This process continues until a predetermined number of training sessions is reached or the quality of the generated images meets the requirements.
For the discriminator, where x is the original data, i.e., the real data, the optimization objective of its training process is to maximize the loss function V(G,D), in other words, it is necessary to make D(x) as large as possible while D(G(t)) is as small as possible, and the process of optimizing the discriminator can be expressed as follows: For the generator, the optimization objective of its training process is to minimize the loss function V(G,D), and the process of optimizing the generator with optimal discriminator parameters can be expressed as follows: Therefore, the objective function of GAN can be represented by the following equation: Except for the introduction of relevant conditional constraints, the implementation process and network structure of CGAN and GAN are basically the same.In the training process of CGAN, the constraints c are added to the inputs of both the generator and the discriminator, and then the loss function and objective function of CGAN are shown in Eq. ( 5) and Eq. ( 6), respectively [49].
is the convolution kernel of the n-th layer, the symbol "*" represents the convolution operation, and    is the bias vector.

Activation layer
Convolution operation is a linear operation, no matter how many convolutional layers are superimposed, the output is a linear combination of the inputs, when the network has limited ability to express features and cannot handle complex nonlinear problems.Therefore, a nonlinear activation function needs to be added after the convolutional layers in order to give the network

Fully connected layer
The fully connected layer is usually the last layer of DCNN, and its role is to flatten the feature maps (usually two-dimensional matrices) output from the convolutional and pooling layers into one-dimensional vectors, and to perform product and addition operations with a set of learnable weight matrices to obtain the final classification results by the SR classifier [22], as shown in       Step 2. Before starting the experiment, the infrared thermal camera parameters were set, and at this time the diesel engine and room ambient temperatures were consistent, both at 19.2°C.
Step 3. The diesel engine speed was set to a constant 500 rpm and one infrared image was acquired every 10s.
Step 4. The experiment time was set to 25 minutes, and the maximal surface temperature of the cylinder had stabilized to 51.5°C by the end of the experiment.150 infrared images were finally acquired for each fault type.
Step 5. Cool the diesel engine to room temperature sufficiently, keep the parameters such as image acquisition equipment, distance, angle, ambient temperature and humidity unchanged, and repeat Step 1 through Step 4 for the other three fault types until data collection is complete.
Note: In the experiment, air conditioners and humidifiers are used to adjust the indoor temperature and humidity.
It should be noted that the reason why an infrared image is collected every 10s is because the maximal acquisition frequency of the infrared thermal camera used in the experiment is every 3s to collect an infrared image.Subject to the hardware limitations, the infrared thermal camera manufacturer suggested that the best acquisition frequency should be set to collect an infrared image every 10s, to ensure that as much experimental data as possible can be collected in a limited period of time, and at the same time, the imaging quality of the infrared thermal camera can be guaranteed.On the other hand, an important topic of this paper is the early fault diagnosis of diesel engines.In the actual operation process, the faults of diesel engines may occur at any moment during the operation process.Therefore, at the The infrared image of the diesel engine under NC after 20 minutes of stable operation is selected and shown in Figure 13.In Figure 13, the temperature unit is °C.The observation shows that the infrared image expresses the temperature distribution on the surface of the diesel engine in a visualized form, however, it also contains many disturbing factors that affect the analysis.For example, in the whole infrared image, the areas with higher temperature and obvious brightness are mainly the cylinder block area, the top turbocharger area, the bottom battery area and individual scattered locations where the brightness and temperature are the highest below the turbocharger.
Further analysis shows that diesel engine infrared images have the following unique characteristics compared to other types of infrared images or visible light pictures: (1) Temperature distribution characteristics: infrared images of the diesel engine reflect the temperature distribution inside the engine.By observing the infrared images, the hot and cold spots inside the engine can be clearly seen.These hot and cold spots reflect the heat flow and heat dissipation inside the engine.
(2) Visualization of the combustion process: infrared images can visualize the process of diesel combustion in the combustion chamber.When diesel fuel burns, it releases a large amount of heat, which can be visualized by infrared images.By observing the infrared image, we can clearly perceive the shape and size of the combustion chamber, as well as the distribution of diesel fuel in the combustion chamber.(2) The temperature rise caused by AFB may be uneven.In an infrared image, it can be observed that the regional temperature distribution is uneven, and the size or shape of hot spots are inconsistent.Large and obvious hot spots may be formed in areas with serious blockage.Analysis of Figure 15 shows that CGAN learns how to generate images from random noise corresponding to the given conditions by means of adversarial training, so that the generated images have the same distribution as the original images, achieving a good image generation effect and thus data augmentation.

Experimental dataset description
The diesel engine operation process is usually accompanied by significant temperature changes.In this experiment, the average temperature of the cylinder surface continued to rise from 19.2℃ at the initial operation to 51.5℃ and then remained basically stable.In this case, an infrared image was acquired every 10s using the infrared thermal camera for 25 minutes, and finally 150 infrared images were acquired for each fault type.In this study, four effective DL models such as DCNN, stacked auto-encoder (SAE), recurrent neural network (RNN), and deep neural network (DNN) will be applied for infrared image feature extraction and combined with SR classifier for fault pattern recognition.
To investigate the early fault diagnosis performance of each model, this paper uses two datasets for the validation of related models, including dataset 1 and dataset 2. Specifically, based on the ROI extraction of infrared images, the acquisition time of each fault type is divided into 5 parts equally, and there are 30 samples in each time zone, then dataset 1 is a sample grid consisting of 5 time zones, including zone 1 (0-5min), zone 2 (5-10min), zone 3 (10-15min), zone 4 (15-20min), and zone 5 (20-25min), each zone is 5 minutes long and contains 120 samples (4 fault types × 30 samples).Dataset 2, on the other hand, is based on the original data and is augmented with data using CGAN.The specific operation is as follows: the infrared images in a single time zone under each fault type are used as the original images, and constraints such as image size and number are added to generate 90 infrared images, i.e., the sample size in a single time zone under each fault type is expanded from 30 to 120, and the total sample size of dataset 2 eventually reaches 4 times that of dataset 1.For example, the shown in Figure 16.
It is uniformly stated that the concepts such as early fault diagnosis and early operation stage of the diesel engine mentioned in the following of this paper are for time zone 1, i.e., the first 5 minutes of diesel engine operation.As shown in Figure 16, the differences between the infrared images of different fault types are very small under the effect of heat conduction of mechanical equipment, and it is difficult to distinguish them visually, so artificial intelligence (AI) techniques need to be introduced to assist in decision making.
In this paper, the training and test sets are randomly divided according to the ratio of 8 to 2 for evaluating the performance of the model and adjusting the hyperparameters of the model.

Specific descriptions of individual temporal zones in dataset 1
and dataset 2 are given in Table 2 and Table 3 It is worth noting that in the paper, due to the limitation of time and experimental equipment, experimental data was collected for each fault mode only once.However, to ensure that this single experiment's results are not influenced by other factors leading to potential errors, it may be more meaningful to conduct multiple sets of experiments for each fault mode, and this aspect will be specifically developed in the future research.

Parameter setting of the involved DL models
In

Results and discussion
In the practical application of diesel engines, functional faults may occur at various stages of the operation process.Therefore, to prevent possible early faults and perform timely fault   As seen in Table 6 and Figure 17, the average diagnostic accuracy of DCNN is the highest in all five independent time zones of dataset 1, and the overall average diagnostic accuracy is also the highest among the four models, reaching 92.42%.
The other three models performed differently within each time   7 and Figure 19, respectively.As can be seen from Table 7 and Figure 19    Furthermore, as can be seen from Table 2 and Table 3, zone 1 after data augmentation has 480 infrared images, while the whole time zone before data augmentation contains 600 infrared images.By comparing the accuracy of dataset 1 in Table 8 and dataset 2 in Table 9, it can be concluded that the accuracy of the Therefore, the proposed IRT-DCNN based fault diagnosis method for diesel engines using CGAN based data augmentation has strong theoretical and practical value.

Effect evaluation of data augmentation
Before extracting infrared image features, this paper uses a data augmentation method based on CGAN to generate infrared images under certain constraints, to achieve the effect of data expansion.From Figure 14, it can be seen that the infrared image generated by CGAN is very close to the original image.10 and Table 11, respectively.12.
As can be seen from To solve this problem, various sensor signals are applied for fault diagnosis of diesel engines, mainly including vibration signals, sound signals, oil signals, pressure signals and speed- can play an important role.Unsupervised learning can learn useful structures and patterns from input data without labels or supervision signals.Through the data augmentation technology based on unsupervised learning, simulated synthetic data with early fault characteristics can be generated, which can be used to train and optimize the diagnosis model and improve the early fault detection ability of the model.Therefore, to make up for the above research deficiencies, a new unsupervised data augmentation based fault diagnosis method is proposed for the problems of inconspicuous infrared image features and insufficient training data at the early stage of diesel engine operation, which can achieve accurate and efficient diagnosis of early faults of diesel engines with the unique advantages of IRT.Firstly, infrared images of diesel engines in different health states are obtained by using infrared thermal camera detection, followed by data augmentation of the experimental dataset with conditional generative adversarial network (CGAN), then feature parameters of infrared images are automatically extracted based on deep convolutional neural network (DCNN), and finally Softmax regression (SR) classifier is deployed for fault pattern recognition of diesel engines, while the experimental data collected from a highpressure common rail diesel engine is used to verify the effectiveness of the proposed method.The remainder of the paper is organized as follows: Section 2 reviews and analyzes the related literature and the current status of research.In Sections 3 and 4, the basic principles and methodological procedures of CGAN and DCNN are introduced, respectively.Section 5 details the basic components of the fault diagnosis system of diesel engines and information about the infrared image acquisition experiment, and analyzes the CGAN-based data augmentation process.In Section 5, the effectiveness and superiority of the proposed method are verified using the experimentally obtained datasets.Finally, Section 7 summarizes the innovative work of the paper.

Furthermore, in the
context of big data, DL can effectively improve the intelligence and adaptability of fault diagnosis.Based on the excellent feature extraction ability of DL, many researchers are working on the application of DL methods in the field of fault diagnosis of mechanical equipment[13,38,17].Verstraete et al.[41] addressed the shortcomings of traditional manual feature extraction methods, such as complicated operation and the need for more expert knowledge of the relevant system, by converting the original data into timefrequency images and then inputting them into DCNN for adaptive feature extraction and fault diagnosis, which can overcome the uncertainty brought by insufficient expert knowledge to the diagnosis results.Jia et al. [18] introduced IRT into fault diagnosis of bearings to solve the problems of rotating machinery damage and noise pollution caused by sensor installation in the process of vibration signal acquisition, and the feature extraction and fault diagnosis effects of bag of visual words (BoVW) and convolutional neural network (CNN) were compared and analyzed.Liu et al. [27] proposed a fault diagnosis method of the rolling bearing based on sound signal analysis, which uses short-time Fourier transform (STFT) to convert the acoustic signal into a spectrogram and input it into a stacked sparse self-encoder (SSAE) network for adaptive extraction of fault features, and finally SR classifier is deployed for fault pattern recognition.Xiao et al. [48] proposed a fault diagnosis model of the insulator string based on IRT and probabilistic neural network (PNN) for the close relationship between insulator string state and its surface temperature distribution, and the automatic identification of fault patterns can be achieved by using the SR classifier.Xu et al. [49] addressed the limitations of existing methods in feature learning, converted the one-dimensional vibration signal into a twodimensional grayscale image based on continuous wavelet Eksploatacja i Niezawodność -Maintenance and Reliability Vol. 26, No. 1, 2024 transform (CWT), and used DCNN and random forest (RF) integrated learning method for fault diagnosis research of bearings, and the experimental results demonstrated that the method has a high accuracy.The combination of IRT and DL has also seen some research progress.Choudhary [4] et al. collected infrared images of rolling bearings in six different states, and used artificial neural networks (ANN) and LeNet-5 for feature extraction and pattern recognition, and demonstrated the superiority of CNN by comparing and analyzing the classification results of the two methods.Li et al. [26, 25] achieved good results in an in-depth study of failure patterns in industrial gearboxes and rotor systems, using IRT and CNN for infrared image acquisition and pattern recognition of common faults.Wang [44] et al. proposed a crack recognition method based on IRT with CNN for image recognition and classification to address the problems of low efficiency and poor interference resistance in steel plate crack detection by traditional non-destructive testing (NDT) techniques, and achieved high recognition accuracy on the test set.In summary, we can see that DL methods have better fault diagnosis performance compared to traditional methods that rely on manually designed features.Currently, the common methods and basic processes for fault diagnosis of diesel engines are mainly signal acquisition, manual feature extraction, and pattern recognition based on machine learning (ML).Under this research theme, Ramteke et al. [34] used fast Fourier transform (FFT) and STFT to extract feature parameters from the collected vibration and sound signals of diesel engines in different wear states and ANN is applied for failure pattern identification.Flett et al. [9] proposed a fault diagnosis method for valve train of the diesel engine based on a Naive Bayes Classifier, which can achieve accurate identification of fault patterns such as valve spring deformation and abnormal clearance by improving the root mean square (RMS).Kowalski et al. [19] proposed a new fault diagnosis method based on the extreme learning machine (ELM) for 15 different health states of impulse program diesel engines.
variable temperature still needs to be explored; for reciprocating machinery such as diesel engines, the research of fault diagnosis based on IRT and DL is in the initial stage, and the early fault diagnosis research of diesel engines under variable temperature is still a blank.To fill the above research gaps, this paper simulates several common fault patterns and collects infrared images under different fault states using a diesel engine experiment bench for the variable temperature process from start-up operation to temperature stabilization, and then CGAN and DCNN are applied for data augmentation and infrared image fault feature extraction for early fault diagnosis of the diesel engine.The infrared image dataset is divided into several sub-datasets according to the same time interval and used to verify the effect of temperature variation on the accuracy and stability of the algorithm.

3. 1 .
The network structure of CGAN GAN consists of two neural network models, a generator G and a discriminator D. The role of the generator is to generate virtual data similar to the real data by inputting random noise, while the role of the discriminator is to determine whether the input data is real or virtual.The iterative training of GAN is the process of continuous adversarial optimization of the generator and the discriminator, and the performance of both the generator and the discriminator can be effectively improved so that the virtual data generated by the generator cannot be distinguished by the discriminator, thus achieving the purpose of image generation.The training of GAN can be stopped when the quality of the output samples of the generator no longer improves, i.e., the discriminator is unable to determine the authenticity of the data generated by the generator.The method flow of GAN is shown in Figure2[23].

Fig. 3 .
Fig. 3.The method flow of CGAN.CGAN is a generative model that learns how to generate images from random noise corresponding to given conditions by means of adversarial training, so that the generated images have the same distribution as the original images, thus achieving the task of image generation.
nonlinear description capability.The activation layer is usually located after the convolutional layer of the DCNN and is used to perform nonlinear transformations on the input data to better extract features and enhance the expressiveness of the model.The choice of the activation function has a great influence on the performance of the neural network.The common activation functions are: sigmoid function, tanh function, ReLU function, etc.The curves of each function and its expressions are shown in Figure 6, where the solid line is the activation function and the dashed line is its derivative.

Fig. 6 .
Fig. 6.Common activation functions.Among the above three activation functions, ReLU function is the most widely used in CNN.Compared with the first two, ReLU function not only has faster convergence speed, but also has certain sparse expression characteristics of the output.Using

Fig. 9 .Fig. 10 . 5 . 1 .
Fig. 9. Fault diagnosis system of the diesel engine.It should be noted that the IRT-CGAN-DCNN based fault diagnosis method of the diesel engine does not require prior knowledge of equipment fault laws, parameter setting methods and other related priori knowledges when applied.It can directly collect infrared images of the diesel engine in different health states, which will be input to DCNN for fault pattern

Fig. 11 .
Fig. 11.(a) Enlarged views of the infrared thermal camera and control panel; (b) IRT-based diesel engine condition monitoring system; (c) Diesel engine.As shown in Figure11, the start and shutdown of the diesel engine is mainly controlled by the switch on the control panel, and the real-time status information such as speed, intake pressure, oil quantity and water temperature can also be read normal condition (NC), single-cylinder misfire (SCM), multicylinder misfire (MCM), and air filter blockage (AFB), were introduced into the diesel engine.Note that SCM and MCM are simulated by disconnecting the cylinder ignition power line, and an outer cover is added to the air filter to simulate AFB.The above fault patterns of the diesel engine are shown in Figure12.

Fig. 12 . 1 .Algorithm 1 .
Fig. 12.(a) NC；(b) SCM；(c) MCM；(d) AFB.In Figure 12(a), the six cylinders of the engine are labeled as ①-⑥, and the specific manifestations of SCM and MCM are misfire of cylinder 1, cylinder 3 and cylinder 4 at the same time, respectively.The engine speed was fixed at 500 rpm and the

5 . 2
early stage of diesel engine operation, the lack of training data due to the short start-up time exists objectively, and the significance of early fault diagnosis research lies in detecting whether the diesel engine has a fault state in the shortest possible time.Eksploatacja i Niezawodność -Maintenance and Reliability Vol. 26, No. 1, 2024 Process of infrared image data augmentation Before performing infrared image data augmentation, the ROI of infrared images are first extracted to narrow down the processing of image data to focus on the cylinder area with obvious temperature changes to reduce the interference of redundant information in the images.Extracting image ROI is a common task in the field of image processing and CV, which can effectively reduce the computational effort of image processing, better image feature extraction, and improve the efficiency and speed of image processing.In the field of CV, extracting ROI has a wide range of applications in tasks such as image classification, target detection, and target tracking [52].

Fig. 13 .
Fig. 13.The infrared image of the diesel engine under NC after 20 minutes of steady operation.

( 3 )
Fault diagnosis: infrared images can be used for fault detection and diagnosis.For example, by observing infrared images, problems such as engine misfire, oil leakage, gas leakage and blockage can be found.At the same time, the performance changes and aging of the engine can be assessed by comparing the differences in the infrared images at different points in time.(4) Condition monitoring: infrared images can be used for real-time monitoring of the engine operating condition.For example, the workload, operating efficiency and emissions of the the engine can be understood through observing the infrared images, which is important for evaluating the performance of the engine and maintaining the normal operation.Moreover, different faults also have corresponding characteristics in infrared images of the diesel engine.For example, (1) The cylinder temperature will have a certain temperature increase under normal operation conditions, but when a misfire occurs in a certain cylinder of the engine, the cylinder temperature will obviously rise beyond the normal working range due to incomplete combustion or inaccurate ignition timing, which makes the infrared image locally bright.

Fig. 14 .
Fig. 14.The Infrared image ROI of the diesel engine under NC After the ROI extraction operation, the infrared image size is reduced from 324 kb to 52.7 kb, indicating that the ROI extraction can effectively reduce the amount of data to be processed, thus improving the speed and efficiency of subsequent fault pattern identification.In the next step, the extracted ROI is input to the CGAN model for data augmentation, as shown in Figure 15.

Fig. 15 .
Fig. 15.(a) The original infrared image ROI under NC; (b) Examples of the infrared image ROI under NC generated by CGAN .
the paper, four DL models such as DCNN, SAE, RNN, and DNN are used for infrared image feature extraction, and finally, SR classifier is deployed for fault pattern recognition.Therefore, for DCNN, this paper uses the AlexNet model for feature parameter extraction.AlexNet model has a complex network structure consisting of multiple convolutional layers (CL), pooling layers (PL) and fully connected layers (FCL), and also uses ReLU activation function, dropout regularization and other techniques, which has the advantages of high accuracy and scalability.The emergence of AlexNet marks the rise of DCNN, which has led to significant progress in research in the field of CV.The detailed structure of the AlexNet model is shown in diagnosis and troubleshooting, this case divides the working process of the diesel engine according to time, aiming to verify the effectiveness of the data augmentation method proposed in this paper for early fault diagnosis of diesel engines.At the same time, the fault diagnosis results are compared with the DL models such as SAE, LSTM, and MLP to prove the superiority of the DCNN-based fault diagnosis method proposed in the paper.First, based on dataset 1, the above four DL models were used for fault pattern recognition, and each model was trained and tested 10 times for different time zones, and the specific classification results are shown in
zone, among which, MLP ends up with the second highest average diagnostic accuracy after DCNN with 89.83%, while SAE and LSTM have lower average diagnostic accuracies of 85.54% and 89.58%, respectively.In terms of early fault diagnosis performance, the Eksploatacja i Niezawodność -Maintenance and Reliability Vol. 26, No. 1, 2024 classification accuracy of the four DL models for different fault patterns in the early operation stage of the diesel engine is mainly analyzed.First, it can be seen that the early fault diagnosis accuracy of all four DL models is low.Obviously, the early fault diagnosis accuracy of DCNN is 65.42%, which is still the highest among the four models, and LSTM has the lowest accuracy, which is only 57.08%.In terms of algorithm stability, the proposed DCNN method also has better performance, with the lowest standard deviation of 0.0246 among the four models in terms of diagnostic accuracy within a single time zone and during the whole operation process, while SAE has the highest standard deviation of 0.0780.To show the performance of DCNN, SAE, LSTM, MLP in early fault diagnosis of the diesel engine, the confusion matrix of classification results of the above four models for the data of zone 1 in dataset 1 is shown in Figure 18.

Fig. 18 .
Fig. 18.Classification results of four methods for zone1 of dataset 1: (a) DCNN; (b) SAE; (c) LSTM; (d) MLP.In Figure 18, the highest diagnostic accuracy among the 10 training and testing results is taken for each DL model.As can be seen from Figure 18, at the early operation stage of the diesel engine, the cylinder temperature and room temperature are close to each other, and the amount of model training data is insufficient, which causes some interference to the early fault diagnosis.By analyzing the above experimental results, it can be concluded that under the condition of dataset 1, the diagnostic accuracy of all four DL models for zone 1 is below 80%, but from zone 2 onwards, the accuracy of each model has
, the average diagnostic accuracy of DCNN is the highest or tied for the highest in all five independent time zones of dataset 2, and the overall average diagnostic accuracy is also the highest among the four models, reaching 99.65%.In addition, LSTM ends up with the second highest average diagnostic accuracy after DCNN, reaching 98.85%, while SAE and MLP have relatively lower average diagnostic accuracies of 96.33% and 96.13%, respectively.In terms of early fault diagnosis performance, in line with the previous study, we mainly analyze the classification accuracy of the four DL models for different fault patterns in the early operation stage of the diesel engine.First, it can be clearly seen that the early fault diagnosis accuracy of all four DL models has improved significantly after CGAN-based data augmentation of the dataset.Among them, the early fault diagnosis accuracy of DCNN is 98.23%, which is still the highest among the four models, and the lowest accuracy is MLP, which is 81.58%.Secondly, the algorithm stability of the four DL models has also achieved a substantial improvement.The DCNN proposed in this paper also has the best performance in terms of algorithm stability, and its standard deviation of diagnostic accuracy is the lowest among the four models in a single time zone and the whole operation process, which is 0.0030.Meanwhile, the standard deviation of diagnostic accuracy of MLP is the largest, reaching 0.0170.Furthermore, in order to show the performance of DL models in early fault diagnosis of the diesel engine after applying the CGAN-based data augmentation method, the confusion matrix of the classification results of the above four models for the data of zone 1 in dataset 2 is shown in Figure 20.

Fig. 20 .
Fig. 20.Classification results of four models for the data of zone 1 in dataset 2: (a) DCNN; (b) SAE; (c) LSTM; (d) MLP.In Figure 20, the highest diagnostic accuracy among the 10 training and testing results is taken for each DL model.Since all the above four DL models used the SR classifier for fault pattern recognition, the analysis results for dataset 1 and dataset 2 can show that DCNN has better infrared image feature extraction capability, which is more beneficial for subsequent fault pattern recognition.Moreover, during the 25 minutes of diesel engine operation, a large part of the time is in the state of gradual temperature rise, so the classification results of both dataset 1 and dataset 2 can achieve the highest fault diagnosis accuracy and the lowest standard deviation, which proves that DCNN has better algorithm stability and stronger resistance to temperature fluctuation interference when performing fault pattern recognition of the diesel engine.To quantify the effect of CGAN-based data augmentation method on the improvement of fault diagnosis accuracy of DCNN, SAE, LSTM, MLP in the early operation stage and whole operation cycle of the diesel engine, the related calculation results are shown in Table8and Table9.

Fig. 21 .
Fig. 21.Comparison results of diagnostic accuracy of DCNN, SAE, LSTM and MLP in five time zones.According to Table 8, Table 9, and Figure 21, it is evident that after data augmentation based on CGAN, the average diagnostic accuracy of the above four models has been improved to a certain extent in various time zones and overall.SAE has the highest overall average accuracy improvement, reaching 12.61%, while LSTM has the highest early fault diagnosis accuracy improvement, reaching 65.15%.In terms of algorithm stability, the above four models have significantly improved in various time zones and overall.LSTM has the most significant reduction in overall and early fault diagnosis accuracy standard deviation, reaching 93.02% and 84.03%, respectively.
diesel engine early fault diagnosis has been greatly improved after data augmentation, in which the accuracy of early fault diagnosis of DCNN and LSTM has exceeded the accuracy in the whole time zone, while the accuracy of SAE and MLP has reached the comparable level as that in the whole time zone.It is enough to prove that a small number of datasets can yield comparable classification accurately when expanded through data augmentation methods as compared to a large real-world dataset.In summary, it can be concluded that the data augmentation method based on CGAN proposed in the paper can greatly improve the accuracy and algorithm stability of DL models for fault diagnosis of the diesel engine.Moreover, relevant experimental results indicate that DCNN has better feature extraction ability and algorithm stability compared to SAE, LSTM, and MLP, especially solving the engineering problem of low early fault diagnosis of diesel engines.The accuracy of early fault diagnosis based on DCNN can reach 98.23%.

Fig. 22 .
Fig. 22.Comparison of feature visualization before and after CGAN-based data augmentation.To verify the role of CGAN-based data augmentation method in early fault diagnosis of diesel engines, the features of DCNN after the last layer of the fully connected operation during fault pattern recognition are selected for visualization based on the data of zone 1 in dataset 1 and dataset 2, as shown in Figure 22.It can be clearly seen from Figure 22 that after data augmentation based on CGAN, the separation of features has been greatly improved, which is conducive to further feature extraction and pattern recognition.To further verify the effect of data augmentation based on CGAN on fault pattern recognition of the diesel engine, dataset 3 and dataset 4 are constructed on the basis of original data, as shown in Table 10

Fault
noted that dataset 3 is the original data without zone division of the diesel engine operation process; dataset 4 is the result of data augmentation of dataset 3 using CGAN, and since dataset 3 itself is all the data of the diesel engine during 25 minutes of operation time, which is already more than sufficient, considering the limited computer resources, dataset 4 is only expanded to twice of dataset 3.For dataset 3 and dataset 4, DCNN, SAE, LSTM, and MLP are trained and tested 10 times with the same settings of relevant network parameters, respectively, then the comparison results are shown in Table

Table 1 .
1. Technical parameters of the diesel engine and infrared thermal camera.

Table 2 .
, respectively.Detailed description of a single zone in dataset 1.

Table 3 .
Detailed description of a single zone in dataset 2

Table 4 .Table 4 .
Model structure and parameter settings of AlexNet.
Note: Std represents standard deviation.

Table 8 and
Table 9.

Table 8 .
Comparison results of the average fault diagnosis accuracy and its standard deviation for dataset 1 and dataset 2 over the whole operation cycle

Table 9 .
Comparison results of the average fault diagnosis

Table 9 ,
Comparison 1 and Comparison 2 indicate the degree of improvement in fault diagnosis accuracy and the Eksploatacja i Niezawodność -Maintenance and Reliability Vol. 26, No. 1, 2024 degree of decrease in its standard deviation before and after data augmentation using CGAN, respectively.According to Table 6 and Table 7, the comparative results of fault diagnosis accuracy of DCNN, SAE, LSTM, and MLP in five time zones during the diesel engine operation are shown in Figure 21.

Table 10 .
Detailed description of dataset 3.

Table 11 .
Detailed description of dataset 4.

Table 12 ,
the CGAN-based data augmentation method can effectively improve the accuracy of fault pattern recognition of the diesel engine.Tab.12.Comparison results of average fault diagnosis accuracy and its standard deviation for each model under dataset 3 and data set 4