A decision system of crack damage rank on infrastructures using Deep Learning

The ageing of infrastructures in Japan is progressing. Along with that, there is a shortage of experts who inspect the structures. Therefore, the support system of inspection work for structures has been demanded. So, in our laboratory, development of the damage rank decision system for structures has been developed. In this paper, we try to decide the damage rank for structures with Deep Learning, one of machine learning which attracts attention in the field of image identification in recent years. There are several types of structural damage. Firstly, we try to make models for cracks. There are about 200 images and numerical information as damage data(training and test data). In construction of the CNN models, we tried 4 input data patterns. As the results, the entire accuracy was 54.6[%] at 300x300, 57.29% at 600x600. In case of accuracy for each rank, we achieved that the best accuracy was 80.95% at rank 2 decision.


Introduction
Most infrastructures, such as office buildings, bridges, high ways, were built in 1970's and 1990's in Japan.Fig 1 shows percentage of the number of bridges which built over 50 years ago in Japan.In 2013, 18% of entire bridge in japan are built over 50years ago.In 2033, it increases to 67%.So, it is important to check and manage the infrastructures on daily basis.Ordinary, the infrastructures are checked by looking and hammering test of specialists.But, there are two problems of the checking.One is a shortage of the specialists, the other is place regulation of infrastructure such as the bridges and highways.We must develop a damage identification monitoring system by which everyone can check the infrastructures without the specialists of checking and hammering test of the infrastructures.Therefore, many monitoring systems have been developed (1)- (4) .we have also developed the health monitoring system for a testbed bridge (4) .
On the other hand, it is also important to check damage points visually on the infrastructures.In Japan, inspectors who got a license of the checking are ranking damage of the bridges.However, the number of the inspectors is decreasing.We assume that they will need a support system of the ranking of the damage on the infrastructures.Therefore, we are also developing a decision system which ranks damage level on the bridges using deep learning.Deep learning methods based on convolutional neural network (called CNN).It allows to dominate the speech recognition, vision based pattern recognition (5) , fault-tolerant control systems (6) , and so on.In particular, CNNs have brought breakthroughs toward object detection and recognition (7) .A variety of CNN applications have been constructed using a huge amount of annotated data for training of the mode such as crack detection system on steel surfaces (8) , concrete cracks (9) , and cracks on nuclear power plant components.
In this paper, we will propose the rank decision system of crack damages using deep learning.We provide a machine learning system made by about 2000 crack images and its damage level data.

Overview of a Health Monitoring System
We are developing a health monitoring system described by Fig 2 .It consists of measurement modules, a damage decision camera, and a cloud server.
The measurement modules are set on the infrastructures.The measurement modules measure vibration, then send the Fourier spectrum of the vibration to the cloud server.The base module sends Fourier spectrums received from many measurement modules to the cloud server by Ethernet.It manages data of the infrastructures in all of our country, and finds damage of them.We are able to check the waves and results of damage identification through Web.
Furthermore, the damage decision camera takes pictures of the infrastructures, then finds the damage and make decision of its damage rank automatically.In order to make decision of the damage rank, the cloud server has deep learning system using CNN.The damage decision camera embeds CNN constructed by the cloud server.
In this paper, we refer the deep learning system on the cloud server.We employ supervised learning to construct the CNN.There are several types of the damage on the infrastructure such as crack, deformation, Free lime, leakage, and so on.We provide the neural network each types of the damage.In this paper, we focused on crack damage on the bridges.The deep learning system enables judgement of damage rank for crack.The crack damage rank is classified from 1 to 5. Rank 1 means slight damage.Rank 5 means heavy damage.

Contents of The Deep Learning Data
We employed data provided by Hokkai University for deep learning.the data consist of damage photograph file and csv file.
From The CSV files consist of photograph file name, photograph's file size, damage parts, the number of the damage, damage area, kind of damage, and damage rank.

Amount of The Deep Learning Data
In case that data set (photo.and CSV file) have a deviation of the amount of data which is decided on a rank, the deep learning system doesn't study precisely.For example, in case that he number of rank 1 damage data is 1/1000 smaller than the number of another rank, a no damage area may decide on rank 1 damage.Therefore, the data set for training have a few deviations of the amount of data.That is, we chose the image data, such as the number of rank 1 data is 338, rank 2 data is 309, rank 3 data is 331, rank 4 data 312 and rank 5 data is 340.The total amount of data is 1630.(d) The resized image files cut out damage area from the original image files (no changed aspect ratio) and the size of the damage areas.

Methodology
In data pattern of (c) and (d), we cut out rectangle damage area like red line of Fig 4~Fig 7. Table 1 shows detail data of Conv, Pool, and FC.We employed common parameters described in Table 1 on pattern (a)~(d).The CNN outputs the rank data as probabilities using Softmax function.Batch size was 30.It executed training 80 times.we employ Adam as optimization methods, and training rate was 0.001.We use 90% of all image data as training data and 10% of them as the test data.

Expansion of The Image Size
There are many original images that have over 300x300 data size (over 1000[pixel] on height).In section 4.1, the deep learning system scales the original images down to 300x300 image data.There is some possibility of luck information about the damage due to scaling down to 300x300 image data.The reason why the system scales down is that the system doesn't have enough GPU's RAM to store 2000 original image data at the same time.
Fig 9 shows the method of batch transformation of the data set.The system sends all data set to computer's RAM, then transfers them from the computer's RAM to GPU's RAM.GPU constructs CNN with one batch that has all data set stored on GPU' RAM.Therefore, we have to reduce original image data to 300x300 image data to store all data set on GPU's RAM.
In order to transform larger image data, we improve the protocol of the transformation described as Fig10.In the improved protocol, we divided data set to several small batches, then the system sends a small batch data to the computer's memory one by one.The system sends the one small batch data on computer's RAM to GPU's RAM.The system enables to input larger data set to the CNN.We employ 600x600 image data in the evaluation, and construct the CNN using pattern (a)~(d).

Accuracy of Rank Decision for Each Pattern
We achieved accuracy of rank decision described as Table 3.We evaluated accuracy for training data pattern (a)~(d) described in 4-1.We took average of accuracy of test data by 3 times.The best accuracy was 54.60% for pattern (a).The accuracy increased according to the number of epoch.Finally, it achieved over 90%.The CNN makes decision of rank for training data precisely.But it is difficult to make decision of rank for test data described in above.Therefore, there is some possibility that over fitting occurs on the CNN.

Accuracy by Each Rank on Pattern (a)
We researched what rank the test data was assigned on pattern (a) described as Table .4.The cracks are classified into five ranks.In table.4,row data indicate correct rank and column data indicate rank that the CNN output.The red number that was assigned on diagonal cells in Table 4 is the number of correct decisions.The accuracy of rank 1 decision is 70.45%, rank 2 is 80.95%, rank 3 is 37.50%, rank 4 is 71.11, and rank 5 is 56.52%.Their values are not close.That is, the difference between best accuracy (80.95% at rank 2) and worst accuracy (37.50% at rank 3) is 43.45%.

Results of 600x600 Image Data Set
The Table 5 shows accuracy on the CNN constructed using 600x600 image data.The entire accuracy increases a little.The best accuracy at 300x300 images is 54.60% and the best accuracy at 600x600 is 57.29%.The best pattern is pattern (a) at 300x300.On the other hand, the best pattern is pattern (d) at 600x600.

Considerations
We evaluated several training data to construct the CNN.Impact factor of the accuracy of rank decision is change according to input data size such as 300x300 or 600x600.In the results of 300x300, it is important to input resized original data.In terms of data area, bigger data area is effective for construction of the CNN.The accuracy of original image (i.e.big area) is better than that of cutting off the crack area (i.e.small area).In the results of 600x600, it is important to cut off the crack area.In terms of data area, bigger data area is not necessarily effective for construction of the CNN.The accuracy of cutting off the crack area (i.e.small area) is better than that of original image data (i.e.big area).
We achieved that the CNN has 54.6[%] accuracy for crack at 300x300, 57.29% at 600x600.It has also potential over 80% accuracy by increase of the amount of training data.A CNN that has over 80% is used practically.

Conclusions
In this paper, we proposed the rank decision system of crack damages using deep learning.We provided four patterns of training image data.We constructed two CNN using 300x300 training image and 600x600 training image.We explained detail construction of the CNN.
As the results, the entire accuracy was 54.6[%] at 300x300, 57.29% at 600x600.In case of accuracy for each rank, we achieved that the best accuracy was 80.95% at rank 2 decision.We found out that size of the training data changes the accuracy of rank decision.In order to be more accuracy, we will find an optimum data size.We also found out that we don't have enough training data to construct the CNN that is able to decide the crack damage rank precisely.We will develop a method of fine-tuning and increase of training image data.

Fig 1 .
Fig 1. Percentage of bridges built over 50 years in japan.

Fig 3
Fig 3 shows method of construction of a CNN and the decision system using it.In Fig 3, Decision Model is the

Fig 4 to
Fig 7 show example of the damage photograph (JPG file) for each rank.Red line areas are inserted in these photographs to indicate the damage area (Deep learning data didn't include the red line).There are cases that one photograph has several damage areas (See Fig 5.).The deep learning system decides on a rank for a damage area.The size of photograph is different form each other.Minimum width size of photograph is 521[pixel], and that of maximum size is 2413[pixel].Minimum height size of photograph is 379[pixel], and that of maximum size is 1697[pixel].

Fig 3 .
Fig 3. Method of construction of a convolutional neural network and a decision system.

Fig 2 .
Fig 2. The organization of the health monitoring system.

4. 1
Training using 4 Data Patterns In this paper, we assigned the damage areas on the image files to construct the CNN for crack rank decision.We have made training data provided by 4 data patterns, then we evaluate what types can make decision damage rank precisely.The 4 data patterns are as follows: (a) The resized image files changed the aspect ratio and the size of the original image files and the damage areas.(b) The resized image files no changed the aspect ratio and the size of the original image files and the damage areas.(c) The resized image files cut out damage area from the original image files (changed aspect ratio) and the size of the damage areas.

Fig 8
Fig 8 shows constructed CNN.All the original image data was resized to 300x300.Conv indicates convolution layer, Pool indicates max pooling layer, and FC indicates fully connected layer.In pattern (c), we input crack size data to FC1.But we didn't input them to FC1 in pattern (d).Table1shows detail data of Conv, Pool, and FC.We employed common parameters described in Table1on pattern (a)~(d).The CNN outputs the rank data as probabilities using Softmax function.Batch size was 30.It executed training 80 times.we employ Adam as optimization methods, and training rate was 0.001.We use 90% of all image data as training data and 10% of them as the test data.

Fig 8 .
Fig 8.The overall architecture of the constructed CNN.

Fig
Fig 11 shows accuracy transition for training data itself in pattern (a).The accuracy increased according to the number of epoch.Finally, it achieved over 90%.The CNN makes decision of rank for training data precisely.But it is difficult to make decision of rank for test data described in above.Therefore, there is some possibility that over fitting occurs on the CNN.

Table 1 .
Parameters of the CNN model Fig 9. Batch transformation of the data set.

Table . 2
Table.2 shows the parameters of CNN at 600x600 image data set.Dropout ratio was same as Table.1.Parameters at 600x600 image data set

Table 4 .
Rank decision in detail on pattern (a).