Classification System for Golf Ball Initial Conditions using CNN Based on High-Time Resolution Images

To accurately estimate the trajectory of a golf ball after a swing impact, the initial conditions—i.e., initial velocity, angle, and backspin—need to be considered. In this paper, we propose an estimation system that can classify these initial conditions of the golf ball by applying a convolutional neural network (CNN) to high-time resolution images captured by a line scan camera. In the proposed system, to characterize the combination of the three parameters on the high-time resolution images, a golf ball is marked with a black line around its great circle. Three CNNs are constructed for each initial condition, and each CNN is trained using simulation images. By applying high-timeresolution images, the three initial conditions unknown to the trained CNNs could be estimated. To evaluate the proposed system, we conducted a validity experiment using simulation images. The initial velocity was ranged from 10–80 m/s at 5 m/s intervals, the angle was ranged from 0–0.79 rad at 0.087 rad intervals, and the backspin was ranged from 100–450 rad/s at 50 rad/s intervals. The results were evaluated in terms of accuracy and the RMSE per range of each parameter (RPR): the initial velocity, angle, and backspin showed accuracies of 100.00%, 98.44%, and 73.13%, and RPR values of 0, 0.013, and 0.079, respectively.


Introduction
Golf is a famous sport in the world (1) . As of 2017, the number of golf players was estimated to be approximately 55 million. This is because golf has some merits for players: it reduces their mental stress and provides enjoyment irrespective of age (2) . Recently, playing golf indoors has become popular because it allows players to enjoy an on-screen golf course via simulation regardless of the weather; moreover, players can analyze their playing data statistically and try to improve their performance (3) . In golf simulation, estimating the ball trajectory after impact is an important factor because the trajectory directory affects the golf score. Additionally, simulations are useful for practicing golf.
To estimate the golf ball trajectory with high accuracy, the initial conditions of the ball (initial velocity, angle, and backspin) need to be considered (4) . Therefore, many studies have proposed methods to estimate them; for instance, estimation methods using infrared (IR) scanning sensors (5,6) and high-speed cameras (7) have been proposed. An estimation method using the following integrated sensors: IR sensor, acoustic sensor, and condenser microphone has also been developed (8) . Additionally, a visual tracking system (9)(10)(11)(12)(13)(14) , an estimation method based on line sensors (15) , and an estimation method using stroboscopes have been proposed (16) .
In this paper, we proposed a system that could estimate the initial conditions after impact using high-time resolution images captured by a line scan camera. The three initial conditions are then estimated using a convolutional neural network (CNN). Fig. 1 shows the entire scheme of the proposed system, which consists of three phases: the measurement phase, where the high-time resolution image is captured using the line scan camera; the learning phase, where three CNNs corresponding to the three initial conditions are trained by the simulation images; and the classification phase, where the initial conditions for the captured images are classified by the three trained CNNs.   A ball used in the proposed system is marked with a black line around the great circle of the golf ball, as shown in Fig. 2. This figure also shows both an actual ball and a simulation ball with the marked line. In the proposed system, the marked ball is set to the initial position before swing impact. A line scan camera is set L m away from the center of the ball so that the scan plane aligns vertically to the marked line, as shown in Fig. 1. The line scan camera captures M ×1 images per frame, where M is the number of pixels of the line scan image. By capturing line scan images for N frames and combining N images, we obtain an M×N image for a golf swing. Fig. 3 shows sample images of a swinging ball after impact, obtained by the line scan camera with the above settings. Figs. 3(a) and 3(b) show a captured image in the actual environment and the simulation image, respectively. In the actual environment, a line scan camera requires a high-intensity light source to obtain sharp images; the lack of such a source explains the apparition of light reflection in Fig. 3 (a). Similar to pre-signal processing, the images are binarized, and the binarized image is compressed along the vertical direction to render an image with a size of N×N that can be input to the CNN during the learning phase. Fig. 4 shows two sample simulation images after conducting the above pre-signal processing with different initial conditions.  In the learning phase, the CNN is trained by N×N-size binarized simulation images. In the proposed system, three CNNs are applied; and each CNN corresponds to an initial condition. Drop(0.3) indicates a drop layer that is used to randomly set the input element to zero with a probability of 0.3. One can refer to the following papers for more information on CNN networks: "Batch normalization" (17) , "Wide Residual Networks" (18) , and "A Simple Way to Prevent Neural Networks from Overfitting" (19) . The line scan images are generated in a certain range from  to  for all the initial conditions. Each CNN is trained by the images using a combination of the abovementioned ranges for the three initial conditions as a label. BN

Classification Phase
In the classification phase, the three initial conditions for an image captured by a line scan camera are estimated by classifying the captured image into a label for the CNN.
The captured image is applied to the same pre-signal processing for the simulation images. The N× N binarized image is then input to each trained CNN. When the image is input into the CNN for initial velocity, the classified initial velocity for the image is obtained. Similarly, the classified angle (and backspin) is obtained by inputting the image into the CNN for angle (and backspin). Therefore, the proposed system estimates the initial conditions of the golf ball after impact.

Validity Experiment
To evaluate the proposed system, we conducted an experiment based on simulation images. Fig. 7 shows the environment in the simulation space for generating the simulation images using Unity (Unity Technologies). In the experiment, we used a golf ball marked with a black line and a light source on a PC. A golf ball was captured one frame at a time. In the simulation space, line scan camera images were generated.
The line of capture   Table 1 shows the ranges from  to  for each initial condition; the initial velocity was ranged from 10-80 m/s at 5 m/s intervals, the angle was ranged from 0-0.79 rad at 0.087 rad intervals, and the backspin was ranged from 100-450 rad/s at 50 rad/s intervals. Hence, there are 1200 combinations of initial conditions. The diameter of the simulated golf ball was set to 0.0428 m; this is typically the size of a common ball. Table 2 shows the parameter settings for each CNN.

Evaluation Criteria
In the experiment based on the simulation images, we evaluated the proposed system in terms of accuracy and root mean square error (RMSE) per range of each initial condition (RPR), which were defined as follows.
In Eq. (2), denotes the true value of the initial condition, and ̂ is the estimated value of the initial condition. Hence, the root part in Eq. (2) represents the RMSE. The accuracy term expresses how accurately the test image can be classified. The RPR expresses how inaccurate the label of a classified image is. We divided all simulation images for each initial condition into training images (75%) and test images (25%). The initial velocity had 15 combinations in the range of 10-80 m/s at 5 m/s intervals; thus, each combination had 80 simulation images because there were a total of 1200 simulation images. Therefore, 60 images from each combination were used to train the CNN to estimate the initial velocity. The angle had 10 combinations in the range of 0-0.79 rad at 0.087 rad intervals; thus, each combination had 120 simulation images and, hence, 90 images from each combination were used to train the CNN to estimate the angle. The backspin had 8 combinations in the range of 100-450 rad/s at 50 rad/s intervals; thus, each combination had 150 simulation images and, therefore, 112 images from each combination were applied to train the CNN for backspin estimation.  Tables 3 and 4 show the results for Accuracy (defined by Eq. (1)) and RPR (defined by Eq. (2)), respectively, for each initial condition. For the initial velocity, the Accuracy was 100.00%; for the angle, the Accuracy was 98.44%. The reason for these high Accuracy values for the initial velocity and angle is because the length, in the horizontal direction, and the tilt of the image of the ball correspond to the initial velocity and angle, respectively. For the backspin, the Accuracy was comparably lower. Hence, the CNN was not trained sufficiently to classify the backspin to a high degree of accuracy. According to Table 4, however, the RPR of the backspin was almost the same as that of the initial velocity and angle. Therefore, increasing the number of simulation images used for training can improve the classification accuracy and RPR.

Discussion
In the proposed system, the backspin is classified using the black mark on the ball. In some cases, however, the mark makes it difficult to classify the backspin. Fig. 8 shows cases where most classified backspins may be considerably different from the actual backspin. Both images in Fig. 8 capture only a part of the golf ball, and the afterimage of the mark on the golf ball becomes a nearly straight line. Hence, even if the backspin is large, the pattern difference between a larger backspin value and lower backspin value cannot be indicated using the images. This explains the larger backspin value, juxtaposed with the error of the initial velocity and angle. In most cases where the initial velocity was very high, the same characteristics were observed.

Conclusions
In this paper, we presented a novel classification system using a line scan camera to estimate the initial conditions of a golf ball after swing; these initial conditions are important for calculating the golf ball trajectory for simulation purposes. In the proposed system, three CNNs were applied to the line scan images to estimate each initial condition. To evaluate the proposed system, we carried out an experiment based on simulation images. From the obtained results, the accuracy of the initial velocity was 100.00%, that of the angle was 98.44%, and that of the backspin was 73.13%. Additionally, we evaluated the system in terms of an index defined as the RMSE per range of each parameter (RPR), which was used to measure how inaccurate the label of a classified image was. According to the RPR of each initial condition, the initial velocity was 0, the angle was 0.013, and the backspin was 0.079.
For future work, the proposed system needs to be evaluated using actual line scan images. In addition, we aim to refine the estimates for these initial conditions by using regression methods. We should reconsider the mark a ball has.