Image Correction for Proper Time of Harvesting Fruit Outdoors

Reduction in the number of farmers and aging of workers have recently become a problem in Japan. Thus, agriculture and industry cooperation is actively performed. One example showing this cooperation is the utilization of a plant factory and harvesting robot for agricultural crops. However, using this type of robots makes it difficult to retain farms and trees, hence the need for a method to produce highvalue fruits without requiring intuition and experience. The proper harvest time for high-quality and -value fruits is decided upon by comparing tree fruits using a color chart defined for each breed. The assessment standard is not unified because this is a sensory exam based on sensibility information. This study aims to develop a system that would decide on the proper time of harvesting fruits from the tree. Previous studies were able to decide on the proper time of fruit harvest indoors. However, the outdoor lighting conditions change according to time and weather. Therefore, directly applying this method is difficult. In this paper, we introduce the image correction method to reduce the influence of lighting conditions. The proposed method is performed using a color chart taken together with fruit. First, we analyzed the color chart for fruit colors taken outdoors and confirmed that the influence of illumination change reduces. Second, we analyzed the fruit color. The fruit color was confirmed to have values along the breeding status. We also proposed image correction without the use of the color chart and acquired a useful feature. The results of the procedure show that quantitatively determining the proper harvesting time of fruit outdoors is possible.


Introduction
The reduction in the number of farmers and aging of workers have recently become a problem in Japan.Thus, agriculture and industry cooperation is actively performed.One example showing this cooperation is the utilization of a plant factory and harvesting robot for agricultural crops.However, using this type of robots makes retaining farms and trees difficult, hence the need for a method to produce highvalue fruits without requiring intuition and experience.
The proper harvest time for high-quality and -value fruits is decided upon by comparing tree fruits using a color chart defined for each breed.The assessment standard is not unified because this is a sensory test based on sensibility information.The inspection machine for the quality evaluation of the fruit is developed.However, these are inefficient because they are directed to fruits after harvest and include immature fruits.Therefore, efficient apple cultivation and support of efficient and sustainable farming are achievable if harvesting only the ripe apples is possible.
This study aims to develop a system to determine the proper time of harvesting fruits from the tree.A previous study [1] focused on the HSV and L*a*b* color spaces.Consequently, it was found that using H and a* in deciding on the proper harvest time was useful.However, acquiring accurate fruit color information is difficult because the lighting conditions change according to time and outdoor weather.Thus, directly applying this method was difficult.We introduce an image correction method herein to reduce the influence of the lighting conditions.

Overview
The proposed method was performed using a color chart for correction (ColorChecker Passport, X-Rite) taken together with the fruit.The color chart used black, white, and gray colors for correction.This method constitutes contrast and gamma corrections.The correction is performed as follows: halation removal, contrast correction, and gamma correction.

Halation Removal
Halation became white in one part of the image because of overexposure.Color information cannot be accurately acquired when there is halation in the image.Therefore, the standard deviation of the RGB value in each pixel is calculated as Eq.(1).Moreover, the pixel lower than the threshold is removed as halation.
where,   ,   , and   are the RGB values in each pixel.
is the average value of each pixel.

Contrast Ccorrection
Contrast correction is performed to the raw image using the pixel value of the white and black charts.The correction is calculated as follows: where, C and P are the pixel values in the correction and raw images.White and black are pixel values acquired from the white and black charts, respectively.

Gamma Ccorrection
Gamma correction is performed to the image that underwent contrast correction using the pixel value of the gray chart (reflectance: 18%).This correction is calculated as follows: where, Q and C are the pixel values in the gamma and contrast correction images.γ is calculated as follows: ()  = 119 (4) where, gray is the pixel value acquired from the gray chart.

Image for Analysis
We took a fruit and a color chart outdoors, then analyzed these using the proposed method.We used 12 apples (Yataka) cultivated in the Akita Prefecture Fruit Experiment Station.These fruits were taken together with the color chart for correction and the fruit color using a digital camera (NEX-3D, Sony).Photography was done at ~1 week intervals until the apples are harvested.The harvested apples were photographed together with both color charts indoors.

Fruit Color Chart Analysis
We verified whether or not the proposed method is useful.The color chart was used to analyze the materials using the ground color of apple.Figs. 1 and 2 show the analysis result focusing on a* and H, respectively.Table 1 shows the standard deviation of each chart number.The images herein were taken at different dates and times.However, the color chart value in the correction image was supposed to have the same value regardless of date.The a* and H values in Fig. 1 greatly varied because of the differences in the recording date.Meanwhile, the corrected image value in Fig. 2 was roughly matched.Table 1 shows that the standard deviation of the color chart in the corrected image was sufficiently small.This result suggests that the proposed method was useful in reducing the influence of the changing lighting condition.

Apple Color Analysis
We then analyzed the surface and ground colors of the apple to verify the method.The halation threshold was set to 35.It should be noted that the surface color is the color of the side of the apple.The ground color is the color of the slight influence of sunlight shown in the bottom part of the apple near the calyx.
Figs. 3 and 4 show the color variation of the surface and ground in sample No. 7. Table 2 shows the coefficient of determination R 2 of the fitting curve (quadratic function) of H and a* in the fruit color changing condition.Figs. 3 and 4 also illustrate a* as an increment and H as a decrement, respectively, with the passage of time in the surface color.Similar trends in the ground color were further observed.Therefore, variations were found in the a* and H values in the raw image.However, the value along the state of growth was observed in the corrected image.Moreover, the corrected image's coefficient of determination of the fitting curve was higher than that of the raw image in both surface and ground.Consequently, the proposed method was considered useful in deciding on the proper harvesting time of fruits outdoors.

Overview
The abovementioned results showed that the image correction process using the color chart was useful in deciding on the proper time of harvesting fruits.However, the work utilizing the color chart is inefficient.Thus, we introduce a correction method without the use of a color chart in this chapter.

Detail
We first acquired the minimum and maximum values greater than the threshold value of the frequency in the pixel histogram.The image is corrected based on the minimum and maximum values as black and white using Eq.(1).
We then calculated the histogram using a V value in the HSV color space and acquired white and black points using the same method.
The difference of the pixels in the range between white and black and the sum of squares were obtained using Eq. ( 5) as follows: DiffRG x,y =  , −  , DiffBG x,y =  , −  , DiffBR x,y =  , −  , S RGB = DiffRG x,y 2 + DiffBG x,y 2 + DiffBR x,y 2 (5) We then picked a pixel with a value less than the threshold and calculated their average value as gray R,G,B .We acquired the average RGB in gray R,G,B as A gray .The image was corrected as follows: Q R,G,B = P R,G,B + (  −  ,, ) (6) where, Q R,G,B and P R,G,B are the pixel values in the output and input images.

Exam
We analyzed the apple image to confirm the usefulness of the method.The images were taken using a time-lapse camera (TCL200, Bicom Corp).The images were acquired with 1 h intervals during ~20 days.Fig. 5 shows the average of each day of surface color and fitting curve (quadratic function).Fig. 5 shows a*, which represents the value along the fitting curve.

Conclusion
This study aimed to develop a method for use in quantitatively deciding on the proper time of fruit harvest.This study also discussed the image correction approach used to reduce the influence of the outdoor lighting    conditions.The color chart analysis result for fruit showed that the standard deviation of the a* and H values in the corrected image is lower than that of the raw image.This finding indicates that the lighting conditions are reduced by the method.Accordingly, the analysis of fruit color expressed the a* and H values along the state of growth in the corrected image.The R 2 of the fitting curve showed the change of the fruit color value along the state of growth.
We consider that the proposed method is useful in deciding on the proper time of fruit harvest outdoors.We then analyze the surface of an apple by taking images of it using a time-lapse camera and performing the correction without the color chart.a* is expressed as a value along the fitting curve.As a result, deciding on the proper time for harvesting is possible.
From now on, we will perform fruit color evaluation considering the proper time of fruit harvesting.This method enabled the determination of the proper harvesting time without using the color chart.

References
[1] K. Tsuchiya and M. Ishii: "Consideration about the method of proper time of harvesting fruits using color difference", The 2014 IEICE General Conference, D-12-85, p160.
of the color chart (corrected) analysis.
of the color chart (raw) analysis.
Fig. shows the average of each day of surface color and fitting curve (quadratic function).Fig. H illustrates that the value widely varied.Accordingly, a* is expressed as the value along the fitting curve.
of the ground color analysis.

Fig. 5
Fig. 5 Result of the surface color analysis.

Table 1
Standard deviation of the color chart.

Table 2
Coefficient of determination R 2 of H and a* under the fruit color changing condition.