Enhancement of Sky and Cloud Type Classification

The sky is an essential component in outdoor images. Sky and cloud type classification has applications in many areas such as image enhancement and sky image retrieval. In this paper, we improve the sky and cloud type classification rate over existing methods. Our work is based on two classification stages: sky image classification stage and sky cloud type classification stage. In sky classification stage, the image is classified into blue sky, cloudy sky, and sunset sky. Due to the impact of descriptor selection in the sky classification, we investigate ten descriptors; we show that the classifiers based on color descriptors are more accurate than the classifiers based on shape descriptors in sky type classification. We improve the sky image classification ratio using K-HSV descriptors. The sky classification with K-HSV descriptors has 77.3% correct classification rate. In cloud type's classification stage, the cloud is classified based on the sky type. For both the blue sky and the sunset sky, the cloud type is classified into six types: cloudless, thin-cirrus, cirrus, cirrocumulus, cumulus, and cumulonimbus. In cloudy sky, the cloud type is classified into three types: stratus, stratocumulus, and altostratus. The clouds are classified based on their shape and color using Gist minimum distance classification. The average correct classification rate of the clouds classifier is over 85% for cloudless, cumulus clouds, and stratus clouds and over 60% for thin-cirrus, cumulonimbus, stratocumulus, and altostratus clouds.


Introduction
Image classification is one of the major challenges to the computer vision community.Scene classification is used in many applications such as content-based indexing and organization, and content-sensitive image enhancement.In this paper, two classifier techniques are used in image classification: minimum distance classifier (4) and bag-of-words classifier (5) .
The minimum distance classifier is used to classify unknown image into classes which minimize the distance between the image features and the class features.The distance is defined as an index of similarity so that the minimum distance is identical to the maximum similarity.
Bag-of-words is the application of local features in image classification that inspires and initiates many research efforts (6) .The selection of descriptors plays an important role in the bag of visual words classification.There are three categories of descriptors used in image classification: color based descriptors, shape based descriptors, and color shape based descriptors.The scene classification approaches are used for well defined shapes (rigid) classes such as indooroutdoor (6,7) and cars-airplanes (8,9) .These classifiers are using descriptors that depend mainly on shape like the SIFT descriptor.
We study classification method for classes of nonrigid objects (sky) that vary in color and shape.The sky types (blue sky, cloudy sky, and sunset sky) have similar structure components and differ in sun light color as shown in Fig. 1.a and Fig. 1.b.The blue sky and sunset sky have the same cloud types.In each sky class the cloud type classes have similar colors and differ on structure contents as shown in Fig. 1.a.In this paper, the structure of the clouds (cloud type) in the sky is estimated by the global Gist feature.
The remainder of this paper is organized as follows: Section 2 reviews the relevant principles and backgrounds; Section 3 introduces the proposed sky image classification technique; Section 4 introduces the proposed cloud type classification technique; Section 5 presents the experimental results and Section 6 concludes this paper.

Related Work
Two subjects most related to ours are scene classification and sky detection.Scene classification is used in many applications such as content-based indexing and organization, and content-sensitive image enhancement.The scene classification problem has two stages: representing scenes (feature extraction) and learning models for classification using these representations.The selection of features is one of the main factors that affect image classification.The bag-ofvisual words categorization system can be break down into two stages (10) : Training Stage 1. Detect keypoints using either Harris-Laplace or dense sampling detectors (11) (11) are used in image classification based on Bag-of-Words: color descriptors, SIFT descriptors, and color descriptors based on SIFT.
The early proposals on sky detection (12,13) calculate a "sky belief map" and detect connected areas for the final sky.While these methods used in annotating sky regions, its segmentation result lacks the spatial consistency due to the low correct classification rates (16) .The techniques (14,15) detect smooth regions which are found on top of the image.These techniques are suitable for video applications.The method (16) used an initial sky probability (color, texture and vertical position, and the features of highly probably areas) to calculate the final sky probability.The Skyfinder (17) technique computes semantic attributes from sky images such as category, layout, richness, and horizon.These sky attributes are used for sky image search system.
This paper studies the problem of effective classification of sky type and cloud type.The Skyfinder (17) uses gray SIFT descriptor followed by HSV mean around each keypoint to produce SIFT-HSV descriptor.The SIFT-HSV descriptors are used in sky type classification.
In Skyfinder (17) , the edge detection methods, sobel edge detection and canny edge detection, are used to compute the sky richness to determine the amount of edges in the sky.The richness values are divided into 5 levels R1 to R5.The richness value is computed using the number of edge pixels in sky however the richness value does not denote the type of cloud.The two skies in Fig.Our work enhances the sky type classification using minimum distance classification and Bag-of-words classification with features mainly depending on color.The cloud types are determined using the global Gist descriptor according to its ability to characterize the principal contours and textures of the image.

Sky Image Classification
The sky is a key element in outdoor images.Given a set of sky images, our objective is to discover the features in each image in an unsupervised manner, and to use these features to perform sky type classification.Therefore we propose to improve the sky classification method (17) by using the appropriate descriptors.The sky images are classified into blue sky, cloudy sky, and sunset sky.The sky images consist of structural shape components such as clouds and sun.The color property of the sky depends on sun light intensity.The three categories of skies are differed mainly on distribution of sun light intensity as shown in Fig. 3. Two different methods for classifying sky images are described: The minimum distance classification and bag-of-words classification.

Minimum Distance Classification
One of the most popular classifiers is the minimum distance (1) classifier (MDC).The MDC classifies an unknown image into a category which has the nearest features to the features of the test image.The minimum distance classifier has many applications in image processing branches such as medical image analyses, recognition of remote sensing image contents, and image noise identification.
We study the minimum distance classifier (4) based on YCbCr color space and minimum distance classifier based on global Gist (18) .The minimum distance classification consists of two stages: training stage and testing stage.

Minimum Distance Classification Based on YCbCr
In training stage, we collect image samples for each class type.

In testing stage, for each pixel i, i =1, 2, 3 … n,
where n is the total number of pixels in the tested image I, the minimum distance is computed using eq.( 1).
Where I Cb (i) and I Cr (i) are the Cb and Cr channels, respectively, of pixel i in YCbCr color space.Each pixel is classified into blue sky, cloudy sky, or sunset sky using eq.( 2).The number of pixels classified into the three classes M j is computed using eq.(3).After the pixel classification, the test image is classified into the class that has the maximum number of pixels belongs to it using eq.( 4).
Where T 1 is a threshold (2)

Gist Minimum Distance Classification
The Gist feature is computed by convolving an oriented filter with the image at different orientations and scales (18) .This way the high-and low-frequency repetitive gradient directions of an image can be measured.The scores for the filter convolution at each orientation and scale are stored in the Gist feature.

Bag-of-Words Classification
The selection of descriptors plays an important role in the bag-of-words image classification (11) .There are three categories of descriptors used in image classification: color based descriptors such as transformed color histogram, color moment invariant, and K-HSV (keypoint HSV), shape based SIFT descriptors, and color shape based descriptors (11) such as C-SIFT, RGB-SIFT, Oppenent-SIFT, and SIFT-HSV.
In the SIFT-HSV descriptors (17) , the gray SIFT descriptors are computed for the dense sampling keypoints extracted from each image.The mean HSV is computed for the patch around the keypoints to produce SIFT-HSV descriptors.in Fig. 6.Thin-cirrus clouds are thin detached clouds.Cirrocumulus clouds are many of small white clouds called cloudlets grouped together at high levels.Cumulonimbus clouds are heavy and dense clouds extending high into the sky in towers or mountain shaped peaks.Altostratus clouds are large mid-level thin grey or blue colored clouds.Stratocumulus clouds are clumps of clouds varying in color from bright white to dark grey.
Blue sky consists of six types of clouds: cloudless, thin-cirrus, cirrus, cumulus, and cumulonimbus.Sunset sky consists of six types of clouds: cloudless, thin-cirrus, cirrus, cumulus, and cumulonimbus.Cloudy sky consists of three types of clouds: stratus, stratocumulus, and altostratus as shown in Fig. 1.

Cloud Type Classification
The clouds within each sky type have similar colors and various shapes and sizes so the classification techniques which depend on local patch descriptors as Bag-of-words do not distinguish between the deferent cloud types.The patches of cumulus and cumulonimbus clouds have similar color and shape as shown in Fig. 6.The clouds in blue sky tend to take white color.In cloudy sky, they take gray or white color.In sunset sky, they take red, orange, or dark gray.The Gist is a global image feature, which characterizes many important statistics about a scene (21) such as capturing the diagnostic structure of the image, and giving an impoverished and coarse version of the principal contours and textures of the image.
We classify the clouds in the sky images according to the sky class type individually.For blue sky and sunset sky, there are six types of clouds in the sky image: cloudless, thin-cirrus, cirrus, cirrocumulus, cumulus, and altocumulus.For cloudy sky, there are three types of clouds in sky: stratus clouds, stratocumulus clouds, and altostratus clouds.

Experimental Results
We collected 2400 sky images from Flickr.com and Google.com.We searched for sky images using keywords "sky", "cloudy sky", "blue sky", "sunset", and "cloud types".1200 images are used in training the sky type classifier.The training data consists of 400 images in each sky class type.1200 images of the downloaded images are used in testing.

Sky Classification Results
We study two techniques of image classification: minimum distance classification and bag-of-words classification.In minimum distance classification based on YCbCr pixel classification, the pixels of the training images are clustered by K-means into 16 clusters.The pixels of the test image are compared with the color means of the clusters for each sky class type as shown in eq. ( 1) to (4).
In the minimum distance classification based on Gist features, the Gist features of the training images for each sky class type are clustered by K-means into 16 clusters.The Gist of the test image is compared with means of the Gist clusters for each sky class type as shown in eq. ( 5) and ( 6).
We study some of the descriptors that are used in bag-of-words image classification.The keypoints are detected by Harris-Laplace point detector.We selected a subset of these points for which the Laplacian-of-Gaussians reaches a maximum over scale (11) (maximum invariant keypoints).The descriptors from section 3.2 are computed over the area around the keypoints.The descriptors of trained images are clustered by K-means for each class into 100 clusters.The descriptors for the input image are classified using the bag-of-words approach.
Table 1.The correct classification ratio using ten descriptors to classify images into three sky categories: blue sky, cloudy sky, and sunset sky.Table 1 shows the correct classification ratio for two classification methods using ten descriptors to classify images into three categories: blue sky, cloudy sky, and sunset sky.Table 1 shows the following: 1.The classification based on shape descriptors such as SIFT descriptors have low correct classification ratio for sky images.2. The descriptions which depend on shape and color such as RGB-SIFT and HSV-SIFT (17) are better than the one that only depends on shape.3. The descriptions which depend on color only such as Mean Ycbcr, Transformed Color Histogram, Color Moment Invariants, and K-HSV have higher correct classification ratio.
In this paper, we improved the sky classification ratio of SkyFinder (17) using classifiers based on color descriptors such as Mean Ycbcr, Transformed Color Histogram, Color Moment Invariants, and K-HSV.
We studied the relation between the number of keypoints and the correct classification rate of sky types.We used three different numbers of keypoints: 200 keypoints, maximum Harris-Laplace keypoints, and all dense sampling keypoints.The correct sky classification rate increases as the number of keypoints increases in SIFT and SIFT-HSV (17) ; where the K-HSV has higher correct classification ratio for the different numbers of keypoints as shown in Fig. 7.If the number of K-HSV descriptors for the sky type classification is reduced to 200 keypoints the sky correct classification ratio is over 80%.

Cloud Type Classification Results
For each cloud type, the Gist feature is calculated.We computed six Gist features for blue sky cloud types, three Gist features for cloudy sky, and six Gist features for sunset sky.The cloud type is determined by comparing between Gist of the input sky image and the Gist features of different cloud images within the same sky type.The cloud type of test image is the type that has the minimum distance to the input image using eq.( 5) and eq.( 6).Table 2 shows the following: 1.The clouds which have similar shape such as cloudless, cumulus clouds, and Struts clouds have high correct classification ratio.2. The clouds which have non-rigid shape distributions as cirrus clouds have low correct classification ratio.
3. The sky images which contain many types of clouds are misclassified, as shown in Fig. 8.The cloud type is classified based on cloud color and shape using Gist features.Gist features are better than the number of edges used in SkyFinder (17) to represent cloud shapes.We study the cloud type classification for nine cloud types.The average correct classification rate is over 85% for cloudless, cumulus clouds, and stratus clouds and over 60% for thin-cirrus, cumulonimbus, stratocumulus, and altostratus clouds.

Conclusions
Sky and cloud type classification can be used in many applications such as image enhancement, sky image retrieval, and weather prediction.In this paper, we improved sky and cloud type classification over SkyFinder (14) .Two image classification techniques; minimum distance and bag-of-words are used with different descriptors.In sky classification, the image is classified into blue sky, cloudy sky, and sunset sky.In sky classification, we investigate ten descriptors; we show that the classifier based on color descriptors such as YCbCr and K-HSV are more accurate than the classifier based on shape descriptors such as SIFT and Gist.We improved the sky image classification ratio using K-HSV descriptors.We obtained 77.3% correct classification ratio.
In cloud type classification stage, the cloud is classified based on sky type.In both cases, blue and sunset, the cloud type is classified into six types: cloudless, thin-cirrus, cirrus, cirrocumulus, cumulus, and cumulonimbus.In cloudy sky, the cloud type is classified into three types: stratus, stratocumulus, and altostratus.The clouds are classified based on their shape and color using minimum distance Gist feature.For the cloud type classifier, we show that the global descriptors such as Gist feature is better than local descriptors such as SIFT.

Fig. 2 .
(2.a, 2.b) have the same richness however the cloud types are different, cloudless and cirrus clouds as shown in Fig.2.a and Fig.2.b, respectively.Fig. 2.c and Fig.2.d show two skies have the same richness however the cloud types are different, cloudless and cumulus clouds.Sky images have the same richness and differ in cloud's type.

Fig. 3 .
Fig. 3. Sky images (3) have similar structure component and differed on sun light color.

.
the training stage, the Gist is computed for the training sky images in each sky class.The Gist vectors are In the testing stage, the Gist is computed for the test sky image I G .The minimum distance is computed by eq. (

Fig. 6 .
Fig. 6.Two cloud types: cumulus clouds and cumulonimbus clouds have similar local patches attributes.

Fig. 7 .
Fig. 7.The relation between the number of keypoints that are used in classification and the correct classification rate.

Fig. 8 .
Sky images have two types of clouds: a) sky image has cumulonimbus and cirrus clouds and b) sky image has cumulus and cirrus clouds.
Compute descriptors for each test image.2. Compute the histogram for each test image based on the quantized vocabulary.3. Classify the image based on the histogram.Three types of descriptors Then we apply K-means algorithm for Cb and

Table 2 .
The correct classification rate for cloud types of the three sky categories: blue, cloudy, and sunset.