Virtual High Dynamic Range Imaging for Underwater Drone Navigation

Digital images contain few orders of magnitude smaller dynamic range than capabilities of a human eye. High dynamic range allows for perception of more details under wide lighting variations. Therefore acquisition of high dynamic range images is of high importance. However, the process of collecting of images at various exposures, then joining them together is not an easy one. Not always it is even possible to acquire a series of images of different exposures. However, even a single image due to not perfect conditions of acquisition usually contains much more information than available on the first sight. In this paper we show a method of so called virtual high dynamic range imaging formation which allows for obtaining at least virtually higher dynamical range just from an image acquired at a single exposure. Thanks to this the images convey more discriminative information as it is shown in the case of navigation of the underwater drone.


Introduction
Digital images from commodity cameras have very limited dynamic range of pixels, usually few order of magnitude lower than human eyes.This greatly limits amount of conveyed information, as well as discriminative properties of pixels.Thus, obtaining high dynamic range (HDR) images finds great interest.Usually this task is done by capturing a number of images of the same scene but each with different exposure settings.This way a series of dynamic range images is obtained which, after a fusion, form a highdynamic range version.However, recovering HDR radiance map from images is a nonlinear tasks.For this purpose, a method of reconstructing image response curve and the radiance values from photographs was proposed by Debevec and Malik (10) .Then only, knowing the response function, it is possible to reliably fuse multiple images in one HDR.
The other question is how to display such a HDR image since its dynamic range is usually much higher than capabilities of standard displays.For this purpose the tone mapping is applied which turns an HDR image into an lowdynamic range (LDR) one, suitable for display.An overview of methods and techniques for HDR creation can be found in the recent paper by Sen and Aguerrebere (18) .However, fusing a series of images captured with different exposures frequently leads to smearing and other artifacts.This is due to scene dynamics and increasing exposure times.
The other possibility of acquiring HDR images is to build special type image sensors and HDR cameras.For example Lapray (14) et al. propose an FPGA based camera that can output a real-time HDR video.The key element of this system is a novel CMOS sensor which allows real-time multi-frame acquisition with various exposures.
However, all of the above solutions suffer from a number of limitations, especially if ordinary cameras are the only solution in presence of dynamic scenes.In our developed underwater drone navigation system we faced such a situation.Thus, to enhance image dynamics we resorted to a third option which is a virtual HDR.The main idea is to work with a single image but post-processed with a number of tone adjusting curves, followed by a fusion and local contrast enhancement modules.Such methods were investigated e.g. by Chang (5) et al., who showed that local contrast enhancement is inversely proportional to the local standard deviation of an image.Further on, Cvetkovic (6) et al. proposed a multi-scale local contrast enhancement method.Then Ke (13) et al. proposed hardware-efficient virtual HDR.Their main idea is to construct a virtual photography out of a single low dynamic range image, followed by a local contrast enhancement.In this paper we follow the latter idea in the system of image enhancement for underwater drone DOI: 10.12792/iciae2018.075navigation.The problem with such images is their significant degradation due to poor visibility, noise, as well as loss of colors associated with light propagation properties in water.
In this paper we propose our version of the VHDR method for enhancement of underwater images.This writes into a broader framework of development of the visual frontend for underwater drone navigation aimed at underwater search.As will be shown, the VHDR imaging allows for better detail perception.

Computation of the Virtual High Dynamic Range Images
Architecture of the image processing chain is shown in Fig. 1.At first, an input LDR image is processed by a number of tone adjustment curves, resulting in a number of tone converted images.The role of these is to expose different ranges of the input image, potentially revealing hidden details.These are then fused back to form one HDR image.After that, image range conversion and contrast enhancement follow.Architecture of the proposed VHDR method.
Luminance adjustment is done with help of the logistic function, in its basic form given as follows where A stands for amplitude, k is a parameter, and x0 denotes the middle point.It is a "S" shape function with the inflection point at x0.However, to generate series of virtual exposures we need three variants of the logistic function: convex, concave, as well as S-shaped, as shown in Fig. 2b.Also important is to assure that the domain and codomain cover the whole allowable range, which for 8-bits spans 256 levels.Thus we propose to modify (1), as follows where D is a domain range constant, A=Pmax is a maximal pixel range of the input low dynamic range image.The parameter D controls the range of values of s1, for which it is effectively different from 0 and 1, that is, these are the saturation values.In our system this parameter is set to 2 times 6 around x0, which results in D=12.Function (2) was further augmented with additional multiplicative and additive components to provide a concave, convex and S like shape, as shown in Fig. 2., as well as to cover full range of the codomain, i.e. in the range [Pmin, Pmax], which in our case is [0,255].The new proposed function reads as follows The consecutive fusion module operates with a number of V tone mapped images obtained in the previous step, as shown in Fig. 1.Then, each pixel ih in the output fused image Ih is computed as a weighted sum of all V pixels iv, as follows (17) where ih is an output pixel, iv is an input pixel from one of the V tone mapped images Iv, and the weight wv is computed as follows where  is a parameter controlling steepness of the weighting function.In our experiments  is set to a value from the range 2.5-3.5.Let us observe that a weight wv depends on a value of a relative distance of an actual pixel intensity iv from the constant middle value Pmid.In effect, "promoted" are intensity values closer to Pmid, whereas the more distant ones get much less influence on the output pixel ih in (6).Pixel values obtained in (6) are usually concentrated around middle values of the allowable pixel range since these are boosted by the weights in (7).Therefore, these are scaled by the image range conversion module, as shown in where the mean value m around a pixel ˆh , in the window controlled by two parameters sw and tw, is computed as follows In our experiments the parameters were chosen as sw=2 and tw =2, which boils down into 3x3 local windows.
When processing color images, each color channel is processed separately in accordance with the aforementioned method.

Experimental Results
The presented method was implemented in C++, using the Microsoft Visual 2017 IDE.Experiments were run on a computer equipped with the Intel® Xeon E-1545 CPU @2.9GHz processor, 64GB RAM, and 64-bit Windows 10 OS.
Fig. 3(a) shows a monochrome frame from an underwater video recorded by our drone.Its VHDR version obtained with the presented here algorithm is visible in Fig. 3(b).In the latter a number of details is discernable which were hidden in the original frame.Such level of details can improve further processing, as will be discussed.As alluded to previously, in the case of multi-channel images, each color channel is processed separately by the VHDR method.VHDR processed frames are shown in the right column of Fig. 4. Again, more details are easily perceived but also the whole image is brightened and with overall better contrast.Since each channels is processed separately, the color gamut also has changed.However, in this case such a change further improves image perception since red color, which was missed due to water filtering properties, is recovered.A qualitative evaluation of the VHDR method is difficult due to lack of ground truth data, especially for the underwater conditions.Thus, to assess its performance in the background subtraction task (11)(15)(16) (19) , an annotated Wallflower (20)(12) dataset is used.In this experiment we measured influence of VHDR preprocessing of the input images on accuracy of the output background subtraction method obtained with the PCA based algorithm, originally proposed by Oliver (15) with implementation from Cyganek (7) .Results are presented in Table 1.For almost all test sequences, preprocessing with the proposed VHDR method leads to accuracy higher by 2-3%.Further research will be devoted to measure some classification methods, as well as to build an annotated database for underwater sequences.Considering processing speed, the proposed method in serial implementation reaches real-time conditions, i.e. 30 frames of 640x480 pixels can be processed in 1s.

Conclusions
In this paper a method for image enhancement is proposed.The method computes a high dynamic range image out from a single low dynamic range one.In the proposed method, first a series of tone mapped versions of the original image is generated.These are then fused by the weighted summation which aims at boosting middle level values of pixels.Further on, the fused image is range extended and finally contrast enhanced with a local highpass filter.Although the method was developed for detail enhancement of underwater images, it can be used in other image processing applications requiring detail enhancement.As shown, image preprocessing with the VHDR filter leads to enhancement of image details which can be useful in further image processing and inferring levels.The method in our serial C++ implementation allows full real-time operation.However, it can be easily improved and parallelized.
Fig. 1.Architecture of the proposed VHDR method.
where for 8-bit input Pmid=(Pmax-Pmin)/2=(255.0-0.0)/2=127.5,which denotes a middle value of the LDR image dynamical range.On the other hand, parameter x0 is chosen from the following three values above values of x0 leads to one type of a curve as shown in Fig. 2.These are: 1. Convex case -The midtone data of the input image is stretched toward the highlight of the output side, resulting in a bright-toned image.2. S-curve -The highlight and shadow of the input signal are enhanced, resulting in a higher contrast image.3. Concave case -The midtone data of the input is stretched toward the shadow of the output, resulting in a dark-toned image.The only one remaining parameter to consider is k which value was chosen experimentally, as shown in Fig.2.In our experiments the following values were selected for each type of a tone mapping curve three x0 values, and two k parameters for each x0, a series of V=6 tone converted images Iv is generated.These parameters were chosen for experiments, although it is possible to set different values.
Fig. 1, to spread their histogram.The scaling is done in accordance with the following formula min(Ih) and max(Ih) are minimal and maximal values of the whole Ih image, respectively.The last step involves image contrast enhancement which is done with the one-scale local spatial filter, as by Cvetkovic (6) et al.Each pixel is processed as follows

Fig. 2 .
Fig. 2. Three versions of the sigmoid function used to generate virtual exposures.On the abscissathe input brightness.On the ordinatethe output brightness.Convex part (a), full S-curve (b), concave part (c).

Fig. 4
Fig.4depicts color frames from the underwater sequence.As alluded to previously, in the case of multi-channel images, each color channel is processed separately by the VHDR

Table 1 .
F value measured on the Wallflower dataset with groundtruth data provided.