Effect of Sorting Algorithms on High-level Synthesized Image Processing Hardware

Image processing methods can be broadly classified into hardware and software processing. Hardware is suitable for embedded systems because of its high performance and low power consumption. In hardware development, high-level synthesis is often used because of its ease of development. However, in order to generate high-performance hardware, it is necessary to write at the software level, considering the configuration of the hardware. Since sorting algorithms are often used inside image processing, it is necessary to generate high-performance sorting algorithm hardware. In previous research, methods for generating high-performance sorting hardware using high-level synthesis and performance comparisons have been conducted, but no comparison has been made for image processing as a whole. In this study, we will examine the dynamic background subtraction method, which is an image processing method that uses sorting algorithms. As a result, it was found that simple algorithms such as bubble sort and odd-even sort can realize pipeline processing, which is a feature of hardware, and produce high-performance image processing hardware.


Introduction
Image processing methods can be divided into hardware processing and software processing. In general, hardware processing is more suitable for embedded systems because it has higher performance and lower power consumption (1)(2) . The market scale of embedded image processing systems is expanding year by year (3) . In order to increase competitiveness and gain market share in such a market, it is essential to develop and launch products with image processing hardware as early as possible. To achieve this, we use high-level synthesis tools that automatically convert high-level languages such as C into hardware description languages such as VHDL. However, high-level synthesis also has its weaknesses, and if the description does not consider the characteristics of the hardware, efficient hardware will not be generated.
If we look at the internals of image processing, there are processes where sorting algorithms are used. For example, in dynamic background subtraction using the median method, a sorting algorithm is used (4) . The median filter, which is used to remove sesame salt noise from images, also calculates the median internally. Thus, the sorting algorithm is used in various image processing and image preprocessing steps, so improving the sorting algorithm can improve the overall performance of the image processing.
Prior research has been conducted on methods for generating high-performance sorting hardware using high-level synthesis and on comparing the performance of sorting algorithms (5)(6)(7) . However, these studies evaluated sorting algorithms alone, and it is not clear to what extent differences in sorting algorithms affect the performance of high-level synthesis hardware in overall image processing. In this study, we examine the dynamic background subtraction method as an image processing method that uses a sorting algorithm, and compare high-level synthesis hardware with different sorting algorithms in terms of processing speed, hardware scale, and power efficiency.

High-level synthesizable sorting algorithms
The high-level synthesis tool cannot convert programs that contain recursive functions into a hardware description language. Therefore, the target of the evaluation is a sorting Proceedings of the 8th IIAE International Conference on Intelligent Systems and Image Processing 2021 DOI: 10.12792/icisip2021.005 algorithm that does not use recursive functions.

Bubble Sort
Bubble sort is one of the simplest methods of sorting. The elements are checked in order from the end of the column, and if there is a reversal of the size relationship, the elements are swapped back and forth. When this process is done from the end of the column to the top, the smallest element is placed at the top. The next step is to do the same process starting from the tail and working up to the second from the top, and the next smallest element will be the second. By repeating this process, the whole thing can be sorted. The program written in C language is shown in Figure 1 (a).

Odd-even Sort
Odd-even sort is an improved version of bubble sort. While bubble sort scans in one direction, odd-even sort scans in pairs, so it can be processed in parallel. The program written in C language is shown in Figure 1 (b).

Selection Sort
Selection sort searches for the minimum or maximum value from unaligned elements and replaces it with the last element of the array. The program written in C language is shown in Figure 1 (c).

Dynamic Background Subtraction Method as Image Processing for Verification
The purpose of this study is to evaluate the effect of sorting algorithms on image processing. In this study, the median-based dynamic background subtraction method is used as the image processing for verification.
First, the background subtraction method will be explained. The background subtraction method is a process of taking the difference between the input image and a background image prepared in advance. Since taking the difference shows the difference from the foreground, it can be applied to surveillance cameras.
There are disadvantages to the simple background subtraction method. In the simple background subtraction method, the foreground image is static. Therefore, in an environment where the background changes from moment to moment, it is not possible to obtain the correct difference image. In order to solve this problem, the dynamic background subtraction method exists. The dynamic background subtraction method is to dynamically update the foreground image and take the difference between that image and the input image. By using this method, the effect of background changes on the difference image is reduced. There are several methods for generating dynamic background images, including the sequential update method and the statistical background subtraction method, but here the median method using a sorting algorithm will be used for verification.
This section describes the median-based dynamic background subtraction method. First, for the generation of the foreground image, any N frames are stored in memory. Then, the data of identical pixels are sorted and the median values are obtained. The median-based dynamic background subtraction method dynamically constructs a foreground image using these median values and calculates the difference between that image and the input image. The program written in C language is shown in Figure 2

Experimental environment
The FPGA boards used in the evaluation are as follows. The configuration of the experimental environment is shown in Figure 3.

Evaluation method
Evaluate in terms of processing speed, hardware scale, and power efficiency.
Processing speed is calculated based on the number of clocks it takes to complete a series of processes.
Power efficiency is calculated by taking the hardware scale into account. The base for SLICE is the value of the simplest sorting algorithm, bubble sort HW. The higher the operating power improvement ratio; OPIR, the higher the performance of the hardware.

Experimental results
In this section, the evaluation is conducted according to the evaluation methodology defined above and the results are summarized. The frequency of the FPGA is 100MHz, and the frequency of the embedded processor is 650MHz. The image size was 256 x 256 pixels, and was randomly and automatically generated internally. The median image was generated from 16 images. A graphical representation of the execution time is shown in Figure 4.
(a) Bubble Sort Summarized in Table 1

Considerations
When image processing was performed in hardware, bubble sort and odd-even transpose sort were very fast compared to selection sort. The computational complexity of all three algorithms compared in this study is ( 2 ). However, the reason why bubble sort and odd-even transpose sort were faster is thought to be because these algorithms are very simple and suitable for pipeline processing. The total number of clocks for the entire image processing is also close to the number of pixels in the image, so it is thought that the performance of the hardware can be maximized.
Since the time spent on sorting accounts for the majority of the processing time in the median-based dynamic background subtraction method, it was found that high performance image processing hardware can be generated by speeding up the sorting algorithm.
In addition to that, the operating power improvement ratio shows that the hardware is more power efficient than the embedded processor; similar results can be obtained when compared to a PC CPU.

Experiments and Discussions
In this study, we evaluated the impact of sorting algorithms on high-level synthesized image processing hardware. Sorting algorithms are used in various types of image processing, so it is important to improve their performance. However, it is unclear to what extent differences in sorting algorithms affect the performance of actual high-level synthesized image processing hardware. Therefore, we used the median-based dynamic background subtraction method as a validation image processing to compare the performance.
In conclusion, we found that simple algorithms such as bubble sort and odd-even sort were fast because they could achieve pipeline processing, which is a feature of hardware. In contrast, selection sort could not achieve ideal pipeline processing and its performance was not as good as the former.
In the dynamic background subtraction method, it was found that the sorting of pixel values accounted for most of the processing time. Therefore, by implementing an appropriate sorting algorithm, we can build a fast system that maximizes the performance of the hardware.
In the future, we would like to verify whether similar results can be obtained with other image processing that uses the sorting algorithm.