Development of Real-Time Image Stitching System of Aerial Photographs

In recent years, aerial photographs have been used in various fields as the usage of drones is increasing. One such application is to grasp the situation of an afflicted area at the time of a disaster. However, only local information can be obtained from one aerial image. It is necessary to stitch multiple aerial photographs into one image to understand the condition of a wide area. In the past, aerial images were accumulated in a drone and stitched when the drone returned to the ground. Thus, it required time to understand a situation. In the proposed method, aerial images are transferred in real time from a drone in flight to a computer on the ground and stitched. Therefore, it is possible to grasp a situation rapidly.


Introduction
Aerial photographs are taken from an aircraft as a method of globally understanding the condition of a ground surface.However, this method is expensive and troublesome; hence, it is difficult to execute rapidly.Inexpensive multiwing compact drones have been used widely in recent years.These drones can be operated more easily compared to conventional radio control helicopters There is no requirement to visually observe these drones because they can be operated using images obtained from an installed camera Furthermore, the function of automatically flying a specified route using GPS position information is provided.For these reasons, drones are widely used in areas where humans cannot enter.For example, they are used for gathering information in dangerous locations such as on site in the event of a disaster.However, the aerial photographs obtained using drones provide only local image information, similar to ordinary camera photographs.Hence, it is difficult to grasp an overall situation.
Hirokawa et al. developed a surveillance system that can aquire the ground surface condition in real time using an original developed UAV (Uninhabited Aerial Vehicle) on his own [1].This system consists of a compact UAV (total length 1.2 m, width 1.7 m, total weight 2 kg) and a ground control station.The UAV can perform automatic flight based on the flight path, altitude, airspeed supported by the ground control station.The control station consists of a laptop computer and a wireless modem, which can display the acquired images by mouse operation.In the standard operation sequence using this, the UAV is taken off by hand throwing, rises to the specified altitude, and passes through the designated flight path automatically.Also, the UAV takes pictures with a builtin camera during flight, and lands on the ground with a parachute.
In order to cope with unpredictable disasters, the information grasping system should be arranged in various places to be used immediately.In terms of cost and maintenance, it would be better to use ready-made products.
In this paper, we propose a real time aerial image stitching system that can be used in various UAV.We are developing a real-time image stitching system that can rapidly collect information about a wide area immediately after a disaster.We have studied high-speed image stitching through downsampling [2].In this method, the feature points detection and matching is performed on the downsampled images, and the projection matrix is generated.Using the matrix, the stitching is conducted from two original images.However, this process is computationally expensive.In the paper [3], fast and robust image stitching algorithm based on position information was presented.Using the position information by the GPS, unnecessary processed of feature detection and image stitching was cut appropriately.Moreover, by reusing the detected feature points, the calculation cost was reduced and the low distortion stitching images was generated [4].
In these methods, the aerial photographs taken by a drone are obtained after flight and processed offline.In this paper, we propose a system that simultaneously processes and acquires photographs to generate stitched images more rapidly than our previous system.The system configuration is shown in Fig. 1.

Real-time image stitching system of aerial photographs
The proposed real-time image stitching system of aerial photographs performs imaging and stitching in parallel.It is not necessary to wait for a drone to return, and it is possible to understand a situation more rapidly.The proposed system consists mainly of image download and aerial image stitching processes.The details of these processes are described below.A flowchart of the proposed system is shown in Fig. 2.

Aerial image download
The drone used in this study is Phantom 3 Professional (hereinafter referred to as drone; manufactured by DJI).Normally, aerial photographs are saved on a microSD card mounted on the drone.We used FlashAir (SD card with wireless LAN access point function; manufactured by Toshiba) to download aerial images from the drone in flight in real time.FlashAir can execute a Lua program written by a user when it starts, HTTP access, etc.In addition, the basic APIs for FlashAir are provided freely for developers and it can access to FlashAir from external devices.Table 1 shows the used APIs in our aerial image download program.Fig. 3 shows our drone with FlashAir.First, the drone is turned on and the server shown in Figure 1  the latest aerial image is downloaded from the drone to the aerial image download program using the FlashAir APIs.When the download of the aerial image is completed, the aerial image path is transmitted to the aerial image stitching program through WebSocket communication.

Image stitching
The

Experimental equipment
The experimental equipment is described below.

FlashAir (SD-UWA064G)
This is an SDHC memory card with wireless LAN created by Toshiba Co., and it is used to wirelessly transmit aerial images from the drone.Programming in Lua language is possible using this product, and a program is executed with the trigger of power on, HTTP access, or file writing.Table 2 shows the specifications of this product.Fig. 4 shows FlashAir used in the experiment.

MacBook Pro (13-inch, 2017)
This is used as a server for downloading and stitching aerial photographs.Table 3 shows the specifications of this machine.

Experiments and results
Experiments were conducted with the aim of generating a stitched image and measuring the processing time of the proposed real-time image coupling system.The experiments were conducted at 2nd sports ground of Shijonawate Campus in Osaka Electro-Communication  In the experiments, we first set up two flight paths, i.e., a straight flight path (Fig. 5) and a round-trip flight path (Fig. 6), using DJI GS PRO.We measured total flight time, coupling time, and the time required for aerial photography when stitching the aerial photographs acquired in these flight paths.Total flight time is defined as the time from the takeoff to the landing of the drone.Stitching time is defined as the time from the first image download request to the completion of stitching the last image.Aerial photographing time is defined as the time from the first image to the download request of the last image.The flight altitude on both flight paths was 19.5 m.The information of aerial photographs is shown in Table 4.
We obtained nine aerial photographs in the straight flying path.Fig. 7 shows the Google Maps satellite image at the same location as that of the stitched image generated from the nine images.The time chart of the straight flight path is shown in Fig. 8. Eighteen aerial photographs were obtained in the round-trip flight path.Fig. 9 shows the Google Maps satellite image at the same location as that of the stitched image generated from these images.The time chart of the round-trip flight path is shown in Fig. 10 Fig. 11 shows a part of the log of the aerial image download program and the aerial image stitching program, which executes the real-time coupling process.Table 5 shows the download time for each fixed distance, and Table 6 shows the stitching times in the straight-flight path and round-trip flight path and the stitching time of two images.

Conclusions
Based on the experimental results, it is confirmed that an image is read at the same time at which the download is completed and that the stitching process is started at the time of downloading the second image.In addition, as the positions and shapes of tracks and silver vinyl sheets are similar in Fig. 7 and Fig. 9, it can be said that reasonable coupling is achieved in both flight paths.Based on the time chart of the straight flight path (Fig. 8), it can be confirmed that the coupling process is completed before landing.Consequently, it can be confirmed that the proposed method generates the stitched aerial image faster compared to the conventional method.However, in the round-trip flight path, as shown in Fig. 10, the coupling process is not completed until landing.This is because of the wireless transmission distance of FlashAir, and it is assumed that this problem can be solved using more powerful wireless equipment in future

Future tasks
In this paper, by installing FlashAir in drones, realtime image stitching processing can be executed with a general purpose drone.However, since the aerial image download program and the aerial image stitching program depend on the operating environment, setup takes time.Therefore, by operating these in a container type virtualization environment, we will make it a system that is versatile and easy to use.

Fig. 1
Fig. 1 System configuration Fig. 2 Flow chart of real-time aerial photography stitching.
aerial image stitching program runs on the same computer as the aerial image download program.It connects to the aerial image download program through the websocket at starting.The aerial image stitching program reads the aerial images based on the received image file path .In our aerial image stitching program, image data and image features are handled in the aerial image class defined by our program.The image path is passed with the constructor argument of this class, and image reading and high-speed feature processing [2] are executed.This successfully realizes image distortion reduction [4].Our program also defines a stitching management class.This class has an array of aerial image classes inside.This class includes communication processing with the aerial image download program by websocket.The processing flow is shown as the follow.First, the aerial image download program downloads the aerial image and transmits the aerial image path via websocket.Next, when receiving the image path, the instance of the stitching management class creates an instance of the aerial image class and adds it to the internal array.When two or more images become, the stitching process is started.

phantom 3 Professional
This drone is manufactured by DJI Corporation, and it consists of a 4K camera.The maximum flight time is approximately 23 minutes.It is used to shoot aerial images.DJI GS PRO This is an application provided by DJI Corporation, and it runs on iPad.It is possible to perform autonomous flight by setting the flight path of the drone.

Fig. 8
Fig. 8 Time chart of straight flight

Table 2
Specifications of FlashAir