Improving object detection of UAV images with Real-ESRGAN

Abstract

Keywords: Super-resolution; ESRGAN; YOLO; object-detection; UAV.

Introduction

Unmanned Aerial Vehicles (UAVs) have rapidly evolved into a versatile and increasingly popular technology in recent years. UAVs find applications across a wide

spectrum, especially in the defense industry with active use, encompassing areas such as border security, target identification, electronic warfare, among others. Beyond defense, they are extensively employed in fields like agricultural surveillance, environmental monitoring, energy

*Corresponding author. *E-mail address: settarygt@gmail.com Published by Yıldız Technical University Press, İstanbul, Türkiye This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023

sector inspections, law enforcement operations, aerial photography and even entertainment drone flights. This diversity stands as one of the fundamental factors contributing to the growing popularity of UAVs. Furthermore, the advancement of technology and the reduction in costs have made UAVs more accessible, leading to a broader adoption of their advantages by a greater number of individuals and organizations. One of the common applications of UAVs is object detection. In these applications, models consisting of YOLO (You Only Look Once) [1] and its derivatives are popularly used. YOLO is a deep learning algorithm and model used in the field of object detection. YOLO aims to detect and classify objects in an image by examining the entire image in a single pass. Unlike some other object detection methods, YOLO can detect and classify multiple objects simultaneously without requiring multiple processing steps. In other words, it can recognize objects in an image all at once. However, since the images obtained from UAVs are generally of low-resolution (LR), it directly impacts the accuracy of object detection applications in a negative way. To address this drawback, various super-resolution (SR) models have been developed [2–7]. In this article primarily focused on models derived from Generative Adversarial Networks (GAN) [3, 4, 8]. GAN networks consist of two comparative structures. The first one, the Generator (G), aims to create a new image from simple noise data. On the other hand, the Discriminator (D), aims to distinguish whether the images generated by the G are real or fake [9]. In super-resolution applications, the G tries to obtain high-resolution (HR) from a LR image, while the D tries to distinguish whether this image is real HR or generated from LR. In object detection, training with SR images has been observed to results 8% better in quality compared to training with LR images [10]. These GAN models are Super-Resolution GAN (SRGAN), Enhancement SRGAN (ESRGAN) and Real-ESRGAN, chronologically. However, due to the normalization algorithms they employ, SRGAN and ESRGAN architectures significantly lag behind RealESRGAN in improving object detection [11]. Therefore, in this article focused on the Real-ESRGAN architecture. Figure 1 illustrates the general architecture of how object detection is performed from the LR image at the proposed method in this article.

The rest of this paper is organized as follows. Section Literature Review provides the related works. Section Methodology presents the proposed system in detail and experimental results. Finally, the concluding remarks are shown in Conclusion.

Literature Review

This article investigates the impact of Super Resolution models on object detection. In this section, we discuss existing methods related to our work. The problem of enhancing the resolutions of LR images has been a long-standing issue, and various solutions have been proposed over the years. In 2017, the SRGAN architecture was introduced for this problem [4]. GAN based methods, such as SRGAN and ESRGAN, demonstrated remarkable performance in improving LR images. In 2018, the SRGAN architecture was further developed, leading to the proposal of the ESRGAN architecture. In this architecture, a structure called Residual Dense Block (RDB) was introduced along with an improved loss function. RDB made learning easier and allowed for the creation of deeper models. Additionally, the Batch Normalization Block in the SRGAN architecture was removed, reducing computational complexity and enabling faster training [8]. In 2021, the Real-ESRGAN architecture focused on a U-Net based Discriminator. The proposed U-Net based architecture provides detailed per-pixel feedback to the generator while maintaining the global coherence of synthesized images by offering global image feedback as well [12]. It has been suggested that using pure synthetic data during the training of the Real-ESRGAN model offers better visual performance [3]. However, despite the realistic appearance of the generated HR images, especially high-frequency details (such as image edges) suffer from degradation. Some studies have suggested that edge information plays a crucial role in object detection and that images generated by GAN networks have a negative impact in this regard. Therefore, preserving this feature enhances object detection accuracy [10]. When combined with edge detection, a 2% increase in object detection accuracy has been observed [13]. Deep learning-based object detection architectures are broadly categorized into two main types: One-stage detectors and Two-stage detectors. While One-stage detectors are

Figure 1. General Architecture of Object Detection from LR Image and SR Image.

Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023

efficient, they have lower accuracy rates compared to Twostage detectors. Xing et al., [14], worked on object detection with the UAV datasets which they created. In their studies, the PReLu function was proposed instead of the generally used ReLu activation function. It has also been suggested to remove the normalization block as in other studies. Through the utilization of SR images, the mean Average Precision (mAP) increased from 64.82% to 68.38, while the missing rate decreased from 16.98% to 14.23% The impact of SR images on object detection was demonstrated in a paper [15]. In the utilized architecture, an Edge Enhancement network was employed between the GAN block and the Detection block. The study used the COWC dataset and the one-stage detector SSD for object detection. When trained and tested with LR data, an accuracy rate of 61.9% was achieved. However, when ESRGAN was used to enhance the resolution and subsequently train and test with SR data, the accuracy increased to 85.8%. Finally, when an Edge Enhancement network was used between ESRGAN and SSD, resulting in edge-enhanced super-resolution GAN (EESRGAN), an accuracy rate of 86% was attained [10]. In another similar study conducted in by Zou et al. [13], a four-stage object detection architecture was presented. In this approach, the ESRGAN block was initially used to increase resolution. Subsequently, an Edge Enhancement block was employed to extract edge information. Then image segmentation was applied, and the processed image was fed into the object detection block (YOLOv3). Using the COCO test-dev dataset, they achieved an accuracy rate of 42.2%, a significant improvement compared to the 31% accuracy rate of YOLOv3 without the enhancement stages. Akhyar et al. [16] conducted a study on the steel industry’s significant problem of defect detection and proposed a solution that involved the integration of GAN and a onestage detector (SSD). They used the Severstal Steel Dataset for testing. When testing with LR images using another one-stage detector (YOLO-X), they achieved a 65% accuracy rate. However, when testing with SR images generated by GAN networks and using SSD for object detection, they reached an accuracy rate of 80.4%. In a different application involving pothole detection, Salaudeen and Çelebi et al. [17] used ESRGAN to enhance SR images and EfficientDet

for object detection. They achieved a 32% accuracy rate with SR images, while the accuracy rate with LR images lagged significantly at 10.6%. In 2023, Lv et al. [18] used Real-ESRGAN in combination with SSD for missing bolt sub-detection, resulting in a 9.59% increase in accuracy compared to without GAN. Zhang et al., [19] used similar method for detection Rice Leaf disease. Also, Chen et al., [20] proposed a method for helmet wearing detection. Maqsood et al., [21] achieving the wheat stripe rust classification using the GAN methods.

Methodology

The proposed architecture aims to enhance the resolution of images taken from UAVs, allowing for the detection of objects that would normally go unnoticed. With this goal in mind, the architecture is presented in two main sections. Initially, the LR image is processed within the GAN block, where a pretrained Real-ESRGAN model is employed to generate an SR image. Subsequently, these SR images are individually fed into the Object Detection block, where the YOLOv7 architecture is utilized for object detection. The results obtained from this process are then compared with the results of directly using YOLOv7 on LR images, and the success rate of the architecture is determined. Real ESRGAN, presented in Figure 2, is an improved version of ESRGAN. One of the key differences is the shift from the classical “first-order” degradation model to a “high-order” degradation model, as previous versions were unable to restore images with unknown and complex degradations. This is because when we take photos with our cellphones, they may exhibit various degradations such as camera blur, sensor noise, sharpening artifacts, JPEG compression, etc. To address this complexity and improve training stability, a new degradation model has been proposed. Additionally, changes have been made in the D structure, incorporating U-Net design and Spectral Normalization, while no changes have been made in the G structure. However, the architecture used in this work is designed to demonstrate the only impact of SR images obtained from LR images on object detection. As stated in the introduction, LR images are first processed using the Real-ESRGAN structure to obtain SR images. The used Real-ESRGAN [3] architecture is a pre-trained model trained with the DIV2K

Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023

Figure 3. General Architecture of Proposed Model. [22], Flickr2K [23], and OutdoorSceneTraining [24] datasets [3]. The resulting SR images are then input into the YOLOv7 model for object detection. YOLOv7 [25] is a significant iteration of the YOLO object detection model, widely recognized in the field of computer vision. YOLOv7 is designed to enhance the speed and accuracy of object detection while reducing model complexity. This version builds upon the success of its predecessors by optimizing the network architecture and introducing various improvements. YOLOv7 utilizes a streamlined model design, reducing computational requirements and enabling real-time performance on a variety of platforms. It incorporates advancements in anchor clustering, network scaling, and feature pyramid networks to enhance object detection precision. YOLOv7 is well-suited for a range of applications, including autonomous driving, surveillance, and image analysis, where efficient and accurate object detection is crucial. In the general architecture showed in Figure 3, the LR image from the VisDrone Dataset is input into the Real-ESRGAN block to obtain a higher resolution SR image. Object detection was performed on the resulting SR image using the pre-trained YOLOv7. The results are provided as output. Experiments and Results The LR images used in the proposed model were extracted from the images in the Validation section of the VisDrone dataset [26] include 548 images. The VisDrone dataset is a comprehensive benchmark dataset for visual object tracking, object detection, and single-object tracking tasks. It consists of high-quality aerial video sequences captured by various drone-mounted cameras. The dataset encompasses diverse scenarios, including urban areas, natural landscapes, and congested public events. It features

a wide range of challenges such as object occlusion, scale variation, fast motion, and complex object interactions. The VisDrone dataset provides a rich collection of annotated data, including object categories, bounding boxes, object trajectories, and attributes, making it a valuable resource for developing and evaluating computer vision algorithms. Researchers and developers use this dataset to advance the field of visual object tracking and object detection, particularly in aerial and drone-related applications. In [27], various object detection architectures were compared using three different datasets. In our paper, we will use the comparative results data to perform object detection with the same dataset after increasing its resolution using our own architecture. This will clearly demonstrate the impact of Super Resolution on aerial images object detection. The comparative results obtained are shown in Figure 4. In this paper, the results will be compared by applying the same object detection models again, using the SR versions obtained by employing the ESRGAN model on the same datasets. The architecture in this paper has not been fully established and the development process continuing. Currently, at this stage of the study, the SR image was generated by employing the pretrained Real-ESRGAN model within an algorithm running on a T4 GPU in the Google Colab environment. The results were obtained as a result of the integration of Real-ESRGAN and YOLOv7. Accordingly, it has been observed that SR image detects more objects and the accuracy rates of the detected objects also increase. The results are shown figures 5-8. In Figures 5 and 7, the object detection results obtained using YOLOv7 with LR images are showed, while Figures 6 and 8 demonstrate the results obtained using SR images.

Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023

Figure 4. Comparison of mAP for three datasets [27]. Figure 4 shows the detection of 15 persons, 18 cars, 1 bus, 4 trucks, and 1 traffic light using LR images with YOLOv7. In Figure 5, object detection with SR images identified 14 persons, 1 bicycle, 29 cars, 1 motorcycle, 1 airplane, 1 bus, 5 trucks, and 3 traffic lights with YOLOv7. The original image has 37 persons, 4 bicycle, 34 cars, 13 van, 1

bus, 16 motor, tricycle 6. This means that while the detection made with LR images achieved a %40,5 mAP (15 of 37) accuracy rate for people and %52,9 mAP (18 of 34) for cars, the detection made with SR images achieved a %37,8 mAP (14 of 37) accuracy rate for people and %85,3 mAP (29 of 34) accuracy for cars.

Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023

Figure 7 shows the detection of 3 persons, 22 cars, 1 bus, 4 trucks, and 1 traffic light using LR images with YOLOv7. In Figure 8, object detection with SR images identified 7 persons, 28 cars, 1 motorcycle, 1 bus, 4 trucks, and 2 traffic lights with YOLOv7. The original image has 13 persons, 4 tricycle, 40 cars, 3 truck, 1 bus, 4 motor. This means that while the detection made with LR images achieved a %23,1 mAP (3 of 13) accuracy rate for people and %55 mAP (22 of 40) for cars, the detection made with SR images achieved a %53,9 mAP (7 of 13) accuracy rate for people and %70 mAP (28 of 40) accuracy for cars.

Conclusion

Object detection in UAV images is one of the popular problems of recent times. Since these images are generally obtained from a safe flying distance, the objects are small and have low resolution. In this article, it is aimed to increase object detection in low resolution images with Real-ESRGAN and YOLO integration. According to the results obtained, more small objects could be detected in high-resolution images and an increase in the accuracy rates of the detected objects was observed. It is envisaged that these results will be further improved with the Edge Enhancement and Image Segmentation blocks obtained in other studies.

Data Availability Statement

The published publication includes all graphics and data collected or developed during the study. VisDrone Dataset: https://github.com/VisDrone/ VisDrone-Dataset

Conflict Of Interest

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics

There are no ethical issues with the publication of this manuscript.

References

1. Networks." arXiv, Mar. 19, 2021. Accessed: Oct. 24,
2. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, 2023.: http://arxiv.org/abs/2002.12655 "You Only Look Once: Unified, Real-Time Object [13] H. Zou, Y. Gao, X. Guo, and M. Zheng, "Small Object Detection." arXiv, May 09, 2016. Accessed: Nov. 02, Detection Based on Super-Resolution Enhanced 2023. http://arxiv.org/abs/1506.02640 [CrossRef] Detection Network," in 2021 International
3. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, Conference on Computer Information Science and "Enhanced Deep Residual Networks for Single Artificial Intelligence (CISAI), Kunming, China: Image Super-Resolution." arXiv, Jul. 10, 2017. IEEE, pp. 99−106, 2021. [CrossRef] Recent Adv Sci Eng, Vol. 3, No. 2, pp. 33−39, December, 2023 39
4. C. Xing, X. Liang, and Z. Bao, "A Small Object [21] M. H. Maqsood, R. Mumtaz, I. U. Haq, U. Shafi, Detection Solution by Using Super- Resolution S. M. H. Zaidi, and M. Hafeez, "Super Resolution Recovery," in 2019 IEEE 7th International Generative Adversarial Network (SRGANs) for Conference on Computer Science and Network Wheat Stripe Rust Classification," Sensors, Vol. Technology (ICCSNT), Dalian, China: IEEE, pp. 21(23), Article 7903, 2021. [CrossRef] 313−316. 2019. [CrossRef]
5. E. Agustsson, and R. Timofte, "NTIRE 2017
6. Y. Chen, J. Li, Y. Niu, and J. He, "Small Object Challenge on Single Image Super-Resolution: Detection Networks Based on Classification- Oriented Super-Resolution GAN for UAV Aerial Dataset and Study," in 2017 IEEE Conference Imagery," in 2019 Chinese Control And Decision on Computer Vision and Pattern Recognition Conference (CCDC), Nanchang, China: IEEE, pp. Workshops (CVPRW), Honolulu, HI, USA: IEEE, 4610−4615, 2019. [CrossRef] pp. 1122−1131, 2017. [CrossRef]
7. F. Akhyar, E. N. Furqon, and C.-Y. Lin, "Enhancing [23] R. Timofte, E. Agustsson, L. V. Gool, M.-H. Yang, Precision with an Ensemble Generative Adversarial L. Zhang, "NTIRE 2017 Challenge on Single Image Network for Steel Surface Defect Detectors Super-Resolution: Methods and Results," in 2017 (EnsGAN-SDD)," Sensors, Vol. 22(11), Article 4257, IEEE Conference on Computer Vision and Pattern 2022. [CrossRef] Recognition Workshops (CVPRW), Honolulu, HI,
8. H. Salaudeen and E. Çelebi, "Pothole Detection USA: IEEE, pp. 1110−1121, 2017. [CrossRef] Using Image Enhancement GAN and Object
9. X. Wang, K. Yu, C. Dong, and C. C. Loy, "Recovering Detection Network," Electronics, vol. 11(12), Article 1882, 2022. [CrossRef] Realistic Texture in Image Super-resolution by
10. H. Lv, A. Zhao, Y. Wang, H. Li, Y. Li, and G. Zou, Deep Spatial Feature Transform." arXiv, Apr. 09, "Research on Missing Bolt Sub-Detection Based on 2018. Accessed: Nov. 03, 2023. http://arxiv.org/ Real-ESRGAN Super-Resolution Reconstruction," abs/1804.02815 in 2023 4th International Seminar on Artificial [25] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, Intelligence, Networking and Information "YOLOv7: Trainable bag-of-freebies sets new state- Technology (AINIT), Nanjing, China: IEEE,. pp. of-the-art for real-time object detectors." arXiv, Jul. 520−523, 2023. [CrossRef] 06, 2022. Accessed: Oct. 27, 2023. http://arxiv.org/
11. P. Zhu, L. Wen, X. Bian, H. Ling, and Q. Hu, "Vision Method Based on a Dual GAN," IEEE Access, Vol. Meets Drones: A Challenge." arXiv, Apr. 22, 2018. 11, pp. 21176−21191, 2023. [CrossRef]
12. S. Chen, J. Lan, H. Liu, C. Chen, and X. Wang, Nov 13, 2023.: http://arxiv.org/abs/1804.07437 "Helmet Wearing Detection of Motorcycle Drivers [27] Z. Saeed, M. H. Yousaf, R. Ahmed, S. A. Velastin, and Using Deep Learning Network with Residual S. Viriri, "On-Board Small-Scale Object Detection Transformer-Spatial Attention," Drones, Vol. 6(12), for Unmanned Aerial Vehicles (UAVs)," Drones, Article 415, 2022. [CrossRef] Vol. 7(5), Article 310, 2023. [CrossRef]

Share and Cite