A Cloud-Based Framework for Large-Scale Monitoring of Ocean Plastics Using Multi-Spectral Satellite Imagery and Generative Adversarial Network

Abstract

Marine debris is considered a threat to the inhabitants, as well as the marine environments. Accumulation of marine debris, besides climate change factors, including warming water, sea-level rise, and changes in oceans’ chemistry, are causing the potential collapse of the marine environment’s health. Due to the increase of marine debris, including plastics in coastlines, ocean and sea surfaces, and even in deep ocean layers, there is a need for developing new advanced technology for the detection of large-sized marine pollution (with sizes larger than 1 m) using state-of-the-art remote sensing and machine learning tools. Therefore, we developed a cloud-based framework for large-scale marine pollution detection with the integration of Sentinel-2 satellite imagery and advanced machine learning tools on the Sentinel Hub cloud application programming interface (API). Moreover, we evaluated the performance of two shallow machine learning algorithms of random forest (RF) and support vector machine (SVM), as well as the deep learning method of the generative adversarial network-random forest (GAN-RF) for the detection of ocean plastics in the pilot site of Mytilene Island, Greece. Based on the obtained results, the shallow algorithms of RF and SVM achieved an overall accuracy of 88% and 84%, respectively, with available training data of plastic debris. The GAN-RF classifier improved the detection of ocean plastics of the RF method by 8%, achieving an overall accuracy of 96% by generating several synthetic ocean plastic samples.

1. Introduction

Detection of anthropogenic marine debris (AMD) accumulated in coastlines, ocean surfaces, marshes, estuaries, and even in marine depths is among the most challenging environmental issues worldwide [1,2,3]. AMD is considered one of the current growing threats to the Earth’s environment’s health, particularly the oceans [4,5,6]. Marine litter can be defined as “any persistent, processed, manufactured solid material discarded, disposed, or abandoned in the coastal or marine environments” [7]. Plastics are about 50% to 90% of the total AMD, according to several studies on the composition of AMD globally [8,9]. It should be noted that the production of plastics raised from 5 to 311 million tons from 1960 to 2014 [10]. Marine debris, specifically the long-lasting plastics that are ubiquitous in the marine environment, has become a global threat [11,12,13]. Marine litter, plastics, in particular, can accumulate in high densities on the seafloor. As a result, it has harmful effects on the livelihood of marine inhabitants. Entanglement and ingestion, which leads to the injury and death of fishes, birds, marine mammals (i.e., seals), and turtles, is considered one of AMD’s negative impacts on the marine environments [14,15,16,17,18,19]. In addition, the marine debris moved to the ground surface (e.g., beaches), in particular the plastic fishing nets, can physically damage the reef and beach environments and continue to entangle and cause the death of inhabitant species located in reefs, ocean, and sea surfaces, as well as the beaches.

Moreover, non-organic and organic pollutants can be transferred by the AMD [20]. Additionally, AMDs are used as shelters for small animals [21]. Consequently, gas exchange between sediments and water columns can be reduced, displacing several benthonic species [21]. On the other hand, with the reduction of aesthetic value that leads to the decrease of economic activities of infrastructures in beaches and other public places, the AMD will negatively affect the livelihood of human beings [1]. Marine pollutants, such as buoys, derelict fishing nets, and ropes that become entangled in reefs have placed a significant threat on the inhabitants and marine species [22]. Although the long-term effects of marine pollutants on the ocean’s health are not apparent, it is believed that AMDs will play a significant role in the potential collapse of aquatic health, particularly in the oceans. The AMD, with other global major factors, including warming waters, rising sea levels, and changes in the chemistry of oceans, have added significant global stress on the livelihood of fauna and fungi globally. As a result, developing and proposing new tools to monitor and remove marine pollutants at sea and ocean, as they have become a danger to critical habitats, is of great importance at national and global levels. Similar to any environmental monitoring problem, it is vital to recognize the limitation and difficulties of detecting marine pollutants at sea/ocean surfaces. Moreover, understanding the behavior and characteristics of AMD is of great importance [22].

Beach surveys that are conducted mainly through volunteering groups are the most common method of marine debris detection [23], which is spatially limited for small-scale marine debris monitoring. The other possible technique that is a low-cost and efficient method of AMD monitoring is the use of webcams [24,25,26], mostly used for specific beaches and remote area marine pollution monitoring. It is worth highlighting that although AMD detection/monitoring with the utilization of webcams is an efficient and low-cost technique, it cannot be used as a large-scale AMD monitoring technique. Light detection and ranging (LiDAR) is another technology proposed for marine debris monitoring [27]. Due to the high cost of data acquisition and processing of LiDAR data, it cannot be used as a large-scale AMD monitoring. Aerial survey with the utilization of fixed-wing planes is another potential tool for the detection/monitoring of abundant AMD accumulated in coastlines and beaches [28]. The utilization of aerial surveys, which require several ground control points, is valuable and efficient for large-scale AMD monitoring. However, it requires a higher cost than conventional volunteering beach surveys and automated webcam beach monitoring. It should be noted the utilization of aerial photographic surveys with high-resolution digital cameras has played a significant role in the efficient large-scale AMD monitoring/detection in recent years [29,30,31].

It is worth highlighting that abundant marine pollutions, specifically plastics, have two possible fates when entering the ocean—floating on the surface of the sea or sinking [32]. In addition, inevitably, macroplastics will degrade into microplastics. Therefore, detection of large marine pollutions would be significantly useful for the protection of marine environments. Several studies conducted on the detection of macroplastics (>5 mm) have shown the possibilities of the use of optical satellite data [33,34,35,36,37,38]. Currently, remote sensing is regarded as the leading technology for the collection of high-quality images of the Earth’s environment. Factors such as limited and coarse spatial, temporal, and spectral satellite data were the reason for the few previous studies conducted on the utilization of satellite data for the detection/monitoring of AMD. For example, although satellites, such as RapidEye and SkySat, have high spatial resolution images, they provide few spectral bands. The launch of Sentinel-2A and 2B by the European Space Agency (ESA) in 2015 and 2017, respectively, substantially improved the spectral and spatial resolution of the available Earth observation data that can be efficiently used for the detection of marine pollutants, specifically the large-size plastics [39,40,41,42].

There has been no research on large-scale mapping/monitoring of marine pollution using advanced technologies, such as state-of-the-art deep learning models and cloud API such as Sentinel Hub. Therefore, we proposed the use of the generative adversarial network (GAN) model on the Sentinel dataset for the large-scale detection and monitoring of marine pollution at sea/ocean surfaces. Our main objective was to assess the potential utilization of the leading technologies, such as Sentinel Hub and Google Colaboratory, to solve the large-scale detection of large-size marine pollutants faced by the current stressed global marine environment. Moreover, we evaluated the performance of two well-known shallow algorithms, namely random forest (RF) and support vector machine (SVM), as well as the deep learning method of the generative adversarial network–random forest (GAN-RF) for the detection of marine pollution (i.e., plastics). It should also be noted that the study was carried out on material floating on the surface of the sea and not on submerged material.

2. Methods

2.1. Study Area and Satellite Data

The study area is located in Mytilene, Greece (Figure 1). Mytilene was built on the southeast of the capital of the Greek Lesbos Island, which is also the capital and administrative center of the region of North Aegean. The land in Mytilene Island is approximately equal to 107.46 km², with a population of 36,196 inhabitants. The average maximum temperature in Mytilene Island is around 30.4 Celsius in July, and the average minimum temperature reaches about 6.7 Celsius in January. It is worth highlighting that Greece has been a subject of extensive research for marine pollution detection, as there have been numerous reports on accumulated pollutions on its various beaches [43,44,45,46].

Training data used in this study are part of the PLP2019 dataset [47]. The data were created for a project to evaluate the possibilities of the utilization of Sentinel-2 optical images for AMD detection (Figure 2) [47]. As the data were collected and prepared to assess the use of optical Sentinel-2 images for the detection of large-size marine debris, 5 Sentinel-2 satellite images can be used and evaluated for the AMD monitoring/detection (

Table 1. Description of the Sentinel-2 level-1 C images that can be used for debris mapping.

Image	Date of Acquisition
S2B_MSIL1C_20190418T085559_N0207_R007_T35SMD_20190418T110441	18 April 2019
S2A_MSIL1C_20190503T085601_N0207_R007_T35SMD_20190503T103221	3 May 2019
S2B_MSIL1C_20190518T085609_N0207_R007_T35SMD_20190518T113032	18 May 2019
S2B_MSIL1C_20190528T085609_N0207_R007_T35SMD_20190528T115440	28 May 2019
S2B_MSIL1C_20190607T085609_N0207_R007_T35SMD_20190607T110335	7 June 2019

) in the pilot site of Mytilene Island. On 18 April 2019, data were created as 45 m by 5 m plastic objects made of plastic bottles (50 percent volume) and plastic bags (50 percent volume). The University of Aegean team positioned 45 m by 5 m plastic targets at a long distance from one another on 3 May 2019. On that day, in addition to the four primary objects, 21 m by 10 m targets were also utilized. One of the added objects was built out of plastic bottles, while the other was made of a mix of plastic bags and plastic mesh. On 7 June 2019, the team performed another test with 45 m by 5 m plastic objects that covered 12 Sentinel-2 grids (for more information, refer to Topouzelis [47]). In our study, we used the Sentinel-2 Level-2 A image collected on 18 April 2019 that is atmospherically corrected. The reason is that there is no proper atmospheric correction library for the Sentinel-2 images in the Google Colaboratory.

The number of training and testing data for the shallow classifiers of the RF, SVM, and the deep learning algorithm of GAN-RF is presented in

Table 2. The number of training and testing data of plastic debris in the pilot site of Mytilene Island, Greece.

Algorithm	Training (Debris/Non-Debris)	Test (Debris/Non-Debris)
RF	15/41	6/19
SVM	15/41	6/19
GANs	29/86	6/19

. It is worth highlighting that as the GAN network generated several synthetic ocean plastic samples, its training data are higher than the RF and SVM classifiers. In other words, the shallow algorithms of the SVM and RF are trained with just actual data (i.e., ocean plastic), while the GAN-RF algorithm is trained by both real and synthetic ocean plastic samples.

2.2. Methods

2.2.1. Sentinel Hub and Google Colaboratory

For the large-scale marine pollution monitoring/detection, we propose the utilization of the integration of Sentinel Hub and Google Colaboratory. Sentinel Hub provides a python package (https://sentinelhub-py.readthedocs.io/en/latest/ (accessed on 14 September 2021)) that can be efficiently imported and implemented in Google Colaboratory (https://colab.research.google.com/notebooks/ (accessed on 14 September 2021)). This Python package provides tools and functionalities required for the downloading and processing of several satellite data, including the Sentinel-2 images. For implementing the machine learning techniques for the detection of marine pollution, the leading notebook of Google Colaboratory was used in this project.

2.2.2. Algorithms

For the detection of marine debris, we evaluated the performance of two well-known shallow algorithms of the SVM, RF, and the deep learning method of GAN-RF implemented in Google Colaboratory. RF is an ensemble tree-based classifier, wherein several decision trees (DTs) are used to reach an optimal global solution [48]. RF is computationally efficient and resistant to the noise in the sample data. In the RF classifier, each DT is built by random vectors sampled independently using the input data. Additionally, a voting procedure is used to find the most popular prediction among various DTs for a given vector data (Figure 3). For instance, majority voting or average values are used to integrate the results of several different DTs to reach an optimal global model [49,50].

Boosting and bagging are two popular ensemble learning algorithms. Boosting can be described as a sequence model where each classifier tries to increase the accuracy of its previous model/classifier in that sequence [51,52]. Bagging (bootstrap aggregation) decreases the error of integrated models while the variance is attempted to be reduced (Figure 4). The bagging technique is considered to be more robust against overfitting over the boosting methods [53]. It is worth noting that the RF classifier integrates the bagging and the random selection methods.

The classification/regression accuracy is improved using RF with the use of bootstrap aggregation [54]. Gini index [55] and information gain ratio criterion [56] are two popular attributes selection techniques. For the RF classifier, the Gini index is usually used. It is worth mentioning that the full-grown trees in the RF classifier are not pruned, which is an advantage compared to other DT algorithms [55].

The SVM is a kernel-based algorithm [57] that is highly used in remote sensing image classification due to its capabilities (e.g., good results with a low number of training data) [58]. The SVM is a non-parametric technique that is insensitive to the input data distribution that is considered one of its most significant advantages. The main goal of SVM is to solve a convex quadratic optimization issue to achieve a globally optimal solution [59,60]. This is noteworthy since we had to address the challenge of a limited number of training data for ocean plastics detection. Therefore, the SVM classifier would be a suitable shallow algorithm in this study.

Currently, deep learning methods, specifically the convolutional neural networks (CNNs), have shown great success in remote sensing image classification, as they learn from representation rather than an empirical feature [61,62,63,64]. Conventional methods, such as RF and SVM, have two separate components of feature extraction and classification [65]. First, the feature extraction procedure transforms the spectral, spatial, and temporal Earth observation (EO) data into feature vectors. Afterward, those extracted features are utilized for training the shallow classifiers (e.g., RF). Therefore, the accuracy of the conventional classifiers depends on the feature engineering part (feature extraction procedure), while internal features are automatically learned in the deep learning methods. As a result, the deep learning algorithms are highly efficient for remote sensing image classification. The disadvantage of the deep learning methods is their dependency on a large number of training samples. This issue can be solved by two approaches of transfer learning and generating new synthetic samples to increase the training data. The deep learning field was revolutionized by the introduction of the new architecture of GAN proposed by Goodfellow et al. [66]. GAN includes two networks of generator and discriminator, as seen in Figure 5.

Using a random noise vector, the generator network generates new synthetic samples, and the discriminator network differentiates between the synthetic samples from the real ones. In other words, the GAN network is trained while the generator creates more realistic samples to fool the discriminator. In contrast, the discriminator tries to recognize the synthetic samples from the real ones [67,68,69]. As we have the challenge of the low number of training data for marine pollution detection, such advanced machine learning technology can be efficiently used to improve marine debris detection. We will evaluate the possibility of using a GAN network to generate synthetic ocean plastics to be used by an RF classifier to improve the marine debris classification accuracy in the next section. To implement the GAN network, we used the PyTorch framework and Sklearn library to implement the RF and SVM classifiers. In our GAN model’s generator, discriminator, and classifier networks, we used a simple structure of six consecutive linear functions and LeakyRelu [70] (Equations (1) and (2)).

$$\mathrm{y}={\mathrm{xA}}^{\mathrm{T}}+\mathrm{b}$$

(1)

$$\mathrm{LeakyRelu}\left(\mathrm{x}\right)=\{\begin{array}{c}\mathrm{x}, \mathrm{if} \mathrm{x}\ge 0\\ \mathrm{negative} \mathrm{slope}, \mathrm{otherwise}\end{array}$$

(2)

We set the number of batch size, noise dimension, hidden layer size, and learning rate to 64, 50, 16, and 0.0005, respectively. Figure 6 presents the architecture of the developed GAN model. First, with the use of a GAN model, synthetic marine pollution samples are generated. Then, the synthetic and real data of plastic marine pollutants are combined and fed to an RF classifier.

2.2.3. Features

As suggested by several previous research for the marine pollution detection, we utilized Sentinel-2 bands of the blue, green, red, red-edge-2, near infrared ($\mathrm{NIR}$), and short wave infrared-1 ($\mathrm{SWIR}1$), as well as the normalized difference vegetation index ($\mathrm{NDVI}$) and the floating debris index ($\mathrm{FDI}$) (Equations (3)–(5)) [71].

$$\mathrm{NDVI}=\frac{\mathrm{NIR}-\mathrm{Red}}{\mathrm{NIR}+\mathrm{Red}}$$

(3)

$$\mathrm{FDI}=\mathrm{NIR}-{\mathrm{NIR}}^{\prime }$$

(4)

$${\mathrm{NIR}}^{\prime }=\mathrm{RE}2+\left(\mathrm{SWIR}1-\mathrm{RE}2\right)\times \left(\frac{{\mathsf{\lambda }}_{\mathrm{NIR}}-{\mathsf{\lambda }}_{\mathrm{Red}}}{{\mathsf{\lambda }}_{\mathrm{SWIR}1}-{\mathsf{\lambda }}_{\mathrm{Red}}}\right)\times 10$$

(5)

where ${\mathsf{\lambda }}_{\mathrm{NIR}}$, ${\mathsf{\lambda }}_{\mathrm{Red}}$, and ${\mathsf{\lambda }}_{\mathrm{SWIR}1}$ are the central wavelength of the bands $\mathrm{NIR}$, $\mathrm{Red}$, and $\mathrm{SWIR}1$, respectively.

2.2.4. Accuracy Assessment

The marine pollution (i.e., ocean plastics) detection results were evaluated in terms of overall accuracy, precision, recall, and F1-score statistical indices (Equations (6)–(9)).

$$\mathrm{Accuracy}=\frac{\left(\mathrm{True}\mathrm{positive}+\mathrm{True}\mathrm{negative}\right)}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{pixels}}\times 100$$

(6)

$$\mathrm{Precision}=\frac{\mathrm{True}\mathrm{positive}}{\left(\mathrm{True}\mathrm{positive}+\mathrm{False}\mathrm{positive}\right)}$$

(7)

$$\mathrm{Recall}=\frac{\mathrm{True}\mathrm{positive}}{\left(\mathrm{True}\mathrm{positive}+\mathrm{False}\mathrm{negative}\right)}$$

(8)

$$\mathrm{F}1-\mathrm{score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(9)

It is worth mentioning that true positive and true negative values are those pixels that are correctly predicted as positive (debris) and negative (non-debris) objects, respectively. The false-positive values are those non-debris objects that are incorrectly classified as debris pixels, while the false negative are those debris objects that are incorrectly classified as the non-debris pixels in the Sentinel-2 imagery.

3. Results

Maps of marine pollution detection based on the RF, SVM, and GAN-RF algorithms with the integration of GAN and RF models in Sentinel Hub are shown in Figure 7.

It should be noted that there were few available training data on ocean plastics for the study area of Mytilene Island, Greece. Based on the results, the algorithms of the RF and SVM achieved an overall accuracy of 88% and 84%, respectively (

Table 3. Performance/confusion matrix based on the validation set data using RF and SVM algorithms.

Algorithm		Plastic	Non-Plastic	OA (%)	F1-Score (%)	Precision	Recall
RF	Plastic	4	2	88	0.73	0.67	0.80
	Non-plastic	1	18		0.92	0.95	0.90
SVM	Plastic	5	1	84	0.71	0.83	0.62
	Non-plastic	3	16		0.89	0.84	0.94
GAN-RF	Plastic	5	1	96	1	0.83	0.91
	Non-plastic	0	19		0.95	1	0.97

). For the ocean plastics detection, the RF classifier obtained precision, F-1 score, and recall values of 0.67, 0.73, and 0.80, respectively. The shallow algorithm of the SVM achieved the recall, F-1 score, and precision values of 0.62, 0.71, and 0.83, respectively, for the ocean debris detection. The GAN-RF classifier obtained the best performance, reaching an overall accuracy of 96%, outperforming both shallow algorithms of the SVM and RF. The GAN-RF classifier’s precision, recall, and F-1 score was about 0.83, 0.91, and 1, respectively. All machine learning algorithms had better performance for the detection of non-plastic pixels. The reason is that there were much higher training data for the non-plastic pixels compared to the marine debris pixels.

The GAN-RF classifier had better performance over the RF and SVM methods, as we generated several synthetic ocean plastic samples with the GAN network and fed the generated synthetic data and the real ocean plastics data to the RF algorithm for the ocean plastic detection.

4. Discussion

Based on our study, one of the most challenging parts of marine debris detection is the lack of a sufficient number of training data that can be used for marine pollution detection with the utilization of satellite images (i.e., Sentinel-2 imagery). Moreover, the spatial resolution of the Sentinel-2 image is equal to 10 m (with resampling methods). As a result, the Sentinel-2 data can be used for the detection/monitoring of large-size AMD (e.g., ocean plastics). It is worth highlighting that with the use of pan-sharpening algorithms, all Sentinel-2 bands (i.e., 20- and 60 m bands) can be resampled to 10 m spatial resolution. Although satellites, such as Planet and RapidEye, have a higher spatial resolution, their spectral bands are limited. The other challenge is the lack of a proper library for the atmospheric correction of the Sentinel-2 images in Python. It should be noted that based on the previous research, the sen2cor Python package is not useful for the atmospheric correction of marine environments. It is suggested to use the sen2cor python package for the urban areas [57], not for the research on the marine environment. It should be noted that atmospheric correction is important, specifically when we are working on time-series satellite data. Errors may arise due to the lack of a proper method of atmospheric correction while working on different temporal resolution satellite images.

It is worth mentioning that in our study, we used Sentinel-2 Level-2 A that is atmospherically corrected and radiometrically calibrated. The reason is that there were no packages for the atmospheric correction available in the Google Colaboratory. Moreover, we only focused on mapping marine debris floating on the surface of the sea. The possibility of the recognition of drowned plastics through optical satellite data can be investigated in future research. On the other hand, based on the achieved results, the GAN-RF with an accuracy of 96% model was superior to both well-known shallow algorithms of RF (88%) and SVM (84%); specifically, the GAN-RF improved the results of marine debris detection of the implemented RF by 16% and 11% in terms of precision and recall statistical indices (Table 3). Therefore, generated synthetic marine pollutions did not increase the error, contradictory they increased obtained accuracy. Results indicate the importance and usefulness of advanced deep learning models such as GAN to solve environmental issues, specifically in study locations with a limited number of ground truth data (e.g., plastic marine pollutions).

5. Conclusions

Marine pollution and climate change factors, such as the sea-level rise and warming water, have become a global threat to the marine environment’s health. There are two fates for the abundant marine pollution, specifically the plastics—floating on the surface of the ocean/sea or sinking. Inevitably, macroplastics will degrade into microplastics, as a result of which the detection of large-size plastics would be significantly valuable for the protection of the marine environment. Therefore, we developed a framework for the detection of marine pollution with the integration of GAN and RF models in Sentinel Hub based on the Sentinel-2 images. Sentinel-2 satellite provides 10-, 20-, and 60 m spatial resolution and 12 spectral bands that can be efficiently utilized to detect large-size marine debris.

To detect ocean plastics in the pilot site of Mytilene Island, Greece, we evaluated the performance of two shallow classifiers of SVM and RF and the deep learning technique of the GAN-RF. The shallow algorithms of RF and SVM achieved F-1 scores of 73% and 71%, respectively, for the detection of plastic marine debris. Both algorithms of the RF and SVM with F-1 scores of 92% and 89%, respectively, had better performance for detecting non-debris pixels than the marine debris detection. The lower F-1 scores for the detection of plastics can be explained by their lower number of training data available in Mytilene Island. By generating several synthetic debris and non-debris samples, the GAN-RF algorithm improved the F-1 score of the RF algorithm by 27% and 3% for the detection of ocean plastic debris and non-plastic debris, respectively. Based on our results, the available tools, and functionality provided by advanced machine learning models, the new technologies of Sentinel Hub and Google Colaboratory can be efficiently used for large-scale marine debris detection at a regional, national, and even global scale.