A Spectral–Spatial Context-Boosted Network for Semantic Segmentation of Remote Sensing Images

Abstract

Semantic segmentation of remote sensing images (RSIs) is pivotal for numerous applications in urban planning, agricultural monitoring, and environmental conservation. However, traditional approaches have primarily emphasized learning within the spatial domain, which frequently leads to less than optimal discrimination of features. Considering the inherent spectral qualities of RSIs, it is essential to bolster these representations by incorporating the spectral context in conjunction with spatial information to improve discriminative capacity. In this paper, we introduce the spectral–spatial context-boosted network (SSCBNet), an innovative network designed to enhance the accuracy semantic segmentation in RSIs. SSCBNet integrates synergetic attention (SYA) layers and cross-fusion modules (CFMs) to harness both spectral and spatial information, addressing the intrinsic complexities of urban and natural landscapes within RSIs. Extensive experiments on the ISPRS Potsdam and LoveDA datasets reveal that SSCBNet surpasses existing state-of-the-art models, achieving remarkable results in $F_1$-scores, overall accuracy (OA), and mean intersection over union (mIoU). Ablation studies confirm the significant contribution of SYA layers and CFMs to the model’s performance, emphasizing the effectiveness of these components in capturing detailed contextual cues.

1. Introduction

Semantic segmentation of remote sensing images (RSIs) stands as a critical and challenging task within remote sensing applications [1]. This process involves not just delineating object boundaries but also assigning a specific category to each pixel in an image. The complexity escalates in urban scenarios due to the resemblance among varied objects and the diversity within similar ones [2]. Consequently, semantic segmentation technologies hold considerable importance across various societal sectors, including water resources management [3,4], agricultural data analysis [5,6], and military applications [7,8].

Traditional approaches to semantic segmentation have relied on low-level features derived from color, shape, and texture [9,10,11]. Subsequently, these features undergo clustering and classification in a higher-dimensional space for complete image segmentation. The limitations of this methodology are evident: first, model efficacy is largely contingent on manually designed features, leading to constrained model expressiveness; second, the low-level features lack sufficient expressivity for segmentation tasks. Consequently, there is a need to extract more abstract, and potentially hierarchical, features from images for vision-related tasks.

Over the past decade, convolutional neural network (CNN) has achieved significant advancements in remote sensing, as evidenced in several studies [12,13,14,15]. These advancements have led to markedly improved performance in various areas, including scene classification, object detection, change detection, and image fusion. Concurrently, in computer vision, semantic segmentation technology has rapidly evolved. While the results from fully convolutional networks (FCNs) [16] were promising, their overly simplistic decoder design resulted in coarse outcomes. This led to the development of more sophisticated encoder–decoder architectures, enhancing accuracy [17,18,19,20]. Nonetheless, in the context of RSIs, with their intricate backgrounds and large sizes, the reliance on FCN-based methods limits long-range dependency, which is crucial for unconstrained scene image segmentation [21].

RSIs, characterized by high intraclass variability and low interclass variability, present significant labeling challenges compared with standard dataset images. Context utilization has emerged as a viable approach to enhance the discriminative power of learned representations. One method involves integrating multiscale feature encoding through varied-scale dilated convolutional layers or pooling functions [22]. For instance, MLCRNet [23], designed for RSI semantic segmentation, demonstrated impressive performance on the ISPRS Potsdam [24] and Vaihingen [25] benchmarks through multilevel context aggregation. Additionally, Zhang et al. [26] designed a multiscale feature fusion network for land cover and land use classification. Du et al. [27] utilized multiscale aggregation to accurately map urban functional zones from very high resolution images.

Another effective strategy involves the integration of attention modules to capture long-range dependencies [28,29]. The attention mechanism, fundamental in human cognition and survival, allows for selective concentration on specific information aspects while disregarding others, i.e., CBAM [30], DANet [31], and OCRNet [32]. This mechanism has substantially improved segmentation accuracy in networks by focusing on information-rich areas [33,34]. Various attention-based methods have emerged in RSI segmentation. For example, Li et al. [35] introduced dual attention and deep fusion strategies for large-scale satellite RSI segmentation. Other notable developments include HCANet [36], which combines cross-level contextual and attentive representations. HMANet [37] and LANet [38] are also notable for their effective capture of spatial, channel, and category correlations, and for bridging the gap between high-level and low-level features with patch attention modules. Overall, the attention mechanism has proven highly effective in the RSI domain, aiding models in recognizing diverse intraclass variance and subtle interclass variance.

Despite the advancements in processing RSIs and feature learning within the spatial domain, previous methods have often overlooked the spectral characteristics of these images. Image processing fundamentally associates inner body features with low-frequency components and edges with high-frequency components. This concept is visually depicted in Figure 1, where an RSI is transformed to the frequency domain by discrete wavelet transformation (DWT). Specifically, $LL$ displays the low-frequency component, and $HH$ illustrates the high-frequency component. This fact reveals that different components carry different information. A robust model should learn both low-frequency and high-frequency details concerning all spectral bands. Additionally, while self-attention mechanisms are designed to improve object internal consistency through similarity measurements, they apply the same learnable parameters across all frequency components. This approach limits their effectiveness in enhancing both internal consistency and edge delineation between objects. Consequently, the emphasis on effectively leveraging frequency domain features, especially for learning the spectral context in RSIs, is crucial.

Conclusively, this paper introduces the spectral–spatial context-boosted network (SSCBNet) for the semantic segmentation of RSIs. Building upon the encoder–decoder architecture, SSCBNet incorporates cross-fusion modules (CFMs) to seamlessly integrate encoded and decoded features. A pivotal element is the synergetic attention layer, which synergistically models a spectral–spatial context. This layer sequentially applies frequential spectral attention (FSA) and position-wise self-attention (PSA) in lieu of traditional convolutional layers. Notably, FSA emphasizes significant spectral bands by analyzing vertical and horizontal channel contexts. The paper’s main contributions can be summarized as follows:

• We propose a frequential spectral attention (FSA) mechanism, which focuses on the spectral context for feature enhancement. Specifically, FSA initially transforms feature maps to the frequency domain using DWT. Subsequently, the intracomponent spectral context is analyzed through a dual directional spectral attention (DDSA) mechanism, which concurrently learns columnwise and rowwise channel attentions, integrated using a fusion operator. DDSA is then applied to condensed full components to address the intercomponent spectral context, allowing FSA to effectively utilize the spectral property in remote sensing images.
• Our design includes a synergetic attention (SYA) module, which consecutively applies frequential spectral attention (FSA) and position-wise self-attention (PSA) to refine representations. FSA enhances spectral contexts, while PSA further interprets position-wise spatial relationships. This approach enables the input feature maps to be refined, ensuring the preservation of high-frequency details and promoting internal consistency.
• Employing an encoder–decoder structure, we developed a spectral–spatial context-boosted network (SSCBNet) for the semantic segmentation of remote sensing images. The encoder, enhanced with the SYA layer in place of the conventional convolution layer, demonstrates superior capability in learning discriminative features. The novel cross-fusion module (CFM) efficiently connects the encoder and decoder, facilitating feature recovery with minimal loss.
• SSCBNet underwent a comprehensive evaluation on two notable benchmarks, ISPRS Potsdam and LoveDA [39], encompassing both aerial and satellite imagery. The tests highlight SSCBNet’s superior performance compared with various leading models. Additionally, the contributions of SYA and CFM are validated.

The paper is structured as follows: Section 2 provides an overview of related works in the semantic segmentation of RSI and the advanced methods based on frequency analysis. Section 3 introduces SSCBNet within its submodules. Section 4 presents the comparisons and ablation studies. Section 5 draws the conclusion of our work and points out future directions.

2. Related Works

2.1. Semantic Segmentation of Remote Sensing Images

The semantic segmentation of RSIs entails the pixel-level classification of images from satellites or aerial platforms into land cover or use categories [40]. The expansive coverage and limited spatial resolution of RSIs render many neural networks, tailored for specific terrestrial targets, inapplicable. The variability in the spectral, spatial, and temporal characteristics of RSIs introduces uncertainties in semantic labeling. Thus, there is a need for versatile models tailored to the unique aspects and varied conditions of RSIs [41].

State-of-the-art methods in remote sensing semantic segmentation often employ the encoder–decoder architecture, prized for its robust feature extraction and transformation abilities. For instance, Mboga et al. [42] explored the efficacy of deep fully convolutional networks in urban aerial image classification in Goma, Democratic Republic of Congo, comparing their performance with a semiautomatic geographic object-based image analysis (GEOBIA) system. Wang et al. [43] developed an encoder–decoder network integrating inner convolution with directional conditional random fields for postprocessing, demonstrating impressive segmentation results on the Massachusetts road dataset. Liu et al. [44] adapted U-Net for the pixel-level identification and grading of maize drought from aerial imagery. Sun et al. [45] examined the structural limitations and learning deficiencies of encoder–decoder networks in RSI segmentation, proposing ensemble training and inference techniques to mitigate these issues. Tan et al. [46] introduced an end-to-end road segmentation approach, leveraging convolutional layers at different levels to precisely delineate road edges and shapes. A scale-sensitive module in this method employs a weight tensor to assess the significance of fused features. U-Net, a widely explored model for RSIs, is augmented in various adaptations. TreeUNet [47] enhances pixel-level classification accuracy by incorporating a Tree-CNN block with a confusion matrix to integrate multiscale features. ResUNet-a [48], an extension of U-Net, incorporates residual connections, atrous convolutions, pyramid scene parsing, pooling, and multitasking inference, boasting substantial parameters and exceptional accuracy.

Recent advancements have seen the introduction of attention-based deep neural networks for RSI semantic segmentation, leveraging their capacity to model extensive contextual information [49,50]. For example, Li et al. [35] developed a model combining deep layer channel attention and shallow layer spatial attention for large-scale satellite imagery segmentation. Moreover, the SE design is extended to a spatial dimension to focus on patchwise semantics, bridging the semantic gap between high-level and low-level features [51]. Sun et al. [52] combined the attention mechanism with deep multiscale feature extraction to enhance the representation of high- and low-level semantic features. AFNet [53] features a multipath encoder structure, attention-fused modules, and a refinement block, tailored for very high-resolution RSIs. LANet [38] employs a patchwise attention module and a cross-level attention fusion mechanism. Other notable attention-focused models include A2FPN [54], SCAttNet [55], and RAANet [56]. More recently, Li et al. [57] identified that existing attention models, which separately address spatial and channel feature affinities before fusing them, often underperform. In response, they introduced the synergistic attention perception neural network (SAPNet) for RSI segmentation. Additionally, GPINet, guided by geometric priors, utilizes features from CNN and transformer branches to infer pixel semantics with increased accuracy [58].

In essence, effective segmentation hinges on extracting and utilizing informative cues. Traditional methods, primarily confined to the spatial domain, often neglect spectral characteristics, leading to distorted contextual representation, particularly in the spectral domain. Our proposed solution entails a combined modeling of spectral and spatial contexts in an attentive framework. This strategy refines representations to maintain both spectral and spatial integrity, offering valuable cues to enhance discriminative capabilities.

2.2. Learning in the Frequency Domain

Recent years have seen a growing focus on frequency domain learning in image processing, especially in the semantic segmentation of RSIs. This approach excels at differentiating frequency components within images, leading to a finer feature extraction. This precision is especially beneficial in RSIs, characterized by varied textures and patterns. In RSIs, frequency domain features maintain spectral properties with minimal distortion, providing an effective way to model the spectral context.

Xu et al. [59] conducted a theoretical evaluation of neural networks’ spectral bias using Fourier analysis, highlighting their propensity for low-frequency functions and challenges in capturing high-frequency details. This insight spurred subsequent research to emphasize high-frequency elements. Azad et al. [60] enhanced the local feature capacity of transformer models through a redesigned self-attention map, decomposing features into the frequency domain to reveal finer details for enriched contextual cues. Ref. [61] utilized frequency domain features for image classification, while [62] focused on converting spatial-domain CNN models into the frequency domain. Another approach involves bypassing complex conversion processes by selecting frequency channels with SE-Blocks, and a frequency channel attention network was introduced to explicitly process spectral information less apparent in the spatial domain [63]. In addition, [64] introduces a frequency channel attention network. As can be observed, spectral analysis in the frequency domain is beneficial for distinguishing subtle spectral signatures of different land cover classes, enabling the model to differentiate between features with similar spatial but distinct spectral properties.

In processing RSIs, Su et al. [65] developed a complete frequency channel attention network (CFCANet) capable of handling noisy RS images without filtering. CFCANet selectively integrates low-frequency information with feature maps, assigning a discrete cosine transformation frequency component to each original feature map and extracting the most significant eigenvalue per channel. CFCANet shows superior noise resistance by leveraging low-frequency image information. For semantic segmentation, Li et al. [66] introduced the spectrum-space collaborative network (SSCNet), capturing both spectral and spatial dependencies to improve the RSI segmentation quality. SSCNet features a joint spectral–spatial attention module (JSSA) employing both spectral (SpeA) and spatial (SpaA) attention.

In summary, leveraging frequency domain representations for spectral analysis in RSIs is a potent method to harness inherent spectral attributes. This approach not only improves differentiation between land cover classes but also enhances model generalization and facilitates the integration of the spectral context in semantic segmentation. The result is more accurate and robust outcomes in remote sensing applications.

3. Method

3.1. Overview of SSCBNet

As depicted in Figure 2, the spectral–spatial context-boosted network (SSCBNet) adopts the encoder–decoder architecture, utilizing cross-fusion modules (CFMs) to effectively connect encoded and decoded features. Specifically, the network comprises four feature encoder blocks, each containing a convolution block (as shown in Figure 3), a synergetic attention (SYA) layer, and a pooling layer. The final encoder block consists of a convolution block and an SYA layer. During the decoding phase, each of the five blocks is equipped with a $1\times 1$ convolution layer, an SYA layer, and a convolution block. Additionally, four CFMs are strategically placed to integrate features from the preceding decoder block and the corresponding encoder block, thereby minimizing information loss during feature map recovery. The application of a Softmax layer yields the final predictions. This innovative combination of SYA and CFM within SSCBNet redefines the encoder–decoder framework for semantic segmentation in RSIs, ensuring that both spectral and spatial contexts are effectively considered and utilized. The subsequent subsections will provide an in-depth description of these submodules.

3.2. Pipeline of SYA

Figure 4 illustrates how the SYA module in SSCBNet sequentially applies FSA and PSA, using element-wise summation as a connector. In FSA, the spectral context is represented and injected in the frequency domain, optimizing diverse frequency components for detailed spectral information using DDSA. Subsequently, a standard PSA is employed to exploit position-wise correlations, further refining the features. This combination in SYA effectively harnesses a comprehensive spectral–spatial context, significantly enhancing the discriminability of the learned representations.

The mechanism of FSA is detailed in Figure 5. For input features, $F_{in}\left(FSA\right)\in {\mathbb{R}}^{C\times H\times W}$, DWT initially projects them onto the frequency domain, resulting in four components: the low-frequency $LL$, the horizontal $LH$, the vertical $HL$, and the high-frequency $HH$, each with dimensions, ${\mathbb{R}}^{C\times \frac{H}{2}\times \frac{W}{2}}$. These components are individually processed by DDSA, which captures the spectral context and accentuates informative bands within each component.

DDSA’s functionality is further explained in Figure 6. Given the diverse properties of frequency domain features along horizontal and vertical axes, DDSA uses two parallel branches to capture frequential bandwise correlations. One branch analyzes features along the horizontal axis and the other along the vertical axis, resulting in two split features, $F_W\in {\mathbb{R}}^{W\times \left(H\times C\right)}$ and $F_H\in {\mathbb{R}}^{H\times \left(W\times C\right)}$. Matrix multiplication then generates initial attention maps, which are processed with an average operator and a Softmax function to produce refined feature maps. These are combined via element-wise summation to produce the DDSA output, where each spectral band is refined by a bandwise context:

$$A{M}_W=F_W^T\times F_W,$$

(1)

where $F_W^T\in {\mathbb{R}}^{W\times \left(C\times H\right)}$, $F_W\in {\mathbb{R}}^{W\times \left(H\times C\right)}$, and $A{M}_W\in {\mathbb{R}}^{W\times \left(C\times C\right)}$. Notably, $F_W$ is the feature reshaped along the horizontal axe. In other words, $F_W$ can be formed as a combination of W matrices of the size of $H\times C$. Then, the matrix $H\times C$ is transposed. This formula represents W times matrix multiplications between $C\times H$ and $H\times C$. Then, an average operator is applied on $A{M}_W$ with regard to each position, generating an attention map by the following:

$$A{M}_v=\mathrm{Softmax}\left(\mathrm{Average}\left(A{M}_W\right)\right),$$

(2)

where $A{M}_v\in {\mathbb{R}}^{C\times C}$ is the attention map over the vertical axis. Likewise, we split and calculate the attention map $A{M}_h$ as follows:

$$A{M}_h=\mathrm{Softmax}\left(\mathrm{Average}\left(F_H^T\times F_H\right)\right),$$

(3)

where $A{M}_v\in {\mathbb{R}}^{C\times C}$ is the attention map over the horizontal axis. Injecting the attention weights into convolved features, we have two refined feature maps. At last of DDSA, an element-wise summation is deployed:

$$F_{out}\left(DDSA\right)=Con{v}_{1\times 1}\left(F_{in}\left(DDSA\right)\right)+F_{va}+F_{ha},$$

(4)

where $F_{out}\left(DDSA\right)\in {\mathbb{R}}^{C\times H\times W}$ is the output of DDSA, in which each spectral band is refined by the bandwise context.

During the initial phase of the FSA, four components are refined using DDSA modules. To be specific, DDSA-refined features, $F_{A-LL}$, $F_{A-LH}$, $F_{A-HL}$, and $F_{A-HH}$, are all with a size of $C\times \frac{H}{2}\times \frac{W}{2}$. To reduce computational costs while preserving feature details, we compress the four components by a convolution layer (kernel size = $3\times 3$, stride = 1, padding = 1, $ou{t}_{channels}$ = 1). Then the concatenated feature is with a size of $1\times \frac{H}{2}\times \frac{W}{2}$. With the refinement of the intercomponent context by DDSA, another convolution layer (kernel size = $3\times 3$, stride = 1, padding = 1, $ou{t}_{channels}$ = 4C) is applied to expand the channels to $4C$, generating $F_{fsa}\left(ic\right)\in {\mathbb{R}}^{4C\times \frac{H}{2}\times \frac{W}{2}}$.

On the other hand, these DDSA-refined components are fused by a concatenation layer, and individual convolution layers with a kernel size of $3\times 3$ compress each component into a single dimension. This process is crucial for modeling the context between components. The formula for this concatenation is represented as follows:

$$F_{fsa}\left(cc\right)=\mathrm{concat}\left(A-LL, A-LH, A-HL, A-HH\right),$$

(5)

where $F_{fsa}\left(cc\right)$ is defined as a tensor in ${\mathbb{R}}^{4C\times \frac{H}{2}\times \frac{W}{2}}$. Each component, $A-LL$, $A-LH$, $A-HL$, and $A-HH$, undergoes convolution, resulting in a dimensionally reduced form of $1\times \frac{H}{2}\times \frac{W}{2}$. Consequently, a feature map of dimensions $4\times \frac{H}{2}\times \frac{W}{2}$ is obtained through concatenation, where each channel signifies the abstracted features of the corresponding frequency component. Following this, DDSA is applied to model and utilize the intercomponent context to produce $F_{fsa}\left(ic\right)$. Prior to iDWT, we sum $F_{fsa}\left(ic\right)$ with $F_{fsa}\left(cc\right)$. Finally, these reprojected feature maps are combined with the initial FSA input, generating spectral attention-refined features, denoted as $F_{FSA}^{output}\in {\mathbb{R}}^{C\times H\times W}$.

It is noted that PSA in SYA follows the design in DANet [30] (see details in Figure 7). More concretely, given the input features from the previous layer, with a size of $F_{PSA}^{input}\in {\mathbb{R}}^{C\times H\times W}$, PSA refines them with the position-wise context. Initially, input features are projected by three convolution layers. At the top branch, we have $F_1\in {\mathbb{R}}^{C\times N}$ with $N=H\times W$. Similarly, $F_2\in {\mathbb{R}}^{N\times C}$ is generated. By applying matrix multiplication, followed by a Softmax function, the position-wise attention maps are produced with $A_p\in {\mathbb{R}}^{N\times N}$. Then, a matrix multiplication and reshape operation is implemented to produce $F_4\in {\mathbb{R}}^{C\times H\times W}$. At last, an element-wise summation with $F_{PSA}^{input}$ helps us to have the PSA-refined feature maps $F_{PSA}^{output}\in {\mathbb{R}}^{C\times H\times W}$.

Overall, SYA is formulated as follows:

$$F_{out}=F_{in}+\left(F_{FSA}^{output}+F_{in}\right)+F_{PSA}^{output}.$$

(6)

This design first emphasizes intracomponent spectral bands in FSA, followed by modeling the component-wise context. The integration of vanilla PSA and residual connections produces a feature map with a fully realized spectral–spatial context.

3.3. Pipeline of CFM

Enhancing contextual cues in feature extraction and transformation is further augmented by fusing feature maps from corresponding encoder and decoder blocks. This approach has been demonstrated to effectively minimize distortions and maintain the integrity of original expressions. Consequently, this fusion ensures that the recovery path retains contextual information to the greatest extent possible.

Figure 8 illustrates the integration of a straightforward cross-attention mechanism within the cross-fusion module (CFM). In this setup, the decoded feature serves as the query component, while the encoded feature is employed as both the key and value components. The process can be formally expressed as follows:

$$F_{dec}^{in}\left(i\right)=\mathrm{Softmax}\left(F_{enc}\left(i\right)\times F_{dec}\left(i+1\right)\right)\times F_{enc}\left(i\right)+F_{enc}\left(i\right),$$

(7)

where i represents the index of the encoder and decoder block, $F_{enc}\left(i\right)$ denotes the feature map from the corresponding encoder block, and $F_{dec}\left(i+1\right)$ is sourced from the preceding decoder block. Notably, $F_{enc}\left(i\right)$ and $F_{dec}\left(i+1\right)$ are with similar sizes. Let $F_{enc}\left(i\right)$ and $F_{dec}\left(i+1\right)$ be $C\times H\times W$; with matrix multiplication and Softmax function, the contextual correlations between $F_{enc}\left(i\right)$ and $F_{dec}\left(i+1\right)$ are modeled. AttnMap is with a size of $(H\times W)\times (H\times W)$. Then, a matrix multiplication is applied to inject such contextual information stored in AttnMap to form $F_{cf}\in {\mathbb{R}}^{C\times H\times W}$. Finally, a element summation is deployed to generate $F_{dec}^{in}\left(i\right)$.

4. Experiments

4.1. Datasets

4.1.1. ISPRS Potsdam Dataset

The Potsdam dataset comprises 38 tiles of TOP images with very fine spatial resolution (ground sample distance (GSD) of 5 cm), each measuring 6000 × 6000 pixels. It shares the same category information as the Vaihingen dataset. Included in this dataset are four multispectral bands (red, green, blue, and near-infrared) along with digital surface model (DSM) and normalized digital surface model (NDSM) data. For our study, IDs 2_13, 2_14, 3_13, 3_14, 4_13, 4_14, 4_15, 5_13, 5_14, 5_15, 6_13, 6_14, and 6_15, along with 7_13, were designated for testing; ID 2_10 for validation; and the remaining 22 images (excluding image 7_10 due to erroneous annotations) for training purposes. We utilized only the red, green, and blue bands and segmented the original image tiles into patches of 256 × 256 pixels for experimental analysis. A random sample is shown in Figure 9.

4.1.2. LoveDA Dataset

The LoveDA dataset, described in [39], encompasses 5987 fine-resolution optical remote sensing images (GSD of 0.3 m), each with dimensions of 1024 × 1024 pixels. This dataset includes seven land cover categories: building, road, water, barren, forest, agriculture, and background. It allocates 2522 images for training, 1669 for validation, and 1796 officially provided images for testing. The LoveDA dataset presents images from two distinct scenes—urban and rural, collected from three cities in China: Nanjing, Changzhou, and Wuhan. The dataset poses significant challenges due to multiscale objects, complex backgrounds, and inconsistent class distributions in these images. A random sample is shown in Figure 10.

4.2. Implement Details

The experimental framework, as summarized in

Table 1. Experimental settings.

Items	Settings
Learning strategy	Poly decay
Initial learning rate	0.002
Loss function	Cross-entropy
Max epoch	500
GPU memory	48 GB
Optimizer	Adam
Subpatch size	$256\times 256$
Batch size	32

, details the settings employed in this study. Our proposed SSCBNet and other semantic segmentation models were implemented using the PyTorch framework on a Linux operating system, with an NVIDIA A40 GPU (Santa Clara, CA, USA) for computational support. To enhance the model’s robustness, data augmentation techniques such as random flipping and cropping were applied across all datasets and networks. The training protocol involved a fixed initial learning rate of 0.02 and a maximum of 500 epochs. We utilized the Adam optimization algorithm, with a learning strategy defined by polynomial decay and a momentum setting of 0.9. The model parameters yielding the lowest validation loss were preserved for further evaluation.

For comparative analysis, eleven advanced approaches were selected: U-Net [20], DeepLabV3+ [22], DANet [30], ResUNet-a [48], DASSN [35], HCANet [36], RAANet [56], SCAttNet [55], A2-FPN [54], LANet [38], and SAPNet [57]. Notably, U-Net [20], DeepLabV3+ [22], and DANet [30] were initially developed for natural image segmentation, while ResUNet-a [48], DASSN [35], HCANet [36], RAANet [56], SCAttNet [55], A2-FPN [54], LANet [38], MSAFNet [67], CLCFormer [68], and SAPNet [57] represent recent state-of-the-art methodologies specifically designed for RSI segmentation.

4.3. Evaluation Metrics

In this research, we evaluated the performance of our predictions on the test set using standard evaluation metrics. These include the classwise $F_1$ score ($F_1$), the average $F_1$ score across all classes (AF), overall accuracy (OA), and mean intersection over union (mIoU). The $F_1$ score, representing the harmonic mean of precision and recall, assesses the balance between false positives and false negatives. OA quantifies the proportion of correctly classified pixels out of the total number of pixels. mIoU serves as a comprehensive metric for accuracy assessment. The formulations for $F_1$, OA, and IoU are specified in Equations (8), (9), and (10), respectively:

$$F_1=2·\frac{precision·recall}{precision+recall},$$

(8)

$$\mathrm{OA}=\frac{\left(TP+TN\right)}{\left(TP+TN+FP+FN\right)},$$

(9)

$$\mathrm{IoU}=\frac{TP}{\left(TP+FP+FN\right)},$$

(10)

where $precision$ and $recall$ are computed as follows:

$$precision=\frac{TP}{TP+FP},$$

(11)

$$recall=\frac{TP}{TP+FN},$$

(12)

where $TP$, $TN$, $FP$, and $FN$ denote the counts of true positives, true negatives, false positives, and false negatives, respectively. mIoU is calculated as the average IoU across all classes.

4.4. Compared with State-of-the-Art Models

4.4.1. Numerical Evaluation of ISPRS Potsdam

The results presented in

Table 2. Results on the ISPRS Potsdam dataset. Classwise $F_1$-score, AF, OA, and mIoU are listed, where the bold text indicates the best results.

Methods	Impervious Surfaces	Building	Low Vegetation	Tree	Car	AF	OA	mIoU
U-Net	87.51	89.32	73.98	87.03	48.19	77.21	75.83	70.02
DeepLabV3+	84.49	86.13	77.37	77.37	85.47	82.17	80.82	73.59
DANet	86.96	92.04	80.14	79.95	89.36	85.69	83.99	77.13
ResUNet-a	89.16	92.76	86.80	83.02	79.25	86.20	85.90	77.99
DASSN	87.54	92.39	81.62	82.49	83.56	85.52	85.21	77.36
HCANet	92.88	96.90	87.25	88.15	93.88	91.81	90.67	81.92
RAANet	90.40	95.70	87.20	81.80	77.60	86.54	85.21	77.95
SCAttNet	91.87	96.90	85.24	87.05	92.78	90.77	89.06	80.83
A2FPN	90.67	95.69	85.54	85.44	91.55	89.78	88.25	80.67
LANet	92.41	96.38	86.82	87.77	93.44	91.36	90.26	81.19
MSAFNet	91.61	96.63	85.00	86.80	92.51	90.51	88.90	80.68
CLCFormer	92.66	96.64	87.05	88.00	93.69	91.61	89.97	81.68
SAPNet	93.00	96.25	86.49	88.27	94.57	91.72	90.44	81.71
SSCBNet (ours)	93.52	97.23	87.90	88.61	96.42	92.73	91.31	83.09

for the ISPRS Potsdam dataset provide a comprehensive overview of the performance of various semantic segmentation methods, including the proposed SSCBNet. A detailed analysis of these results yields several insightful observations.

First, SSCBNet exhibits superior performance across most metrics when compared with the other evaluated methods. Notably, it achieves the highest scores in terms of impervious surfaces (93.52), building (97.23), and car (96.42) categories, as well as in the OA and mIoU metrics. This indicates that SSCBNet effectively captures and integrates detailed features from the dataset, leading to more accurate segmentation, particularly in distinguishing between complex urban features like buildings and cars. The performance in the low vegetation and tree categories, while not the highest, is still commendably high, suggesting a well-rounded capability in handling various land cover types.

Comparatively, other advanced methods such as HCANet, LANet, and SAPNet also demonstrate robust performance, with HCANet showing particular strength in categories like impervious surfaces and building. However, SSCBNet’s balanced approach across all classes, as evidenced by its high AF score, signifies its comprehensive efficacy in semantic segmentation tasks. The lower performance of models like U-Net and DeepLabV3+ in certain categories, despite their overall good results, underscores the advancements in more recent methods that focus on specific challenges in remote sensing image analysis.

Specifically, when compared with MSAFNet and CLCFormer, two recent and robust models represented in blue, our approach demonstrates superior performance. SSCBNet achieves a remarkable classwise $F_1$-score for ‘impervious surfaces’ at 93.52%, outpacing MSAFNet’s 91.61% and CLCFormer’s 92.66%. Moreover, the OA and mIoU of SSCBNet are 91.31% and 83.09%, respectively, which show significant improvements over MSAFNet’s OA of 88.90% and mIoU of 80.68%, and even CLCFormer’s OA of 89.97% and mIoU of 81.68%.

It is also noteworthy to observe the performance trends in different land cover categories. While most methods perform well in detecting impervious surfaces and buildings, categories such as low vegetation and car tend to have more variation in performance. This could be attributed to the intrinsic challenges associated with these categories, such as the complex textures in vegetation and the smaller size and diverse shapes of cars. The high performance of SSCBNet in these challenging categories highlights the effectiveness of its design in dealing with intricate and varied features in RSIs.

In conclusion, the experimental results demonstrate the effectiveness of SSCBNet in semantic segmentation tasks on the ISPRS Potsdam dataset. Its ability to outperform other state-of-the-art methods across multiple metrics underscores the benefits of its spectral–spatial contextual approach. This analysis not only validates the superiority of SSCBNet but also provides insights into the evolving landscape of semantic segmentation methodologies in the field of remote sensing.

4.4.2. Visual Inspections of ISPRS Potsdam

The visual representations provided in Figure 11 complement the quantitative results from the ISPRS Potsdam dataset discussed earlier. These visualizations enable a qualitative assessment of each method’s performance by comparing the predicted segmentation against the ground truth.

In the first row of images, the segmentation results are juxtaposed with the original infrared–red–green (IRRG) images and the ground truth. It is immediately noticeable that models like SSCBNet (our method), SAPNet, and HCANet demonstrate closer alignment with the ground truth, particularly in complex areas where multiple classes intersect or are in close proximity. The fidelity of SSCBNet’s predictions to the ground truth, especially in delineating the ‘car’ and ‘building’ classes, corresponds with the high $F_1$ scores and mIoU values reported in the table.

In contrast, models like U-Net and DeepLabV3+ show a tendency to misclassify certain regions, as evidenced by the apparent discrepancies between their predicted segments and the ground truth. For example, areas classified as ‘low vegetation’ or ‘tree’ are sometimes incorrectly labeled as ‘building’ or ‘impervious surfaces’, which can be attributed to the challenges of differentiating between similar spectral signatures in these classes.

The middle and bottom rows of images further illustrate the segmentation performance across different urban landscapes. It is evident that the more advanced models, including ours, tend to produce smoother and more coherent segment shapes that are more representative of the actual land cover, whereas simpler models display fragmented or mis-shapen segments.

Overall, the visualizations provide compelling evidence of the advancements in semantic segmentation for RSIs. The superior performance of SSCBNet in maintaining structural integrity and class distinction, as visualized, reaffirms the quantitative results. These visual insights into model performance highlight the importance of considering both spectral and spatial contexts, as effectively performed by SSCBNet, in achieving high accuracy in complex urban environments.

4.4.3. Numerical Evaluation of LoveDA

The results from the LoveDA dataset as presented in

Table 3. Results on the LoveDA dataset. Classwise $F_1$-score, AF, OA, and mIoU are listed, where the bold text indicates the best results.

Methods	Background	Building	Road	Water	Barren	Forest	Agriculture	AF	OA	mIoU
U-Net	50.21	54.74	56.38	77.12	18.09	48.93	66.05	53.07	51.81	47.84
DeepLabV3+	52.29	54.99	57.16	77.96	16.11	48.18	67.79	53.50	52.30	47.62
DANet	54.47	61.02	63.37	79.17	26.63	52.28	70.02	58.14	54.64	50.18
ResUNet-a	59.16	64.08	66.73	81.01	32.23	55.81	75.79	62.12	59.65	54.16
DASSN	57.95	66.90	68.63	76.64	44.35	54.96	70.49	62.85	60.35	55.42
HCANet	66.39	70.76	75.11	88.29	51.14	63.92	81.07	70.95	69.47	62.77
RAANet	55.02	62.19	65.58	81.03	29.25	54.11	74.07	60.18	58.95	53.93
SCAttNet	65.95	71.88	77.04	86.61	50.79	61.19	82.00	70.78	67.31	61.09
A2FPN	65.17	73.32	75.19	88.01	48.82	59.96	79.71	70.03	66.89	61.14
LANet	67.04	74.19	77.54	87.54	52.23	64.78	80.80	72.02	69.11	62.16
MSAFNet	65.51	73.71	75.59	88.47	49.08	60.28	80.13	70.40	67.17	60.76
CLCFormer	67.17	74.34	77.69	87.71	52.34	64.91	80.96	72.16	70.45	62.55
SAPNet	68.27	76.81	78.77	92.20	51.15	62.82	83.51	73.36	71.83	64.90
SSCBNet (ours)	69.45	80.18	82.25	91.86	53.16	65.87	84.48	75.32	72.44	65.92

showcase the performance of various semantic segmentation methods, with a particular emphasis on SSCBNet. SSCBNet outperforms other methods in several key metrics, indicating its robustness and efficacy in RSI segmentation.

SSCBNet demonstrates the highest $F_1$-scores across most land cover categories, notably in ‘building’ and ‘road’, which are critical classes for urban planning and monitoring applications. The increase in $F_1$-score for these categories suggests that SSCBNet is particularly adept at handling urban features, which are often characterized by complex structures and varying spectral signatures. In ‘water’, ‘forest’, and ‘agriculture’ classes, SSCBNet also performs impressively, showing its versatility across diverse natural landscapes. The superior performance in ‘water’ and ‘forest’ is notable, given that these classes can be challenging due to their spectral similarity to other classes. The ‘agriculture’ class often presents difficulties due to the heterogeneous nature of agricultural fields, yet SSCBNet manages to achieve high accuracy, as reflected in the $F_1$-score and mIoU metrics.

OA and mIoU are critical global metrics for evaluating segmentation performance. SSCBNet’s leading scores in both OA and mIoU indicate that it not only correctly classifies a higher proportion of pixels overall but also maintains high precision and recall balance across classes. This balance is essential in applications where misclassification can lead to significant consequences, such as land use change detection and environmental monitoring.

Comparing SSCBNet with other advanced methods like SAPNet, LANet, and HCANet reveals that while these methods also achieve commendable results, SSCBNet’s innovative approach to integrating spectral–spatial contexts gives it an edge. The results suggest that SSCBNet’s architecture effectively captures and integrates detailed spatial and spectral features, leading to improved performance, especially in distinguishing between closely related categories. Notably, SSCBNet’s classwise $F_1$-score for ‘building’ is 80.18%, which is a significant improvement over MSAFNet’s 73.71% and CLCFormer’s 74.34%. Furthermore, the OA and mIoU obtained by SSCBNet are 72.44% and 65.92%, respectively, surpassing the 67.17% OA and 60.76% mIoU of MSAFNet and the 70.45% OA and 62.55% mIoU of CLCFormer.

In summary, SSCBNet establishes new benchmarks on the LoveDA dataset, underscoring the potential of leveraging both spectral and spatial information in semantic segmentation tasks. These results validate the effectiveness of SSCBNet in handling the complex nature of remote sensing data and highlight the importance of continued research in this area to further enhance segmentation accuracy and reliability.

4.4.4. Visual Inspections of LoveDA

The visualizations provided in Figure 12 correspond with the quantitative data from Table 3, offering a qualitative perspective on the performance of various semantic segmentation models on the LoveDA dataset.

At a glance, the visual comparisons between the predicted results and the ground truth for each method reveal significant differences in segmentation accuracy. SSCBNet, identified as ‘Ours’ in the figure, shows remarkable congruence with the ground truth, particularly in delineating complex and mixed land cover classes. The model appears to accurately segment the intricate patterns of agricultural fields and urban structures, which is consistent with the high $F_1$-scores and mIoU reported in the table. This visual alignment with the ground truth underscores the effectiveness of SSCBNet’s approach to integrating spectral and spatial details, leading to superior performance in semantic segmentation tasks.

Other methods such as HCANet, LANet, and SAPNet also demonstrate strong performance, with a notably clear segmentation of larger areas like water bodies and roads. However, they show varying degrees of fragmentation and misclassification in more challenging areas, such as the transition zones between urban and natural environments. For example, in the areas where buildings are adjacent to vegetation, there is a noticeable difference in the precision of the boundaries as delineated by SSCBNet compared with other models.

The lower rows in the figure emphasize the models’ performance in highly heterogeneous scenes. The ability of SSCBNet to maintain consistency in segmentation across different scenes, evident in its visual output, correlates with the robustness indicated by the overall accuracy metric. Models like U-Net and DeepLabV3+ show less precise segmentation in these areas, which may be attributed to the difficulty of segmenting small or narrow features such as roads and waterways amidst varied background classes.

In summary, the visualization of segmentation results provides a clear illustration of the strengths and weaknesses of each model, with SSCBNet displaying a high level of accuracy and detail retention across diverse land cover categories. The visual data support the quantitative findings from the LoveDA dataset, reaffirming SSCBNet’s proficiency in semantic segmentation and highlighting the importance of advanced modeling techniques that can handle the complexity and variability inherent in remote sensing imagery.

4.5. Impacts of SYA

By replacing the SYA layer with a convolution block (see Figure 3), we study the effects of SYA. The ablation study presented in

Table 4. Ablation study of SYA. Results are in form of AF/OA/mIoU.

Models	ISPRS Potsdam	LoveDA
SSCBNet	92.73/91.31/83.09	75.32/72.44/65.92
SSCBNet w/o SYA	86.80/85.47/77.78	69.41/66.75/60.74

provides insightful findings on the impact of the SYA layer within SSCBNet. The study compares the performance metrics of the original SSCBNet against a variant where the SYA layer is replaced with a standard convolution block. The results clearly indicate the significance of the SYA layer in enhancing the network’s performance on both the ISPRS Potsdam and LoveDA datasets.

For the ISPRS Potsdam dataset, the presence of the SYA layer in SSCBNet leads to a notable improvement in AF by approximately 5.93%, OA by 5.84%, and mIoU by 5.31%. Similarly, on the LoveDA dataset, the improvements are 5.91 in AF, 5.69 in OA, and 5.18 in mIoU. These increments underscore the efficacy of the SYA layer in boosting the discriminative capability of the network, leading to more precise segmentation results.

The decrease in performance when the SYA layer is omitted suggests that the ability of SSCBNet to accurately capture and utilize contextual information is significantly diminished. The SYA layer, designed to leverage both spectral and spatial contexts by modeling long-range dependencies and interclass relationships, evidently plays a crucial role in the network’s ability to discern and classify complex features within the images. This is particularly reflected in the mIoU metric, which measures the accuracy of the segmentation by considering both true-positive predictions and the overlap between the predicted and ground truth segments.

Furthermore, the reduction in AF across both datasets when the SYA layer is absent indicates that the model without SYA struggles to maintain a balance between precision and recall, leading to a higher rate of false positives and negatives. This could be particularly detrimental in applications where accurate segmentation is critical, such as land cover mapping for environmental monitoring or urban planning. Additionally, as shown in Figure 13 and Figure 14, we visualize the predictions on two test sets. We zoom in specific areas in images to make a clear comparison to validate the effects of SYA.

In conclusion, the SYA layer is a vital component of SSCBNet, as evidenced by the ablation study. Its inclusion facilitates a significant enhancement in the network’s segmentation accuracy and the ability to handle the diverse challenges presented by different landscapes. These results not only validate the design choices behind SSCBNet but also highlight the importance of attention mechanisms in advancing the field of semantic segmentation for RSIs. The study reinforces the notion that careful architectural considerations can lead to substantial improvements in model performance.

4.6. Impacts of CFM

By removing CFM in SSCBNet, we examine the effects of CFM in our network. The ablation study detailed in

Table 5. Ablation study of CFM. Results are in form of AF/OA/mIoU.

Models	ISPRS Potsdam	LoveDA
SSCBNet	92.73/91.31/83.09	75.32/72.44/65.92
SSCBNet w/o CFM	90.33/88.94/80.93	73.37/70.56/64.21

assesses the impact of CFM on the performance of SSCBNet. CFM is integral to SSCBNet’s architecture, designed to fuse features from encoder and decoder blocks to enhance the network’s representational power. By comparing the standard SSCBNet against a version without CFM, we can discern the contribution of CFM to the model’s overall effectiveness.

The results indicate that CFM plays a significant role in the network’s performance. In the ISPRS Potsdam dataset, the removal of CFM leads to a decrease in AF by 2.40%, OA by 2.37%, and mIoU by 2.16%. A similar trend is observed in the LoveDA dataset, with reductions of 1.95% in AF, 1.88% in OA, and 1.71% in mIoU. These results suggest that CFM is pivotal in enhancing the segmentation accuracy and the model’s ability to generalize across different classes.

Without CFM, SSCBNet experiences a notable drop in its ability to accurately classify pixels, especially in complex urban environments where different classes share similar spectral characteristics. CFM’s role in effectively merging low-level and high-level features is crucial for maintaining the spatial hierarchy of features, which is particularly important for distinguishing fine-grained details in high-resolution imagery. The decrease in mIoU further emphasizes that without CFM, the model’s precision in identifying the overlap between predicted segmentation and the ground truth is compromised.

The impact of CFM removal is also evident in the network’s reduced capacity to handle class imbalances and varied object scales, challenges that are inherently present in remote sensing data. CFM assists in bridging the gap between the detailed, local information captured by the encoder and the broader, contextual information reconstructed by the decoder. This symbiosis is essential for achieving high-quality segmentation that accurately reflects the diverse landscapes within RSIs. Additionally, Figure 13 and Figure 14 visualize the results to examine the effects of CFM.

In conclusion, CFM is a critical component of SSCBNet, enhancing both the quality and the accuracy of the semantic segmentation. The ablation study confirms that CFM substantially contributes to the network’s state-of-the-art performance on both the ISPRS Potsdam and LoveDA datasets. This discussion underscores the importance of feature fusion in deep learning architectures, particularly for complex tasks such as semantic segmentation in remote sensing. The study not only validates the architectural design of SSCBNet but also serves as a testament to the effectiveness of integrating sophisticated modules like CFM for improving model performance.

5. Conclusions

This study introduced SSCBNet, a novel approach for the semantic segmentation of RSIs that effectively integrates both spectral and spatial contexts. Our comprehensive experiments, conducted on the ISPRS Potsdam and LoveDA datasets, demonstrate that SSCBNet outperforms existing state-of-the-art methods across various metrics. The model’s superior performance is particularly noteworthy in urban and agricultural landscapes, where the intricate interplay of spectral signatures poses significant segmentation challenges.

The proposed SSCBNet showcases an advanced capability to maintain the structural integrity of segmented objects and offers high precision in delineating complex urban features. The integration of the SYA layer and CFMs within the SSCBNet architecture has been pivotal in achieving these results. Ablation studies have further confirmed the importance of these components, indicating that their removal leads to a marked decrease in accuracy and model performance.

Looking ahead, the success of SSCBNet opens new avenues for research in the remote sensing domain. There is potential for adapting the network to other challenging scenarios within RSIs, such as those with higher levels of class imbalance or varying object scales. Additionally, future work could explore the integration of SSCBNet with other deep learning architectures or its application to other forms of environmental monitoring and land management tasks. Moreover, this study has centered on the methodological advancements and segmentation accuracy of SSCBNet, setting the stage for future investigations into the model’s computational efficiency, which we plan to rigorously evaluate to balance performance with practical applicability.

As remote sensing technology continues to advance, the need for highly accurate and reliable semantic segmentation models becomes ever more critical. SSCBNet represents a significant step forward in meeting this need, offering a robust framework that leverages deep learning to understand and interpret the complex data our world presents.