\documentclass[paper=a4, fontsize=12pt]{article} % A4 paper and 11pt font size
%\usepackage[T1]{fontenc} % Use 8-bit encoding that has 256 glyphs
%\usepackage{times new roman} % Use the Adobe Utopia font for the document - comment this line to return to the LaTeX default
\usepackage[english]{babel} % English language/hyphenation
\usepackage{amsmath,amsfonts,amsthm} % Math packages
\usepackage{amsmath}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{float}
\restylefloat{table}
\usepackage{caption}
\usepackage{booktabs}
%\usepackage{subcaption}
\renewcommand{\thefigure}{S-\arabic{figure}}
\renewcommand{\thetable}{S-\arabic{table}}
\renewcommand{\thealgorithm}{S-\arabic{algorithm}}
\renewcommand{\thesection}{S-\arabic{section}}
%\usepackage{subcaption}
%\captionsetup{compatibility=false}
\usepackage{alphalph}
\usepackage[table,xcdraw]{xcolor}
\usepackage{multirow}
\usepackage{float}
\usepackage{lineno}
\usepackage{color}
\usepackage{color,soul}
\usepackage[framemethod=tikz]{mdframed}
\usepackage{lipsum} % Used for inserting dummy 'Lorem ipsum' text into the template
\usepackage{comment}
\usepackage{setspace}
\usepackage{sectsty} % Allows customizing section commands
\usepackage{cite}
\usepackage{tabularx}
%\allsectionsfont{\centering \normalfont\scshape} % Make all sections centered, the default font and small caps
\usepackage{fancyhdr} % Custom headers and footers
\pagestyle{fancyplain} % Makes all pages in the document conform to the custom headers and footers
\fancyhead{} % No page header - if you want one, create it in the same way as the footers below
\bibliographystyle{unsrt}
\fancyfoot[L]{} % Empty left footer
\fancyfoot[C]{} % Empty center footer
\fancyfoot[R]{\thepage} % Page numbering for right footer
\renewcommand{\headrulewidth}{0pt} % Remove header underlines
\renewcommand{\footrulewidth}{0pt} % Remove footer underlines
%----------------------------------------------------------------------------------------
%   TITLE SECTION
%----------------------------------------------------------------------------------------
\doublespacing
\newcommand{\horrule}[1]{\rule{\linewidth}{#1}} % Create horizontal rule command with 1 argument of height
\captionsetup[figure]{name=Fig.}
\title{ \textbf{A novel semi-supervised approach for semantic segmentation of aerial remote sensing images under limited ground-truth availability}
\normalfont \normalsize
%\textsc{} \\ [25pt] % Your university, school and/or department name(s)
%\horrule{0.5pt} \\[0.4cm] % Thin top horizontal rule
%\huge Problem Formulation and Comprehensive Survey \\ % The assignment title
%\horrule{2pt} \\[0.5cm] % Thick bottom horizontal rule
\date{}
}
\author{Anisha Chakravorty, ~Shounak Chakraborty}  % Your name
\begin{document}
\maketitle % Print the title
\begin{center}
	\section*{Supplementary Material}
\end{center}
\section{Description of the datasets}
Experimentation has been conducted on two very high resolution (VHR) remotely sensed image datasets viz. MiniFrance (MF) and Dense Labeling Remote Sensing Dataset (DLRSD) to validate the efficacy of the proposed semantic segmentation model. MiniFrance (MF) is a diverse dataset featuring high-resolution aerial images and is uniquely designed for semi- supervised learning with labeled and unlabeled images. The land-use maps of these 10000× 10000 pixels coloured images have 16 distinct classes in the form of pastures, industrial and commercial transport units, forests, arable lands, wetlands to name a few. The dataset contains 2,121 labelled images that have been patchified into 12,000 useful labelled images, while the unlabelled images have been patchified into 19,000 useful unlabelled images. Out of the annotated images, 2,000 are chosen for training and 1000 images are earmarked for testing purposes. The proposed approach involved randomly selecting 2,500 images from the unlabeled pool of images. The second dataset, DLRSD is a dense (pixel-wise) labelling dataset
for multi-label remote sensing image retrieval suited for supervised, unsupervised and semi-supervised methods. The 2,100 images of this dataset sized at 256×256 pixels have been annotated with 17 class descriptors in the likes of air-plane, bare-soil, buildings, cars, chaparral court, dock, field, grass, mobile-home, pavement, sand, sea, ship, tanks, trees and water. The dataset has been adapted for experimentation by considering some of the training images as unannotated. Out of these 2,100 image-mask pairs, 210 images are used as test images in the current investigation. The remaining images are split into 890 labelled images and 1,000 unlabelled images.
\section{Details of Implementation}
As already mentioned, the validation of the proposed scheme has been carried out over the MF and DLRSD dataset. The number of annotated training image patches for MF (i.e., n in $T_L$) is chosen as 2000, while 1000 labelled image patches have been used for testing, initially when the availability of all the training images (100\%) is assumed. Later, the proposed approach has also been tested assuming 80\% and 50\% of the training data taking n to be 1600 and 1000, respectively. The number of unlabelled image patches (i.e., m in $T_{UL}$) were randomly chosen to be 2,500. Similarly, for DLRSD dataset, n is 890 and the quantity of unlabelled images (i.e., m) is selected as 1,000; while 210 images are used for testing. The resilience of the proposed methodology has further been ascertained by assuming 80\% (n = 712) and 50\%(n = 445) of the labelled images for training. The dataset in each of the three settings was split in the training and validation ratio of 90:10. The number of training and validating images corresponding to each of the three settings for both the dataset have been tabulated in Table S-6. All the images have been initially resized into 256 × 256 as a preprocessing task.

The I2M model used here is a three layer deep attention Unet model as shown in Fig. \ref{i2m} and is built on two symmetrical contraction and expansion paths.The traditional downsampling stages have been replaced by the ResNet50 model pretrained on ImageNet, where the encoder operations involve a series of three residual blocks with $d$ filters to reduce the spatial resolution from $a\times a$ to $a/8 \times a/8$, while increasing the number of feature maps to $8d$. These residual blocks consist of sequences of convolutional layers incorporating skip connections. In each of the three layers of the upsampling path, a \(3\times3\) transposed convolution operation is performed over the product of activation coefficients and \(2\times2\) upsampled image of the previous lower level.
The attention gate in the network (shown with symbol A in Fig. S-1) operates by receiving the previous decoder layer vector and the respective skip connection vector from the encoder. Both vectors are aligned to the same dimension and then subjected to element-wise summation. Following this, the resulting vector undergoes ReLU activation and a 1x1 convolution. This processed vector is then passed through a sigmoid function to yield attention coefficients, signifying the relevance of features. These attention coefficients are then upsampled via trilinear interpolation to match the original dimensions of the vector from the encoder. Finally, the attention coefficients are element-wise multiplied with the original vector from the encoder before integration into the skip connection pathway. The process is repeated at each layer to finally generate segmented masks of size  $a\times a$ and channels as many as the number of classes $c$.
The deep learning network has been fine-tuned with Adam optimizer at a learning rate of \(1e^{-4}\) for 100 epochs. The loss is calculated with a weighted sum of categorical cross-entropy and dice co-efficient loss function (weight of each fixed at 0.5), while the metrics of intersection over union (IoU) score has been used for training up to 100 epochs.

The M2I block, as shown in Fig. \ref{m2i}, similarly exercises the use of a three layered deep pre-trained attention Unet architecture as the generator. The U-Net model, leveraging a ResNet50 backbone, processes an input image of size (256,256,3) through its network. In the encoder phase, the input undergoes a sequence of operations across four ResNet layers . These operations involve adapting the output channels with convolutional layers, normalizing through batch normalization, and applying Leaky ReLU activation. In each decoder block, spatial resolution is progressively increased using Conv2DTranspose layers with specified filters. The first decoder block employs 512 filters, the second uses 256 filters, and the third utilizes 128 filters. Additionally, the final decoder block ensures compatibility with the 3(RGB) input channels  by employing Conv2DTranspose with 3 filters. Throughout the process, each decoder block concatenates its output with the corresponding encoder output. ReLU activation functions contribute to the refinement of the output image, and these operations collectively result in an output size of (256,256,3). 
The discriminator model, designed to assess the authenticity of image pairs, each with dimensions (256,256,3). When given both the generated and original images, the model, as shown in Fig. \ref{disc}, combines them by concatenation, creating a unified input with dimensions (256,256, 2X3). Following this, a series of convolutional layers processes the concatenated image, progressively capturing features at different scales. Specifically, the convolutional layers include operations with 64, 128, 256, and 512 filters, each employing LeakyReLU activation and batch normalization. The final convolutional layer produces a single-channel output through a sigmoid activation, generating a patch-level prediction regarding the authenticity of the input image pair. Each value in the output corresponds to a patch in the input image pair. The output close to 1 suggests the discriminator perceives the corresponding patch as authentic, while values closer to 0 indicate the patch is likely generated.

The generator is trained to translate images from masks (deviating from conventional generators that generate images from random seeds), while the discriminator feeds in the generated image and the original image to to discern between authentic and synthesized images. The total loss for the consolidated adversarial network is calculated as the weighted sum of generator loss and discriminator loss. This adversarial model has been trained at a learning rate of \(1e^{-4}\) for 1000 epochs.

The validity of the I2M predicted (soft)masks are assessed by the measure of a similarity between the M2I generated reconstructed images (from the (soft)masks) and the original unlabelled images. This similarity, taxonomised as reconstruction score, has been evaluated using Structural Similarity Index (SSIM). Although metrics like PSNR (Peak Signal-to-Noise Ratio) can also be used, the choice of SSIM is backed by its ability of visual perception. While, Feature Similarity Index Measure (FSIM) gives comparable results with SSIM, the preference for SSIM is justified by its less computational complexity. It also is simpler as compared to contrastive learning pipelines like MOCO or SIMCLR, that requires computationally intensive training and careful hyperparameter tuning.
The ‘most-confident’ image-mask pairs are chosen on the basis of  threshold $\tau$, calculated heuristically on strictness measure s (here, s = 0.5) in \(\tau = \mu +s\sigma\), where \(\mu\) and \(\sigma\) are the mean and standard deviation of the obtained reconstruction score. The number of these ‘most-confident’ image-mask pairs (i.e. p, where,\(p \propto 1/\tau \) ), were found to be 411 in MF and 128 in DLRSD. All the experiments were conducted on a Param Shavak system with 96GB DDR4 2200 MHz RAM and an Nvidia RTX P5000 (16GB) graphic card.
 The effectiveness of the proposed scheme is shown by carrying out the experiments considering further reduction in the amount of labelled images to 80\% and 50\%. 

\section{Ablation study}
The ablation analysis, presented in Table \ref{ablation}, analyses the variation in the observations with two different backbone for the I2M component of the proposed approach.
At the onset, the proposed approach was implemented with simple Unet as the backbone for the I2M component followed by the other components as proposed. The model yield an mIoU score of 38.63\% on MF and 45.21\% on DLRSD. In the subsequent phase, the backbone was replaced by an attention Unet, resulting in a substantial improvement in mIoU scores, with values of 57.17\% on MF and 63.46\% on DLRSD. This improvement can be relegated to the fact that the redundant features coming from the initial layers due to poor feature representation were removed.

Analysis was conducted with different variations of segmentation loss functions as shown in Table \ref{ablation1}.Initially, the I2M component underwent training using the categorical loss function, followed by dice loss. Although a marginal enhancement was observed for the MF dataset, the performance metrics for DLRSD remained largely consistent. In the final iteration, a novel approach was adopted, utilizing a weighted sum of dice loss and categorical cross-entropy as the segmentation loss function during the training of the I2M component. This innovative strategy yielded masks with an impressive mIoU score of 57.17\% on MF and 63.46\% on DLRSD. This significant improvement over previous methods can be attributed to the holistic consideration of both multi-class classification and overlapping classes. Optimal weights (alpha and beta) were empirically determined as 0.5 for both components, ensuring a balanced integration of the two loss functions.

The impact of varying the threshold on the proposed method has been analysed by changing the value of parameter ``s", as shown in Table \ref{ablation2}.The optimum value was experimentally chosen as 0.5 for the proposed approach. On increasing the threshold by increasing the value of s, the number of 'most-confident' image-(soft)mask pairs reduced. This strictness in threshold selection results in a marginal dip in the performance for MF, while it remains almost similar for DLRSD. When the strictness in threshold was relaxed by decreasing the the value of s, the number of 'most-confident' image-(soft)mask pairs increased. This led to a significant depreciation in the performance owing to the augmentation of noisy labels.

\begin{table}
    \centering
    \caption{Ablation study for varying backbone}\label{ablation}
    \begin{tabular}{|p{2.5cm}|p{2.5cm}|p{2.5cm}|}
        \hline
        Backbone & mIoU (MF) & mIoU (DLRSD) \\
        \hline
        UNet & {38.64} & 45.21 \\
        \hline
        Attention Unet  & 57.17 & 63.46 \\
        \hline
    \end{tabular}
\end{table}

\begin{table}
    \centering
    \caption{Ablation study for varying segmentation loss}\label{ablation1}
    \begin{tabular}{p{3cm}p{1.5cm}p{0.5cm}p{0.5cm}p{1cm}p{1cm}}
       \hline
        Training set & Test set & CE & Dice & mIoU (MF) & mIoU (DLRSD) \\
       \hline
        Labelled + (m-c) Unlabelled & Unlabelled & $\checkmark$  & $\times$  & 55.02 & 62.87 \\
        \hline
        Labelled + (m-c) Unlabelled & Unlabelled & $\times$  & $\checkmark$  & 55.97 & 62.69 \\
        \hline
        Labelled + (m-c) Unlabelled  & Unlabelled & $\checkmark$  & $\checkmark$  & 57.17 & 63.46 \\
        \hline
    \end{tabular}
\end{table}
\begin{table}
    \centering
    \caption{Ablation study for varying threshold $\tau$}\label{ablation2}
    \begin{tabular}{|p{3cm}|p{2.5cm}|p{2.5cm}|}
        \hline
        Parameter(s) in $\tau$ & mIoU (MF) & mIoU (DLRSD) \\
        \hline
        0 &  54.32 & 62.89\\
        \hline
        0.5  & 57.17 & 63.46\\
        \hline
        1  &  {57.04} & 63.74\\
        \hline
    \end{tabular}
\end{table}
\begin{table}[tbp]
	\caption{List of important symbols}
	\scriptsize\centering
	\begin{tabular}{|p{1.5 cm}|p{6 cm}|l|l|}
		\hline
		\textbf{Symbol}   & \textbf{Significance}\\\hline
		$T_{L}$       & Labelled training set \\
        $n$           & Number of labelled images and masks \\
        $i$           & Values from 1 to n \\
		$I_{i}$       & $i^{th}$ labelled image\\
		$M_{i}$       & Mask for the $i^{th}$ labeled image \\
		$T_{UL}$      & Unlabelled set of images\\
        $m$           & Number of unlabelled images\\
        $j$           & Values from 1 to m \\
		$I^{'}_{j}$   & $j^{th}$ unlabelled image\\
		$M^{'}_{j}$   & Predicted (soft)mask for the $j^{th}$ unlabelled image\\
		${I^{''}_{j}}$   & Image reconstructed from mask $M^{'}_{j}$\\
		$T_{test}$    & Set of images used for testing\\
        $q$           & Number of test images \\
        $l$           & Values from 1 to q \\
		${I^{t}}_{l}$ & $l^{th}$ test image\\
        ${M^{t}}_{l}$ & $l^{th}$ test mask\\
        $\hat{M_{i}}$ & Predicted Mask for $I_{i}$ in training of I2M\\
        $\mathcal{U}$ & Trained I2M model\\
        $\mathcal{G}$ & Generator of M2I model\\
        $\mathcal{D}$ & Discriminator of M2I model\\
        $I^{g}_{i}$         & Image generated for $M_{i}$ by generator in training of M2I\\
        $c$           & Number of LULC classes of the images \\
		$R_j$         & Reconstruction score of ${I^{''}_{j}}$ as compared with $I^{'}_{j}$ \\
		$\tau$        & Threshold to select the `most-confident' image-mask pair depending on $R$ \\
		$T_{mc}$       & Set of `most-confident' image-mask pairs\\
        $p$           & Number of `most-confident' unlabelled images and mask pairs \\
        $k$           & Values from 1 to p \\
		$I^{mc}_{k}$       & $k^{th}$ `most-confident' image\\
		$M^{mc}_{k}$       & $k^{th}$ `most-confident' (soft)Mask\\
		$T_E$   & $T_L\cup {T_mc}$ (Enhanced training set)\\
        ${M^{t'}}_{l}$ & Predicted mask for ${I^{t}}_{l}$ \\
        $a\times a$       & Dimension of the images \\
        $d$               & Number of filters used in convolution \\
		\hline
	\end{tabular}
\end{table}

\begin{table}[]
\caption{The training time / epoch for the approaches}
	\centering
\begin{tabular}{|c|cccc|}
\hline
\multicolumn{1}{|l|}{}                           & \multicolumn{4}{l|}{Training Time / epoch (seconds)}                                                                                                                                                                                                                                                                     \\ \cline{2-5} 
\multicolumn{1}{|l|}{\multirow{-2}{*}{Approach}} & \multicolumn{2}{c|}{\textbf{DLRSD}}                                                                                                                                   & \multicolumn{2}{c|}{\textbf{MF}}                                                                                                                 \\ \hline
\cellcolor[HTML]{FFFFFF}Baseline                 & \multicolumn{2}{c|}{108}                                                                                                                                              & \multicolumn{2}{c|}{117}                                                                                                                         \\ \hline
WNet                                             & \multicolumn{2}{c|}{143}                                                                                                                                              & \multicolumn{2}{c|}{134}                                                                                                                         \\ \hline
CV-GAN                                           & \multicolumn{2}{c|}{214}                                                                                                                                              & \multicolumn{2}{c|}{221}                                                                                                                         \\ \hline
\cellcolor[HTML]{FFFFFF}MGAN                     & \multicolumn{2}{c|}{197}                                                                                                                                              & \multicolumn{2}{c|}{205}                                                                                                                         \\ \hline
\cellcolor[HTML]{FFFFFF}s4GAN                    & \multicolumn{2}{c|}{273}                                                                                                                                              & \multicolumn{2}{c|}{259}                                                                                                                         \\ \hline
\multicolumn{1}{|l|}{}                           & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Segmentation\\ (I2M)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Adversarial \\ (M2I)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Segmentation\\ (I2M)\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Adversarial \\ (M2I)\end{tabular} \\ \cline{2-5} 
\multicolumn{1}{|l|}{\multirow{-2}{*}{Proposed}} & \multicolumn{1}{c|}{126}                                                          & \multicolumn{1}{c|}{207}                                                          & \multicolumn{1}{c|}{132}                                                          & 211                                                          \\ \hline
\end{tabular}
\end{table}

\begin{table}[]
\caption{No.of training and validating images}
	\centering
\begin{tabular}{|l|cccc|}
\hline
\multirow{3}{*}{\% of labelled images} & \multicolumn{4}{c|}{No. of images}                                                                           \\ \cline{2-5} 
                                       & \multicolumn{2}{c|}{DLRSD}                                      & \multicolumn{2}{c|}{MF}                    \\ \cline{2-5} 
                                       & \multicolumn{1}{c|}{Training} & \multicolumn{1}{c|}{Validation} & \multicolumn{1}{c|}{Training} & Validation \\ \hline
\multicolumn{1}{|c|}{100\%}            & \multicolumn{1}{c|}{800}      & \multicolumn{1}{c|}{90}         & \multicolumn{1}{c|}{1800}     & 200        \\ \hline
\multicolumn{1}{|c|}{80\%}             & \multicolumn{1}{c|}{650}      & \multicolumn{1}{c|}{72}         & \multicolumn{1}{c|}{1440}     & 160        \\ \hline
\multicolumn{1}{|c|}{50\%}             & \multicolumn{1}{c|}{400}      & \multicolumn{1}{c|}{45}         & \multicolumn{1}{c|}{900}      & 100        \\ \hline
\end{tabular}
\end{table}


\begin{figure*}[tbp]
\begin{center}
\includegraphics[width=1\textwidth]{i2m.pdf}
\caption{Image-to-mask (I2M) model architecture : The encoder portion of the attention Unet has been replaced by pre-trained residual blocks. The model is trained on the labelled images. Thereafter, on feeding the model with unlabelled images it generates the (soft)masks}
\label{i2m}
\end{center}
\end{figure*}
\begin{figure*}[tbp]
\begin{center}
\includegraphics[width=1\textwidth]{m2i.pdf}
\caption{Mask-to-image (M2I) model architecture: An adversarial model with a sequential flow of (i) a generator (attention Unet) that generates reconstructed images from (soft)masks and (ii) a discriminator (a convolution network) that takes in original and reconstructed images to discern between authentic and synthesised images. While training, the weights are optimized by generator loss, $L_G$ (Equation 5) and   discriminator loss, $L_D$ (Equation 6).}
\label{m2i}
\end{center}
\end{figure*}
\begin{figure*}[tbp]
\begin{center}
\includegraphics[width=1\textwidth]{discriminator.pdf}
\caption{Discriminator architecture used in M2I model}
\label{disc}
\end{center}
\end{figure*}
\begin{figure*}[tbp]
\begin{center}
\includegraphics[width=1\textwidth]{recon.pdf}
\caption{Reconstructed images obtained from M2I over (a) DLRSD and (b) MF. Visualisations show I2M predicted mask, M2I reconstructed images and Original unlabelled image}
\end{center}
\end{figure*}
\begin{figure*}[tbp]
\begin{center}
\includegraphics[width=0.8\textwidth]{res1.drawio.pdf}
\caption{In (a) \& (b), the first row shows the input images; the second row shows the ground-truth. The consecutive rows shows the semantic segmentation masks obtained over (iii)baseline, (iv)wnet, (v)CV-GAN, (vi)MGAN, (vii)s4GAN, (viii) SimSiam and (ix) proposed methodology.}
\end{center}
\end{figure*}
%\bibliography{ref.bib}
%\newpage
%\newpage
%\bibliographystyle{unsrt}
%\bibliography{ref,Thesis_ref}
%\clearpage
\end{document}