%% Submissions for peer-review must enable line-numbering 
%% using the lineno option in the \documentclass command.
%%
%% Camera-ready submissions do not need line numbers, and
%% should have this option removed.
%%
%% Please note that the line numbering option requires
%% version 1.1 or newer of the wlpeerj.cls file, and
%% the corresponding author info requires v1.2


\documentclass{article}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{johd}

\title{A Transformer-Based Ensemble Approach For DNA Splice Junction Classification}


%\maketitle

%\begin{center}
\author{
K. Swaraja$^{a}$, 
Pardhu Thottempudi$^{e}$,
Biswaranjan Acharya$^{f}$
Srilakshmi Aouthu$^{b}$,\\ 
M. Mahitha$^{a}$,
K. Meenakshi$^{a}$, 
Padmalaya Nayak$^{c}$,
H. B. Valiveti$^{a}$, \\
T. R. Vijaya Lakshmi$^{d}$,
Syed Abid Hussain$^{g}$,
Saurav Mallik$^{h,i}$\\ 
\\
\small $^{a}$Department of Electronics \& Communications Engineering, GRIET, Hyderabad, India;\\ \small kswaraja@gmail.com, mahithamudireddy@gmail.com,mkollati@gmail.com,valiveti.bindu@gmail.com \\
\small $^{b}$Department of Electronics \& Communications Engineering, Vasavi College of Engineering, Hyderabad, India ; \\ \small a.srilakshmi@staff.vce.ac.in \\
\small $^{c}$Department of Computer Science \& Engineering, Gokaraju Lailavathi Engineering College, Hyderabad, India ; \\ \small drpadmalaya2010@gmail.com\\
\small $^{d}$Department of Electronics \& Communications Engineering, Mahatma Gandhi Institute of Technology,\\ Hyderabad, India; trvijayalakshmi\_ece@mgit.ac.in\\
\small $^{e}$Faculty of Engineering \& Technology, Department of Electronics \& Communications Egineering,\\ \small Marwadi University, Rajkot, Gujarat, India; pardhu.t@ieee.org\\
\small $^{f}$Department of Computer Engineering - AI \& BDA, Marwadi University, Rajkot, Gujarat, India;\\ \small biswaranjanacharya2020@gmail.com \\
\small $^{g}$ Department of Computer Science and Engineering, Bakhtar University, Kabul, 1001, Afghanistan;\\
\small dr.abid@bakhtar.edu.af, ORCID ID: 0000-0003-3660-5629\\
\small $^{h}$ Department of Environmental Health, Harvard T H Chan School of Public Health,\\ Boston, MA 02115, USA; sauravmtech2@gmail.com\\
\small $^{i}$ Department of Pharmacology \& Toxicology, University of Arizona, Tucson, MA 85721, USA\\
\\
\small $^{*}$Corresponding author: Syed Abid Hussain (dr.abid@bakhtar.edu.af); Pardhu Thottempudi (pardhu.t@ieee.org),\\ Saurav Mallik (sauravmtech2@gmail.com); Biswaranjan Acharya 
(biswaacharya@ieee.org) \\
}
%\end{center}


\date{}
\begin{document}

%\flushbottom
\maketitle

% \keywords{Keyword1, Keyword2, Keyword3}

\begin{abstract}
The precise foretelling of DNA splice junctions is one of the most fundamental problems in computational biology and is also important for understanding the structure and function of the gene. The most complex thing is identifying the type of junction IE, EI or N from the given DNA sequence. Junction prediction is critical for determining the gene expression patterns, splicing regulation, disease etiology and genome architecture. It is very difficult to localize the boundaries for intron removal and exon inclusion, because there is no single rule that governs the process of RNA splicing. To address this problem, the proposed work incorporates an ensemble learning approach in the form of a two-layered hybrid model. In the first layer, feature extraction is done by two transformers DNABERT and DISTILBERT. DNABERT, a transformer model tailored for DNA sequences, is pretrained on genomic data using the masked language model approach, which helps capturing long and short range dependencies, hence improving feature extraction. DISTILBERT's lighter architecture ensures that efficient feature extraction, while maintaining competitive performance. The second layer uses Stacking Ensemble classification model trained on the features fed into it from the first layer. The proposed model splits the splice junction dataset into training and testing sets using cross validation, allowing a thorough analysis of the classifier’s generalization capacity. The ensemble model is trained on various divisions of training set and testing set. The UCI splice junction dataset was used to test our approach. In contrast to other machine learning models for DNA splice junction classification, the proposed DNABERT + Stacking ensemble method achieved an accuracy of 95\%, and the DISTILBERT+ Stacking ensemble method achieved an accuracy of 91\%. These findings specify that DNABERT transformer is better suited for tasks requiring detail contextual understanding, DISTILBERT transformer offers significant computational efficiency, making it a better choice for applications with resource constraints.
\end{abstract}


\thispagestyle{empty}

\section*{Introduction}

DNA, a biological molecule, contributes significantly to the definition of the genetic instructions required for the growth and operation of an organism. It is made up of two nucleotide strands that are held together by complementary base pairs in a double helix configuration. The DNA sequence is assembled from four building blocks, which are the nucleotides Adenine, Guanine, Cytosine, and Thymine abbreviated as A, G, C and T respectfully \citep{zhang1998statistical}. Pre-mRNA is an intermediate molecule transcribed from DNA, it carries single stranded genetic information. Its composition differs from DNA in that it has uracil (U) instead of thymine (T). pre-mRNA consists of coding regions termed exons, and non-coding regions termed introns. To produce the protein coding transcript known as mature mRNA, the intron that lies between two exons is generally removed during RNA splicing. Introns, the non-coding portions of the pre-mRNA, are cut out during RNA splicing, a critical step in mRNA maturation, and replaced with exons to form a finished mRNA molecule. This is the fundamental step for making protein mRNA. Thus, introns which are non-coding sequences of proteins are removed, and exons are combined into a single open reading frame of DNA. The interface between an intron and an exon is called the splice junction and it plays an important function in splicing. This includes 5’ splice site (donor site), regularly featuring the GT (or GU in pre-mRNA) sequence at the exon-intron boundary, and 3’ splice site (acceptor site), nodded by AG at the intron-exon boundary above. A branch point is located within the intron and helps facilitate lariat formation during splicing at the 3’ splice site. The splicing process is a multi-phased procedure. The first phase of splicing occurs where the spliceosome ‘slices’ pre-mRNA and assembles at the splice junction. The spliceosome then cleaves the pre-mRNA at the 5’ splice site to set the exon free and then forms a loop-like structure by binding the 5’ end of the intron to the branch point-adenine. From there the pre-mRNA undergoes further cleaving at the 3’ span site, leading to ligation of the exons and continuance of the sequence code. Ultimately the lariat intron is released and the mature mRNA is ready for translation and export into the cytoplasm. Conserved sequences such as GT at the 5’ donor site, AG at the 3’ acceptor, as well as the ATG start codon and the TAG, TAA, and TGA stop codons \citep{singh2022splice}in exons are critical for splicing. The chance one- gene encodes for many proteins is increased for alternative splicing. The phenomenon can be explained because different combinations of exons lead to different mRNA molecules. GCAT is usually 20-30 nucleotides long and has conserved motifs known as splice junctions, the short pre-mRNA sequences which serve as exon-intron borders for RNA splicing. Splice junctions are often between 10 and 50 per a DNA sequence \citep{srivastava2019biochimica}, depending on the complexity and length of the gene. More complex and longer genes are likely to have more splice junctions due to having more complex splicing needs. The splice junctions are mainly classified into three types:
\begin{itemize}
\item IE boundary: At the IE boundary, the DNA contains an exon which indicates the transition from the intron featuring the AG splice junction dinucleotide at the 3’ end. The beginning is marked by a sequence that is defined as a border together with the exon, which requires specification to guarantee that the exons are accurately joined, so that the introns will be excluded from the splic-ing process. Misunderstanding in identification of this boundary can lead to negative conse-quences in mRNA and protein synthesis. It is a predictive tool for computational models of splice site determination which is one of the main components.
\item EI boundary: A change from coding exon to an intron is marked by an exon boundary. As in the previous boundary, the starting dinucleotide is subdivided into two parts, GT which is replaced by GU in the transcription to RNA. This abbreviation signifies the commencement marker in the boundary, which defines the first phase of the splicing process which includes the removal of in-trons from pre-mRNA. Spliceosome coordination is vital in properly distinguishing this barrier because it plays a significant part towards gene expression.
\item	N boundary: identifies a specific area of a DNA or RNA sequence that does not have a boundary known as a splice-site, which is normally an exon-intron or intron-exon boundary. Lacking such conserved splice site motifs as GT or AG, these sections are non-functional for normal splicing. Those with N boundaries could be non-coding, regulatory, or untranslated.
\end{itemize}
A complete understanding of boundary types (intron-exon, exon-intron, or no splice site) enables accurate gene annotation and splicing analysis. Such understanding allows foreseeing existence of splice sites, mutation analysis related to a number of genetic diseases, and provides insights into alternative splicing research. These findings help understanding various dimensions of gene activity, protein with such a broad range of function, errors in splicing, can lead to diseases and disfunctions. This type of knowledge allows genome annotation and accurate localization of exon-intron borders which is essential in elucidating gene structure. Prediction of splice junctions enables splicing errors detection, alongside mutations related to prolonged conditions like cancer, neurodegenerative disorders, and even hereditary diseases \citep{xing2006alternative}. Focused treatment of the root cause of the conditions enables the development of novel therapeutics such as the use of RNA for spinal muscular atrophy. The study of alternative splicing, which seeks to understand how different splicing patterns are responsible for protein variety and regulation, relies a lot on the prediction of splice junctions. There is great promise in the classification of DNA splice junctions using machine learning algorithms \citep{meher2016prediction},\citep{meher2016computational},\citep{meher2016identification}, but there efficacy is limited by a number of issues. First and foremost, the variations and complexity in splice junctions make it difficult to generalize the algorithms accurately across differing genes and species. If there is high noise in the provided data, such as poor quality genomic sequence annotations, then the model may not be able to figure out distinguishable patterns. Furthermore, conventional machine learning algorithms often struggle with underrepresented data because of the prevailing assumptions and biases. Also, some systems may display better performance on training sequences than on actual sequences due to overfitting to the extensive dimensionality of the genomic data. The reasoning for the predictions made for specific splice junctions is often the most challenging part to understand due to the opacity of certain AI systems \citep{chen2005prediction},\citep{yeo2003maximum}. In order to address the limitations of traditional machine learning approaches to classifying DNA splice junctions, DNABERT and ensemble models were adapted in the proposed framework. DNABERT is an example of a transformer model that works on DNA sequences, it is highly efficient in feature extraction. DNABERT enables us to capture intricate details and dependency structures in genomic data that a conventional model would fail to exploit. Consequently, we are able to analyze large scale genomic data set effectively, which in turn improves the accuracy of the predictions regarding splice junctions. In addition, ensemble learning approaches are beneficial in combining different models in a single framework so that the overall accuracy, robustness, and generalization capabilities of the complete system is improved. This helps to overcome class imbalance and overfitting. Fusing feature extraction transformers with ensemble techniques enhances the accuracy and reliability of the results by addressing the issue of splice junction classification. The rest of this article is structured as follows: The Related works part presents an overview of research focus areas for DNA splice junction classification, with deep learning methods, and a survey of the metaheuristic methods and their approaches to solving the DNA problem. The Methodology describes the hybrid approach that has been developed using the DNABERT transformer.
\\The key contributions of the proposed work are as mentioned below:
\begin{itemize}
\item Using Transformers: By including DNABERT Transformer and DISTILBERT Transformer into the encoder architecture, the proposed work enhances its ability to extract rich features from images.
\item Integration of Ensemble Method: The Stacking ensemble method improves classification perfor-mance by integrating base classifiers and meta classifier, reducing overfitting, and enhancing general-ization. This approach of DNA splice junction classification ensures robust and accurate predictions through the use of feature randomness and bootstrap sampling.
\item Computational Efficiency : By leveraging the lightweight architecture of DISTILBERT , the proposed approach ensures faster computation and reduced resource consumption while maintaining high clas-sification accuracy.\\

\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{Figure 1.png}
\caption{Process of DNA Splicing}
\label{fig:Figure 1}
\end{figure}

The process of gene expression is depicted in Figure ~\ref{fig:Figure 1}, which begins with DNA transcription into pre-mRNA and continues with splicing, which joins exons and removes introns to create mature mRNA. Each exon contributes to the structure and function of the functional protein that is produced from this mRNA. In Section II, related work is discussed.  In Section III, the proposed hybrid approach of DNABERT transformer + ensemble learning and DISTILBERT transformer +ensemble learning for DNA splice junction classification are discussed. In Section IV simulation results and analysis were presented, and Section V concludes the proposed work.

\section*{Related Work}

Section II breaks down existing research on DNA splice junction classification into three main categories. (A) Deep Learning Algorithms for DNA Splice Junction Classification looks into classic deep learning models such as CNNs and RNNs, which are widely used for feature extraction and sequence analysis. (B) Transformers for DNA Junction Classification focuses on transformer-based architectures like DNABERT, which use self-attention mechanisms to improve sequence understanding. (C) Ensemble Methods for DNA Junction Classification looks at methods that combine multiple models, such as stacking and boosting, to improve predictive accuracy and robustness.
\subsection*{A.	Deep learning algorithms for DNA Splice Junction classification}
Deep-learning architectures have shown significant promise in predicting splice junctions. SpliceFinder is proposed in \citep{wang2019splicefinder}, a novel tool for ab initio prediction of splice sites using convolutional neural networks (CNNs). To classify sequences, the method uses a single-layer CNN with a fully connected layer. Human genomic data (GRCh38) is used for training, with iterative dataset reconstruction to reduce class imbalance. SpliceFinder outperforms existing tools like SpliceMachine and SpliceRover, with a classification accuracy of 90.25\%. The REIDS model is introduced in \citep{van2018usage}, which is a novel method for detecting alternative splicing events using exon-exon junction information and Human Transcriptome Arrays (HTA). To identify differential splicing, the model uses a mixed-effects framework that breaks down variance into between-array and within-array components. Junction probes are used to determine whether exons were spliced confidently or not. The REIDS framework outperforms methods such as AltAnalyze and iGEMS, with a validated alternative splicing event recall rate of up to 77\%. ASJA, a program for detecting and annotating splice junctions from RNA-seq data, is introduced in \citep{zhao2019asja}. ASJA combines STAR aligner and StringTie assembler outputs to identify and quantify splice junction expression levels. It outperforms tools such as MapSplice2 and deFuse in terms of linear junction accuracy (91.5\% sensitivity) and fusion detection precision. ASJA is speed-optimized and can identify novel junctions. \citep{lee2020epigenome} introduces ESPRNN, a model for splicing prediction based on epigenomic data and RNN architectures. It is highly accurate (F1 score of 0.8472) and detects critical features such as H3K36me3 for splicing regulation. ESPRNN outperforms traditional methods, revealing the temporal dynamics of chromatin context during transcription.


\begin{table}[h]
    \caption{Recent DNA splice junction classification approaches.}
    \label{tab:1}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row spacing
    \footnotesize % Optional: reduce font size if table is wide
    \begin{tabular}{|p{4cm}|p{2cm}|p{5cm}|p{2cm}|}
        \hline
        \textbf{Reference} & \textbf{Year} & \textbf{Technique} & \textbf{Performance} \\
        \hline
        \cite{van2018usage} & 2018 & Linear mixed models & 65-77\% (accuracy) \\
        \hline
        \cite{wang2019splicefinder} & 2019 & CNN & 90.25\% (accuracy) \\
        \hline
        \cite{zhao2019asja} & 2019 & Assembling Splice Junctions Analysis & 91.5\% (sensitivity) \\
        \hline
        \cite{lee2020epigenome} & 2020 & LSTM and Gated Recurrent Unit & 86\% (F-measure) \\
        \hline
        \cite{zabardast2023automated} & 2023 & CNN & 92\% (accuracy) \\
        \hline
        \cite{ji2021dnabert} & 2021 & DNABERT Transformer & 92.3\% (accuracy) \\
        \hline
        \cite{akpokiro2022ensemblesplice} & 2022 & EnsembleSplice & 94.1\% (accuracy) \\
        \hline
    \end{tabular}
\end{table}

A novel framework is introduced by \citep{zabardast2023automated} for evaluating deep learning models for splice site prediction. This automated framework compares CNN, BLSTM, and BGRU architectures, making it easier to select the best splice site predictors. CNN models outperformed BLSTM and BGRU, scoring 92\% accuracy and 89\% F1 on the HS3D dataset. Furthermore, the framework emphasizes CNN's faster training and better extraction of sequence patterns from genomic data. Table~\ref{tab:1} summarizes recent techniques for DNA splice junction classification.
\subsection*{B.	Transformers for DNA Junction classification}
DNABERT, a Transformer-based pre-trained model for genomic sequences, is described in \citep{ji2021dnabert}. It uses attention mechanisms to capture nucleotide context and achieves the highest splice site prediction accuracy (92.3\%). The Nucleotide Transformer is a transformer-based pre-trained model for genomic sequences that incorporates embeddings and fine-tuning techniques to achieve high performance across 18 genomic tasks,\citep{dalla2024nucleotide} includes a 95\% splice site prediction accuracy and 0.98 precision-recall AUC.  Transformer-based architecture and attention mechanisms for genomic data are discussed in \citep{choi2023transformer}. The models use attention to handle long-range dependencies and improve genome task accuracy, such as RNA-protein binding site predictions, which have an accuracy of 92.5\%.
\subsection*{C.	Ensemble Methods for DNA Junction Classification}
An improvement in DNA splice junction prediction is made by \citep{hamouda2024hybrid} suggesting a hybrid ensemble learning approach that combines Grey Wolf Optimizer (GWO) with Support Vector Machine (SVM) classifiers. The StatLog DNA dataset's features were classified with 96.63\% accuracy using this two-stage method that builds an ensemble model with SVM base classifiers and uses GWO to optimize the feature selection process.EnsembleSplice is an ensemble deep learning model introduced in \citep{akpokiro2022ensemblesplice} to predict splice sites. The model achieves better classification accuracy and robustness by integrating convolutional neural networks (CNNs) and deep neural networks (DNNs) within an ensemble learning framework. With regards to the Homo sapiens dataset, EnsembleSplice produced the best results with the highest accuracy for predicting splice sites as compared to other methods such as DeepSplicer and SpliceFinder, achieving 94.16\% accuracy for acceptor and 95.97\% for donor sites.

\section*{Proposed Methodology}
Classification of DNA splice junctions is one of the main computational biology problems, which involves determining the type of junction (IE, EI or N) associated with a DNA sequence. The problem is addressed by employing a two‐layer hybrid model, which is based on feature extraction and ensemble learning. The first layer applies a DNABERT transformer, which employs pattern dependencies for the features that are often missed by other models, to extract complex patterns and features from the DNA sequences. The extracted features are then fed into second layer which uses Stacking ensemble for the splice junction prediction model.
\subsection*{1.DNABERT model}
The BERT architecture used in natural language processing served as the inspiration for DNABERT \citep{yeo2003maximum}, a transformer-based model created especially for DNA sequence analysis. It introduces a paradigm of pre-training and fine-tuning, which leverages vast amounts of unlabelled data to first generate general-purpose representations, before solving a variety of applications using task-specific data with little architectural change. DNABERT is trained in the same way as BERT. The process begins with tokenizing the DNA sequence into overlapping $k$-mers (e.g., AGC, GCA) to capture biological patterns, with special tokens \texttt{[CLS]} and \texttt{[SEP]} marking the start and end of the sequence. These tokenized sequences are then transformed into embeddings, which include positional embeddings that show the location of each $k$-mer, token embeddings that represent $k$-mer sequences, and their combined input embeddings. The embeddings are sent through 12 transformer layers, each of which uses feed-forward networks to process the data further and multi-head self-attention to capture dependencies between $k$-mers. To stabilize training and increase efficiency, layer normalization is used within each layer. Once the transformer layers are completed, the model produces high-dimensional hidden states that correspond to features that were taken from the DNA sequence. These extracted features are used as input to a stacking ensemble classifier for splice junction classification.

\subsubsection*{1.1 Multi-Head Self-Attention}
The key component of DNABERT’s architecture is the \textbf{multi-head self-attention} mechanism, which allows the model to capture contextual relationships between tokens (in this case, $k$-mers) across the entire sequence. The input to the attention mechanism is a matrix $M \in \mathbb{R}^{n \times d_{model}}$, where $n$ is the number of $k$-mers in the sequence, and $d_{model}$ is the embedding dimension.

The self-attention mechanism computes three linear projections of $M$, mapping it into \textit{queries} ($Q$), \textit{keys} ($K$), and \textit{values} ($V$), as shown in Equation~\ref{eq:QKV}:
\begin{equation}
Q = M W^Q, \quad K = M W^K, \quad V = M W^V
\label{eq:QKV}
\end{equation}
where $W^Q$, $W^K$, and $W^V$ are learnable weight matrices of size $d_{model} \times d_k$. These matrices project the input embeddings into different subspaces that represent the query, key, and value vectors, respectively.

For each token, the attention score with respect to every other token is computed using the \textbf{scaled dot-product attention} mechanism. The similarity between $Q$ and $K$ is calculated, scaled by the square root of the key dimension $d_k$, and then passed through a softmax function to obtain attention weights. The weighted sum of the values ($V$) yields the output of the attention layer. This computation is defined in Equation~\ref{eq:attention}:
\begin{equation}
\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{Q K^\top}{\sqrt{d_k}} \right) V
\label{eq:attention}
\end{equation}

The division by $\sqrt{d_k}$ in Equation~\ref{eq:attention} prevents the dot product from becoming too large, ensuring that the softmax function operates in a numerically stable regime, which in turn stabilizes the gradients during training.

In \textbf{multi-head attention}, $h$ parallel attention layers (heads) are applied to allow the model to jointly attend to information from different representation subspaces at different positions. Each head computes its own self-attention operation with separate learnable projections $W_i^Q$, $W_i^K$, and $W_i^V$, as shown in Equation~\ref{eq:head}:
\begin{equation}
\text{head}_i = \text{softmax} \left( \frac{M W_i^Q (M W_i^K)^T}{\sqrt{d_k}} \right) M W_i^V
\label{eq:head}
\end{equation}

The outputs of all $h$ heads are then concatenated and projected back into a single vector space of dimension $d_{model}$ using a learnable matrix $W^O$. This process is described in Equation~\ref{eq:multihead}:
\begin{equation}
\text{MultiHead}(M) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^O
\label{eq:multihead}
\end{equation}

The multi-head self-attention mechanism defined by Equations~\ref{eq:head} and \ref{eq:multihead} enables DNABERT to capture diverse and complex relationships between $k$-mers in the DNA sequence across multiple representation subspaces simultaneously.


\subsubsection*{1.2.Training of the DNABERT model}
DNABERT follows a two-stage training process: pre-training and fine-tuning. In the pre-training phase, DNABERT is trained in a self-supervised fashion on unlabeled DNA sequences, typically sampled from the human genome. During this phase, 15\% of the $k$-mers in each sequence are randomly masked. The model then attempts to predict these masked tokens based on their surrounding context. This process allows DNABERT to learn the statistical dependencies and biological patterns inherent in genomic sequences without requiring labeled data.

The pre-training objective uses a categorical cross-entropy loss function, which measures the discrepancy between the true and predicted distributions of masked tokens. The loss function is defined as:
\begin{equation}
L_{pre} = - \sum_{i=1}^{N} \sum_{j=1}^{|V|} y_{ij} \log \left( \hat{y}_{ij} \right)
\label{eq:pretrain_loss}
\end{equation}
where $N$ is the number of masked tokens in the batch, $|V|$ is the vocabulary size (i.e., the number of possible $k$-mers), $y_{ij}$ is the true one-hot encoded label for token $i$, and $\hat{y}_{ij}$ is the predicted probability for class $j$.

Following pre-training, DNABERT is fine-tuned on labeled datasets for specific tasks, such as splice site prediction. During fine-tuning, masking is disabled, and the tokenized sequence, including the special \texttt{[CLS]} and \texttt{[SEP]} tokens, is fed directly into the model. The output hidden state corresponding to the \texttt{[CLS]} token, denoted as $h_{[CLS]}$, is used as a representation for classification tasks.

For binary classification tasks, the prediction probability that the label $y$ is equal to $1$ given $h_{[CLS]}$ is computed using a sigmoid activation function as:
\begin{equation}
P(y=1 \mid h_{[CLS]}) = \sigma \left( W h_{[CLS]} + b \right)
\label{eq:binary_prediction}
\end{equation}
where $W \in \mathbb{R}^{1 \times d_{model}}$ is the classifier’s weight vector, $b$ is the bias term, and $\sigma(\cdot)$ is the sigmoid activation function.

The fine-tuning objective is the binary cross-entropy loss, which is minimized to train the model. It is defined as:
\begin{equation}
L_{fine} = - \frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log \left( \hat{y}_i \right) + \left( 1 - y_i \right) \log \left( 1 - \hat{y}_i \right) \right]
\label{eq:finetune_loss}
\end{equation}
where $y_i$ is the ground truth label, and $\hat{y}_i$ is the predicted probability for the $i$-th sequence.

DNABERT can handle sequences of up to 512 tokens. For sequences longer than this limit, DNABERT-XL segments them into smaller sub-sequences, processes them independently, and aggregates their representations to perform classification on longer DNA sequences.

As shown in Equation~\ref{eq:pretrain_loss}, the pre-training loss ensures that DNABERT learns to predict masked tokens effectively from context. During fine-tuning, Equation~\ref{eq:binary_prediction} describes the classification probability computation, while Equation~\ref{eq:finetune_loss} defines the objective function optimized for classification tasks.



\subsubsection*{1.2.1 Tokenization}

Tokenization is the process of splitting a DNA sequence into overlapping $k$-mers. Given a DNA sequence of length $L$, the total number of $k$-mers generated from the sequence can be computed as:
\begin{equation}
N_{kmer} = L - k + 1
\label{eq:kmer_count}
\end{equation}

As shown in Equation~\ref{eq:kmer_count}, for a sequence of length $L$, the number of $k$-mers depends linearly on $L$ and the value of $k$. For example, if $k=3$ and the DNA sequence is ``GCTAGC'', then the resulting number of $k$-mers is:
\[
N_{kmer} = 6 - 3 + 1 = 4
\]
which produces the $k$-mers: \texttt{GCT}, \texttt{CTA}, \texttt{TAG}, and \texttt{AGC}.

Each $k$-mer is mapped to a token ID and then looked up in an embedding table, which stores dense vector representations for every possible $k$-mer. This embedding enables DNABERT to capture contextual information about the sequences.

Special tokens are also introduced to facilitate model operations:
\begin{itemize}
    \item \texttt{[CLS]}: Used to represent the entire sequence during classification tasks. The embedding corresponding to this token is typically used for sequence-level predictions.
    \item \texttt{[SEP]}: Separates sequences or segments when needed, particularly in tasks that involve pairs of sequences.
    \item \texttt{[MASK]}: Used during pre-training for masked language modeling (MLM) to hide specific $k$-mers that the model must predict.
    \item \texttt{[PAD]}: Used to pad sequences to a uniform length in batch processing.
    \item \texttt{[UNK]}: Represents unknown tokens or $k$-mers that are not part of the vocabulary.
\end{itemize}

The model’s vocabulary size depends on the choice of $k$. Since each position in a $k$-mer can be occupied by one of the four nucleotides (A, T, G, or C), the total number of unique $k$-mers in the vocabulary is:
\begin{equation}
V_{size} = 4^k
\label{eq:vocab_size}
\end{equation}

For example, with $k=6$, the vocabulary size becomes:
\[
V_{size} = 4^6 = 4096
\]
As shown in Equation~\ref{eq:vocab_size}, the vocabulary grows exponentially with the length of $k$-mers, which allows the model to capture increasingly complex patterns at higher values of $k$.


   

\begin{figure}[ph]
\centering
\includegraphics[width=0.7\linewidth]{figure 2.jpg }
\caption{DNABERT Transformer + Stacking Ensemble}
\label{fig:figure 2}
\end{figure}

Figure ~\ref{fig:figure 2} illustrates transformer-based ensemble approach for DNA sequence processing. DNA sequences are first tokenized into k-mers, and certain k-mers are replaced with [MASK] tokens as part of a self-supervised learning strategy. This allows the model to learn meaningful patterns in DNA sequences even without labelled data. The tokenized sequences are then embedded and processed through multiple transformer layers, which use self-attention and feed-forward networks to generate contextual embeddings. The transformer's final hidden state is fed into a Stacking ensemble classifier, which determines the final class using majority voting.

\subsubsection*{1.2.2 Pre-training}

During pre-training, DNABERT processes sequences of varying lengths (typically 5 to 510 tokens) sampled from the human genome. Initially, 15\% of the $k$-mers in each sequence are randomly masked, and later this proportion is increased to 20\%. DNABERT is pre-trained for a total of 120{,}000 steps, using a batch size of 2000 and the Adam optimizer. The learning rate follows a two-stage schedule: it increases linearly from $0$ to $4 \times 10^{-4}$ over the first 10{,}000 steps, which serves as the warm-up phase, and then decays linearly for the remaining steps.

The pre-training objective is the \textit{masked language modeling} (MLM) task, where DNABERT learns to predict masked $k$-mers based on their surrounding context. This objective enables the model to capture the statistical dependencies and patterns inherent in genomic sequences. Specifically, the model is trained to minimize the categorical cross-entropy loss, which quantifies the difference between the predicted probability distribution and the actual masked token labels.

The pre-training loss function, referred to in Equation~\ref{eq:pretraining_loss}, is given by:
\begin{equation}
L_{pre} = - \sum_{i=1}^{N} \sum_{j=1}^{|V|} y_{ij} \log(\hat{y}_{ij})
\label{eq:pretraining_loss}
\end{equation}

Here, $N$ represents the number of masked tokens in the batch, $|V|$ is the size of the vocabulary (i.e., the number of unique $k$-mers), $y_{ij}$ is a one-hot encoded label that indicates the correct class for token $i$, and $\hat{y}_{ij}$ is the model's predicted probability for class $j$. As described in Equation~\ref{eq:pretraining_loss}, minimizing this loss encourages DNABERT to assign higher probabilities to the correct masked $k$-mers, thereby improving its ability to understand DNA sequence structures and relationships.

The pre-training phase utilizes masked language modeling in a self-supervised fashion, without requiring labeled data. By effectively predicting masked tokens, DNABERT builds rich contextual representations of DNA sequences that are transferable to downstream tasks during fine-tuning.

\subsubsection*{1.2.3 Fine-Tuning}

In fine-tuning, DNABERT adapts to specific tasks such as splice site classification. Using the pre-trained weights as initialization, DNABERT is further trained on labeled datasets relevant to the specific downstream task. During this phase, the masking step used in pre-training is disabled. The entire tokenized sequence, including the special tokens \texttt{[CLS]} and \texttt{[SEP]}, is passed through the transformer model to produce hidden states for each token.

The hidden state corresponding to the \texttt{[CLS]} token, denoted as $h_{[CLS]} \in \mathbb{R}^{d_{model}}$, is typically used as the aggregate representation of the entire sequence. This representation is passed through a dense layer followed by a sigmoid activation function to obtain the probability that the sequence belongs to the positive class. The output probability $P(y=1|h_{[CLS]})$ is computed using the following equation:
\begin{equation}
P(y=1|h_{[CLS]}) = \sigma(W h_{[CLS]} + b)
\label{eq:classification_probability}
\end{equation}
where $W \in \mathbb{R}^{1 \times d_{model}}$ is the weight vector of the classification layer, $b$ is the bias term, and $\sigma(\cdot)$ represents the sigmoid activation function defined as:
\[
\sigma(z) = \frac{1}{1 + e^{-z}}
\]

The model is optimized using the binary cross-entropy loss function, which measures the difference between the true labels $y_i$ and the predicted probabilities $\hat{y}_i = P(y=1|h_{[CLS]})$ for each training example $i$. The binary cross-entropy loss $L_{fine}$ is given by:
\begin{equation}
L_{fine} = - \frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]
\label{eq:fine_tuning_loss}
\end{equation}
where $N$ is the number of training samples, $y_i \in \{0, 1\}$ is the ground-truth label for the $i$-th sample, and $\hat{y}_i$ is the model’s predicted probability for the same sample.

DNABERT-XL extends this framework to accommodate sequences longer than 512 tokens. This is achieved by splitting longer sequences into smaller segments that fit within the model’s maximum input size. Each segment is processed independently, and their contextual representations are aggregated or concatenated to generate a unified feature representation for classification.


\subsection*{1.3 Classification Training Layer}

Once features are extracted from DNABERT, they are passed into a stacking ensemble classifier \citep{breiman2001random} to improve classification performance on the splice junction detection task. Stacking combines the predictions of multiple base classifiers to build a robust predictive model by learning how to optimally combine their outputs.

\subsubsection*{1.3.1 Base Classifiers}

The base classifiers, also referred to as Level-0 models, include the Random Forest Classifier (RFC) and the Gradient Boosting Classifier (GBC).

\paragraph{Random Forest Classifier (RFC):}

The RFC is an ensemble learning algorithm that constructs a multitude of decision trees during training and outputs the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. In this study, the RFC consists of $T$ decision trees, each trained on a bootstrap sample of the training data.

The ensemble prediction of the RFC is calculated by averaging the predictions of all the individual trees, as shown in Equation~\ref{eq:rfc}:
\begin{equation}
P_{RFC}(x) = \frac{1}{T} \sum_{t=1}^{T} h_t(x)
\label{eq:rfc}
\end{equation}

Here, $h_t(x)$ represents the prediction of the $t$-th decision tree for the input feature vector $x$. The number of trees in the ensemble, denoted by $T$, is controlled by the hyperparameter \texttt{n\_estimators}, which is set to $100$ in this configuration. The \texttt{random\_state} is fixed at $42$ to ensure reproducibility.

\paragraph{Gradient Boosting Classifier (GBC):}

The GBC is another ensemble technique that builds an additive model in a forward stage-wise fashion. It combines weak learners (usually decision trees) into a strong learner by sequentially adding new models that correct the errors made by the previous ones.

At each iteration $m$, a new weak learner $h_m(x)$ is trained on the negative gradient of the loss function of the current ensemble model. The final prediction of the GBC after $M$ boosting iterations is computed as shown in Equation~\ref{eq:gbc}:
\begin{equation}
F_{GBC}(x) = \sum_{m=1}^{M} \gamma_m h_m(x)
\label{eq:gbc}
\end{equation}

In Equation~\ref{eq:gbc}, $\gamma_m$ represents the learning rate (step size) that controls the contribution of each weak learner to the final prediction. The number of boosting iterations, denoted by $M$, is governed by the hyperparameter \texttt{n\_estimators}, set to $100$ in this study. As with the RFC, \texttt{random\_state} is set to $42$ to ensure the reproducibility of results.

\paragraph{Hyperparameter Tuning:}

The performance of both the RFC and GBC depends significantly on their hyperparameters. In this study:
\begin{itemize}
    \item \texttt{n\_estimators} $=100$ is used to ensure the ensemble contains enough trees to capture the complex patterns in the data.
    \item \texttt{random\_state} $=42$ is fixed to ensure consistent results across different runs.
\end{itemize}

The base classifiers serve as the foundation for the stacking ensemble, whose outputs are subsequently combined by a meta-classifier, further described in Section~1.3.2.


\subsubsection*{1.3.2 Meta Classifier}

The predictions from the Random Forest Classifier (RFC) and the Gradient Boosting Classifier (GBC) are concatenated and passed to the meta-classifier, which is implemented as a Logistic Regression model \citep{zou2019logistic}. The Logistic Regression meta-classifier computes the final probability of the positive class as shown in Equation~\ref{eq:meta_classifier}:
\begin{equation}
P(y = 1 \mid x) = \sigma \left( w_1 P_{RFC}(x) + w_2 F_{GBC}(x) + b \right)
\label{eq:meta_classifier}
\end{equation}
where $w_1$ and $w_2$ are the weights learned during training, $P_{RFC}(x)$ is the prediction probability from the Random Forest Classifier, $F_{GBC}(x)$ is the prediction from the Gradient Boosting Classifier, and $b$ is the bias term. The $\sigma(\cdot)$ represents the sigmoid activation function, which ensures the output probability lies between 0 and 1.

For a more generalized case with $n$ base classifiers, the prediction from the meta-classifier can be expressed as shown in Equation~\ref{eq:meta_classifier_general}:
\begin{equation}
y = \sigma \left( w_1 h_1(x) + w_2 h_2(x) + \dots + w_n h_n(x) + b \right)
\label{eq:meta_classifier_general}
\end{equation}
where $h_i(x)$ is the output from the $i$-th base classifier and $w_i$ is the corresponding weight assigned by the meta-classifier.

As shown in Equation~\ref{eq:meta_classifier_general}, the meta-classifier learns to combine the strengths of multiple base classifiers by assigning optimal weights to their predictions, resulting in a robust ensemble prediction.

\subsubsection*{1.3.3 Cross-Validation in Stacking}

The stacking classifier is trained using 5-fold cross-validation ($cv=5$). The training data is divided into five equal subsets or folds. In each iteration of cross-validation, four subsets are used to train the base classifiers (RFC and GBC), while the remaining subset is used to validate their predictions. The out-of-sample predictions from the validation fold are then used to train the meta-classifier.

This process ensures that the meta-classifier is trained on predictions made by the base classifiers on unseen data, thus reducing the risk of overfitting. By leveraging 5-fold cross-validation, each data point in the training set is used once for validation and four times for training. This strategy results in a more reliable and generalizable meta-classifier, particularly beneficial for small or imbalanced datasets, as described by \citep{allgaier2024cross}.

The cross-validation approach contributes to the stacking ensemble’s ability to generalize well on new, unseen data, as the meta-classifier is trained with unbiased inputs derived from base classifiers evaluated on out-of-sample data.


 \subsection*{2.DISTILBERT model}
The proposed pipeline for DNA sequence classification begins with the tokenization of raw DNA sequences. Each input sequence
\begin{equation}
    S = \left( s_1, s_2, \dots, s_N \right)
    \label{eq:input_sequence}
\end{equation}
where $s_i$ represents a nucleotide in the DNA alphabet $\{A, C, G, T\}$, is segmented into overlapping $k$-mers, resulting in a sequence of tokens
\begin{equation}
    T = \left( t_1, t_2, \dots, t_M \right)
    \label{eq:token_sequence}
\end{equation}
where $M = N - k + 1$. To provide structural information to the model, special tokens \texttt{[CLS]} and \texttt{[SEP]} are appended at the beginning and end of the token sequence, respectively, forming
\begin{equation}
    T' = \left( \texttt{[CLS]}, t_1, t_2, \dots, t_M, \texttt{[SEP]} \right)
    \label{eq:special_tokens}
\end{equation}

Each token $t_j$ in $T'$ is transformed into a dense vector representation $\mathbf{x}_j \in \mathbb{R}^d$ through an embedding function $E(t_j)$, learned as part of the DISTILBERT model \citep{sanh2019distilbert}, as shown in
\begin{equation}
    \mathbf{x}_j = E(t_j)
    \label{eq:embedding}
\end{equation}
The entire sequence of embeddings is represented as
\begin{equation}
    \mathbf{X} = \left[ \mathbf{x}_{\texttt{[CLS]}}, \mathbf{x}_1, \dots, \mathbf{x}_M, \mathbf{x}_{\texttt{[SEP]}} \right] \in \mathbb{R}^{(M+2) \times d}
    \label{eq:embedded_sequence}
\end{equation}

These embeddings are input to DISTILBERT’s transformer layers, which capture both local and long-range dependencies using a combination of Multi-Head Self-Attention (MHSA) and Feed-Forward Neural Networks (FFN). In each attention head $h$, the attention scores are computed by projecting $\mathbf{X}$ into query, key, and value spaces through learned weight matrices $\mathbf{W}_h^Q$, $\mathbf{W}_h^K$, and $\mathbf{W}_h^V$, resulting in
\begin{align}
    \mathbf{Q} &= \mathbf{X} \mathbf{W}_h^Q \label{eq:query} \\
    \mathbf{K} &= \mathbf{X} \mathbf{W}_h^K \label{eq:key} \\
    \mathbf{V} &= \mathbf{X} \mathbf{W}_h^V \label{eq:value}
\end{align}

The scaled dot-product attention is then computed according to
\begin{equation}
    \text{Attention}_h(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax} \left( \frac{\mathbf{Q} \mathbf{K}^\top}{\sqrt{d_k}} \right) \mathbf{V}
    \label{eq:attention}
\end{equation}

The outputs from all $H$ heads are concatenated and projected back into the model dimension using $\mathbf{W}^O$, yielding the MHSA output described by
\begin{equation}
    \text{MHSA}(\mathbf{X}) = \text{Concat} \left( \text{Attention}_1, \dots, \text{Attention}_H \right) \mathbf{W}^O
    \label{eq:mhsa}
\end{equation}

Each token’s representation is then passed through a feed-forward network (FFN), as defined in
\begin{equation}
    \text{FFN}(\mathbf{z}) = \text{ReLU} \left( \mathbf{z} \mathbf{W}_1 + \mathbf{b}_1 \right) \mathbf{W}_2 + \mathbf{b}_2
    \label{eq:ffn}
\end{equation}

Layer normalization and residual connections are applied after both the MHSA and FFN sublayers, as shown in
\begin{align}
    \mathbf{H}^{(l)} &= \text{LayerNorm} \left( \text{MHSA}(\mathbf{H}^{(l-1)}) + \mathbf{H}^{(l-1)} \right)
    \label{eq:mhsa_norm} \\
    \mathbf{H}^{(l)} &= \text{LayerNorm} \left( \text{FFN}(\mathbf{H}^{(l)}) + \mathbf{H}^{(l)} \right)
    \label{eq:ffn_norm}
\end{align}

After processing through $L$ transformer layers, the final output representation is denoted as
\begin{equation}
    \mathbf{H} = \mathbf{H}^{(L)} \in \mathbb{R}^{(M+2) \times d}
    \label{eq:final_output}
\end{equation}

For classification, the contextualized representation corresponding to the \texttt{[CLS]} token is extracted as the aggregate feature vector for the entire sequence:
\begin{equation}
    \mathbf{h}_{\texttt{[CLS]}} = \mathbf{H}_{0}
    \label{eq:cls_feature}
\end{equation}

This feature vector is then input to a stacking ensemble classifier. Each base classifier $f_i$, where $i = 1, 2, \dots, N$, predicts a class probability vector
\begin{equation}
    y_i = f_i(\mathbf{h}_{\texttt{[CLS]}})
    \label{eq:base_classifier}
\end{equation}

These outputs are concatenated into a meta-feature vector:
\begin{equation}
    \mathbf{y}_{\text{meta}} = \left( y_1, y_2, \dots, y_N \right)
    \label{eq:meta_feature}
\end{equation}

A meta-classifier $F$ then combines these predictions to produce the final classification:
\begin{equation}
    \hat{y} = F \left( \mathbf{y}_{\text{meta}} \right)
    \label{eq:final_prediction}
\end{equation}

The predicted class $\hat{y}$ corresponds to one of the predefined categories:
\begin{equation}
    \hat{y} \in \left\{ \text{Donor}, \text{Acceptor}, \text{Non-junction} \right\}
    \label{eq:final_classes}
\end{equation}

The proposed pipeline for DNA sequence classification begins with the tokenization of raw DNA sequences. Each input sequence $S = \left( s_1, s_2, \dots, s_N \right)$, where $s_i$ represents a nucleotide in the DNA alphabet $\{A, C, G, T\}$, is segmented into overlapping $k$-mers, resulting in a sequence of tokens $T = \left( t_1, t_2, \dots, t_M \right)$, where $M = N - k + 1$ (Equation~\ref{eq:token_sequence}). To provide structural information to the model, special tokens \texttt{[CLS]} and \texttt{[SEP]} are appended at the beginning and end of the token sequence, respectively, forming $T' = \left( \texttt{[CLS]}, t_1, t_2, \dots, t_M, \texttt{[SEP]} \right)$ (Equation~\ref{eq:special_tokens}).

Each token $t_j$ in $T'$ is transformed into a dense vector representation $\mathbf{x}_j \in \mathbb{R}^d$ through an embedding function $E(t_j)$, learned as part of the DISTILBERT model \citep{sanh2019distilbert}, as shown in Equation~\ref{eq:embedding}. The entire sequence of embeddings is represented as $\mathbf{X} = \left[ \mathbf{x}_{\texttt{[CLS]}}, \mathbf{x}_1, \dots, \mathbf{x}_M, \mathbf{x}_{\texttt{[SEP]}} \right] \in \mathbb{R}^{(M+2) \times d}$ (Equation~\ref{eq:embedded_sequence}).

These embeddings are input to DISTILBERT’s transformer layers, which capture both local and long-range dependencies using a combination of Multi-Head Self-Attention (MHSA) and Feed-Forward Neural Networks (FFN). In each attention head $h$, the attention scores are computed by projecting $\mathbf{X}$ into query, key, and value spaces through learned weight matrices $\mathbf{W}_h^Q$, $\mathbf{W}_h^K$, and $\mathbf{W}_h^V$, resulting in $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, as expressed in Equations~\ref{eq:query}--\ref{eq:value}. The scaled dot-product attention is then computed according to Equation~\ref{eq:attention}.

The outputs from all $H$ heads are concatenated and projected back into the model dimension using $\mathbf{W}^O$, yielding the MHSA output described by Equation~\ref{eq:mhsa}. Each token’s representation is then passed through a feed-forward network (FFN), as defined in Equation~\ref{eq:ffn}. Layer normalization and residual connections are applied after both the MHSA and FFN sublayers, as shown in Equations~\ref{eq:mhsa_norm} and~\ref{eq:ffn_norm}, respectively. After processing through $L$ transformer layers, the final output representation is denoted as $\mathbf{H} = \mathbf{H}^{(L)} \in \mathbb{R}^{(M+2) \times d}$ (Equation~\ref{eq:final_output}).

For classification, the contextualized representation corresponding to the \texttt{[CLS]} token, $\mathbf{h}_{\texttt{[CLS]}} = \mathbf{H}_{0} \in \mathbb{R}^{d}$, is extracted as the aggregate feature vector for the entire sequence (Equation~\ref{eq:cls_feature}). This feature vector is then input to a stacking ensemble classifier. Each base classifier $f_i$, where $i = 1, 2, \dots, N$, predicts a class probability vector $y_i = f_i(\mathbf{h}_{\texttt{[CLS]}})$ (Equation~\ref{eq:base_classifier}). These outputs are concatenated into a meta-feature vector $\mathbf{y}_{\text{meta}} = \left( y_1, y_2, \dots, y_N \right)$ (Equation~\ref{eq:meta_feature}). A meta-classifier $F$ then combines these predictions to produce the final classification $\hat{y} = F \left( \mathbf{y}_{\text{meta}} \right)$ (Equation~\ref{eq:final_prediction}).

The predicted class $\hat{y}$ corresponds to one of the predefined categories: donor, acceptor, or non-junction splice sites (Equation~\ref{eq:final_classes}). By leveraging the deep contextual embeddings from DISTILBERT \citep{sanh2019distilbert} alongside the generalization power of stacking ensembles, the proposed approach provides a reliable and accurate framework for the classification of splice junctions in genomic sequences.


\begin{figure}[ph]
\centering
\includegraphics[width=0.7\linewidth]{Figure 3.jpg}
\caption{(a) Transformer Layer (b) DISTILBERT Transformer}
\label{fig:Figure 3}
\end{figure}

 Figure ~\ref{fig:Figure 3}.a shows transformer layer and Figure ~\ref{fig:Figure 3}.b shows shows DISTILBERT Transformer which contains BERT Base and DISTILBERT. BERT is a powerful transformer model that uses bidirectional context to understand text, making it useful for tasks such as classification. DISTILBERT is a smaller, faster version of BERT developed through knowledge distillation that provides comparable performance with fewer layers. Both are used to classify the DNA splice junctions by treating DNA sequences as tokenized k-mers and taking advantage of their ability to detect contextual patterns in data.
                                                               
\section*{Simulation Results & Analysis}
This section aims at providing a complete analysis through performance evaluation across different tasks for the trained DNABERT Transformer's and DISTILBERT Transformer's. The tests performed were designed to implement and test the usability of the proposed model architecture. A holistic understanding of the model's performance is provided through the discussion of both visual and quantitative results. The experiments have been executed in anaconda prompt and vs code which uses deep learning methods for optimal sequence classification.\\
\subsection*{A.Dataset Description}
For the proposed work, the UCI splice junction Dataset is utilized to test the DNA splice junction classification model. This dataset has 3190 records of different DNA sequences, each containing a total of 60 base pairs. The dataset is subdivided into three classes: exon-intron junctions (767 sequences), intron-exon junctions (768 sequences), and non-junctions (1655 sequences). Each DNA sequence consists of nucleotides, which are represented as A(Adenine), T(Thymine), C(cytosine), and G(guanine). The dataset is well-known for its simplistic and proportional portrayal of splice junctions, which makes it suitable to validate the proposed models claim of accuracy and effectiveness. Splice junction datasets are also common in bioinformatics and machine learning for the purpose of testing splice site detection algorithms. The test cases performed on the dataset evaluated the model’s performance through two approaches. The first experiment was cross-validation, and it was meant to test the generalizability of the model. This methodology guaranteed that the model was both trained and tested on various portions of the dataset so that its predictive performance could be comprehensively appraised. The second experiment splits the dataset into 70\% for training and 30\% for testing. This division was made to assess the model’s results on new data, imitating real-life situations where attempts are made to guess on fresh genomic sequences. These experimental setups allowed the model to be evaluated for accuracy and tested for robustness.

\begin{table}[h]
    \caption{Training and Testing data.}
    \label{tab:2}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row spacing
    \footnotesize % Optional: reduce font size for better fit
    \begin{tabular}{|p{1cm}|p{5cm}|p{3cm}|p{3cm}|p{3cm}|}
        \hline
        \textbf{S.No} & \textbf{Dataset Name} & \textbf{Total Sequences} & \textbf{Training Sequences} & \textbf{Testing Sequences} \\
        \hline
        1 & UCI-Splice Junction Dataset & 3190 & 2233 & 957 \\
        \hline
    \end{tabular}
\end{table}


Tabe ~\ref{tab:2} shows the training and testing sequences where total sequences are 3190, 70\% of 3190 are training sequences they are 2233 sequences, 30\% of 3190 are testing sequences they are 957. 

\begin{table}[h]
    \caption{UCI splice junction dataset sequences.}
    \label{tab:3}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row spacing
    \begin{tabular}{|p{0.75cm}|p{16cm}|} 
        \hline
        \textbf{Class} & \textbf{DNA Sequences} \\
        \hline
        
        EI & AGCCAGGGCACTCACCAGGCTGCAAGAACAGTGCTGGGGTAAGAGGGGAGCGGGGGATCC… \\
        \cline{2-2}
           & GGCTGCCGGAGCCCCTCATGCTGAGATGGAGTAAGGAGGGAGATGGAGGCATCATGTCTG… \\
        \cline{2-2}
           & CACTGGTTATCACCCTTTACTGCAACCACAGTAAGTCCCGAGGAATCGCGGCGGGAAGGA… \\
        \hline
        
        IE & TCCTCATTCCTGCGTCTGCTTCCTCCCCAGCAAAAGCGTGATCTTGCTGGGTCGGCACAG… \\
        \cline{2-2}
           & TCCCTCCTGATCCCATCCCTCCTTTCCCAGGGAGTGGAGGATGCCTTCTACACGTTGGTG… \\
        \cline{2-2}
           & GCGACTGCCTGACTTCTCCTTTCATTTCAGCACCATGAAGCTTCTCACGGGCCTGGTTTT… \\
        \hline
        
        N  & CTTGTTACAGGGGTGCTGGCCTTCCCTGGCGCCTGCCCCTGTCGGCCCCGCCCGAGAACC… \\
        \cline{2-2}
           & GCCATGGTCAAGATTCTCCAGCTGTGTATTTGAAAAAGGGCCAAGGATCCACCACAACAC… \\
        \cline{2-2}
           & AGGAAGTCGGGAGGCGTGGAGGGTGACCACACCTTGGCCTTCTGGGAACTCTCCTTTGTG… \\
        \hline
    \end{tabular}
\end{table}


Table~\ref{tab:3} shows how dataset is constructed. It contains three classes EI, IE , N and the DNA sequences related to them.  
\subsection*{B.	Evaluation Metrics}
The proposed model's performance is evaluated using several metrics derived from the test set's confusion matrix. True Positive (TP) denotes cases in which the model correctly predicts positive outcomes, whereas True Negative (TN) denotes cases in which negative outcomes are correctly predicted. False Positive (FP) and False Negative (FN) are examples of when the model incorrectly predicts positive and negative outcomes, respectively. The efficiency of the proposed methodology is evaluated using the following metrics:
  
\item Accuracy: The percentage of correct classifications, which includes all DNA splice junction types (EI, IE, or N), is calculated using accuracy. It is calculated as the proportion of correctly predicted outcomes (TP + TN) to the total number of predictions (TP + TN + FP + FN); it is calculated using Eq.\ref{eq:6}
\begin{equation}
\text{Accuracy} = \frac{(\text{True Positives} (TP) + \text{True Negatives} (TN))}{\text{Total number of samples}} \times 100
\label{eq:6}
\end{equation}
\item	Precision: The ratio of correctly predicted positive outcomes (TP) to total predicted positive outcomes (TP + FP), which demonstrates the model's ability to avoid false positives; it is calculated using Eq.\ref{eq:7}
\begin{equation}
\text{Precision} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Positives (FP)}}
\label{eq:7}
\end{equation}
\item	Recall: The ratio of correctly predicted positive outcomes (TP) to total actual positive outcomes (TP + FN), indicating the model's ability to identify all positive instances;it is calculated using Eq.\ref{eq:8}
\begin{equation}
\text{Recall} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}
\label{eq:8}
\end{equation}
\item	F-measure: is given as the harmonic mean of precision and recall, providing a balanced evaluation of the model's precision and recall; it is calculated using Eq.\ref{eq:9}
\begin{equation}
F\text{-measure} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
\label{eq:9}
\end{equation}
\item	AUC: Area under curve, it shows the relationship between the actual positive rate and the
False positive rate at different thresholds ; it is calculated using Eq.\ref{eq:10}
\begin{equation}
AUC = \frac{1}{2} \left( \frac{\text{ (TP)}}{\text{ (TP)} + \text{ (FN)}} + \frac{\text{(TN)}}{\text{ (TN)} + \text{ (FP)}} \right)
\label{eq:10}
\end{equation}

\subsection*{C.Quantitative metrics}
DNA splice junctions were annotated using several machine-learning classifiers, including Range Forest, K Nearest Neighbour (KNN), Logistic Regression, and Support Vector Machine (SVM). Each classifier was assessed on a set of performance criteria such as accuracy, F-measure, Precision, and Recall for three classes of DNA splice junctions; exon-intron (EI), intron-exon (IE), and Non-Junctions (N). Random Forest proved to have an accuracy of 81.41\%, suggesting good predictive power, which was better than the rest of the classifiers. It had the highest precision and recall for all classes, especially for the non-junction (N) class, where it achieved an F-measure of 0.8956. K-Nearst Neighbour (KNN) \citep{gu2021learning} achieved lower accuracy of 77.63\% and also lower recall values, most notably in the IE class (0.6173), signifying low ability to discern intron-exon junctions.


\begin{table}[h]
    \caption{Evaluation of Different Classifiers}
    \label{tab:4}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row spacing
    \footnotesize
    \begin{tabular}{|p{3cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}|}
        \hline
        \textbf{Model} & \textbf{Accuracy} & \textbf{F-Measure} & \textbf{Precision} & \textbf{Recall} & \textbf{Classes} \\
        \hline
        
        \multirow{}{}{Random Forest Classifier} 
        & \multirow{}{}{81.41} & 0.8211 & 0.8012 & 0.8456 & EI \\
        \cline{3-6}
        &  & 0.6234 & 0.7267 & 0.6298 & IE \\
        \cline{3-6}
        &  & 0.8956 & 0.8591 & 0.9412 & N \\
        
        \hline
        
        \multirow{}{}{K-Nearest Neighbor} 
        & \multirow{}{}{77.63} & 0.8097 & 0.7764 & 0.8245 & EI \\
        \cline{3-6}
        &  & 0.7564 & 0.6754 & 0.6173 & IE \\
        \cline{3-6}
        &  & 0.8734 & 0.8512 & 0.8976 & N \\
        
        \hline
        
        \multirow{}{}{Logistic Regression} 
        & \multirow{3}{*}{79.62} & 0.8356 & 0.8245 & 0.8435 & EI \\
        \cline{3-6}
        &  & 0.6781 & 0.7568 & 0.7413 & IE \\
        \cline{3-6}
        &  & 0.8897 & 0.8014 & 0.8876 & N \\
        
        \hline
        
        \multirow{}{}{Support Vector Machine} 
        & \multirow{}{}{79.92} & 0.8234 & 0.8123 & 0.8345 & EI \\
        \cline{3-6}
        &  & 0.7356 & 0.8197 & 0.7342 & IE \\
        \cline{3-6}
        &  & 0.8897 & 0.7985 & 1.0000 & N \\
        
        \hline
    \end{tabular}
\end{table}

With an accuracy of 79.62\%, logistic regression provided a moderate balance of precision and recall, ranking first in the N class (F-measure: 0.8897). Meanwhile, the Support Vector Machine (SVM) \citep{hastie2009elements} achieved an accuracy of 79.92\% in classifying EI and N junctions, but had a lower recall value for the IE class (0.7342). Notably, SVM successfully classified the N class with a recall of 1.00, indicating that all non-junction sequences were correctly identified and are as shown in Table~\ref{tab:4}.


\begin{table}[h]
    \caption{Evaluation of Transformer Models}
    \label{tab:5}
    \centering
    \renewcommand{\arraystretch}{1.3} % Adjust row spacing
    \footnotesize % Optional: reduce font size
    \begin{tabular}{|p{4cm}|p{2cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}|}
        \hline
        \textbf{Model} & \textbf{Accuracy (\%)} & \textbf{F-Measure} & \textbf{Precision} & \textbf{Recall} & \textbf{Classes} \\
        \hline
        \multirow{}{}{DNABERT Transformer + Stacking Ensemble} 
        & \multirow{}{}{95} 
        & 0.94 & 0.95 & 0.89 & EI \\
        \cline{3-6}
        &  & 0.91 & 0.93 & 0.85 & IE \\
        \cline{3-6}
        &  & 0.95 & 0.90 & 0.99 & N \\
        \hline
        
        \multirow{}{}{DISTILBERT Transformer + Stacking Ensemble} 
        & \multirow{}{}{91} 
        & 0.91 & 0.95 & 0.87 & EI \\
        \cline{3-6}
        &  & 0.89 & 0.94 & 0.84 & IE \\
        \cline{3-6}
        &  & 0.92 & 0.88 & 0.97 & N \\
        \hline
    \end{tabular}
\end{table}


DNA splice junctions were successfully categorized as shown in Table ~\ref{tab:5} by using transformer-based deep learning models, DNABERT and DISTILBERT. In the N class, DNABERT showed exceptional performance and ensured well-balanced classification with an accuracy of 95\% and high precision (0.95) and recall (0.99). With an accuracy of 91\%, DISTILBERT also demonstrated competitive performance. It maintained high precision in all classes, but it excelled in the EI class with a precision of 0.95. The ability of transformer architectures to precisely identify DNA splice junctions was demonstrated by both models, underscoring their potential to advance genomic sequence analysis.

\subsection*{D.Visual Results}

While quantitative metrics provide a numerical assessment, visualizing the model's predictions offers a qualitative understanding of its performance.

  \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 4.jpg}
\caption{Confusion matrix for (a) Random forest classifier  (b)K-Nearest Neighbour.}
\label{fig:figure 4}
\end{figure}
   
Figure ~\ref{fig:figure 4} shows the confusion matrix for two classification algorithms: Random forest classifier and K-Nearest Neighbour. It demonstrates good classification performance for Class 2 and Class 0, whereas Class 1 has the lowest accuracy indicating more misclassification. Overall, the model performs well, but there is scope for improvement in distinguishing Class 1.

 \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 5.jpg}
\caption{ROC curve for (a) Random forest classifier (b) K-Nearest Neighbour.}
\label{fig:figure 5}
\end{figure}


 \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 6.jpg}
\caption{Precision-Recall curve for (a) Random forest classifier  (b)K-Nearest Neighbour.}
\label{fig:figure 6}
\end{figure}

Figure ~\ref{fig:figure 5} and Figure ~\ref{fig:figure 6} shows the multi-class ROC curves and Precision-Recall curves for the Random forest classifier and K-Nearest Neighbour models. Both models perform well in Class 0 and Class 2, with high AUC and AUPR values. However, Class 1 has the lowest AUC and AUPR values  in both models, indicating increased misclassification.

 \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 7.jpg}
\caption{ Confusion matrix for (a) Logistic regression (b) support vector machine.}
\label{fig:figure 7}
\end{figure}

Figure ~\ref{fig:figure 7} shows the confusion matrices for the Logistic Regression and Support Vector Machine (SVM) models. Both models perform well in Class 2. Class 0 is also accurately classified in both models, with a large number of correctly predicted samples. Class 1 has the lowest accuracy in both models, with significant misclassification into classes 0 and 2.

 \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 8.jpg}
\caption{ ROC curve for c) Logistic regression d) support vector machine.}
\label{fig:figure 8}
\end{figure}


 \begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 9.jpg}
\caption{ Precision-Recall curve for (a) Logistic regression (b) support vector machine.}
\label{fig:figure 9}
\end{figure}
 
Figure ~\ref{fig:figure 8} and Figure ~\ref{fig:figure 9} shows the multi-class ROC curves and Precision-Recall curves for the Logistic Regression and Support vector machine. Both models perform well in Class 0 and Class 2, with high AUC and AUPR values. However, Class 1 has the lowest AUC and AUPR values in both models, indicating increased misclassification.

\begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 10.png}
\caption{ Performance Comparison of four models}
\label{fig:figure 10}
\end{figure}
 

Figure ~\ref{fig:figure 10} depicts a performance comparison of four machine learning models Random Forest, K-Nearest Neighbours (KNN), Logistic Regression, and Support Vector Machine (SVM) using four evaluation metrics: accuracy, precision, recall, and F1-Score.

\begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 11.jpg}
\caption{ Confusion matrix a) DNABERT Transformer +Stacking ensemble method b) DISTILBERT Transformer + Stacking Ensemble method.}
\label{fig:figure 11}
\end{figure}
 
Figure~\ref{fig:figure 11} shows the confusion matrix for two transformer models: DNABERT and DISTILBERT. DNABERT Transformer + Stacking ensemble method shows excellent classification performance, particularly in Class 2, with 635 correctly classified instances (96.2\%). The overall diagonal dominance indicates high accuracy, which means that the majority of predictions match the true labels. In addition, Class 0 has 286 correctly classified instances (91.9\%), while Class 1 has 287 correctly classified instances (91.4\%). Misclassification rates are low, indicating that the model generalizes well across all classes

\begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 12.jpg}
\caption{ ROC curve for (a) DNABERT Transformer +Stacking ensemble method (b) DISTILBERT Transformer + Stacking Ensemble method.}
\label{fig:figure 12}
\end{figure}
 
Figure ~\ref{fig:figure 12} shows the multiclass ROC curves for two classification models. (a) depicts DNABERT Transformer+ Stacking ensemble with AUC values of 0.99 for all three classes (Class 0, Class 1, and Class 2), indicating excellent classification performance and near-perfect discrimination between classes. (b) depicts DISTILBERT Transformer+ Stacking Ensemble with AUC values of 0.99 for all three classes, indicating excellent classification performance.

\begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 13.jpg}
\caption{ Precision-Recall curve for (a) DNABERT Transformer +Stacking ensemble method (b) DISTILBERT Transformer + Stacking Ensemble method.}
\label{fig:figure 13}
\end{figure}

Figure ~\ref{fig:figure 13} depicts the multiclass precision-recall curves for two classification models. Figure~\ref{fig:figure 13}.a depicts the DNABERT Transformer + Stacking Ensemble, which achieves high precision across all recall levels and exhibits strong classification performance. Figure~\ref{fig:figure 13}.b shows the DISTILBERT Transformer + Stacking Ensemble, which maintains high precision with minimal drop-off at higher recall values, indicating effective class discrimination. Both models perform extremely well, with precision-recall curves indicating strong predictive capabilities.

\begin{figure}[ph]
\centering
\includegraphics[width=\linewidth]{Figure 14.png}
\caption{ Training and Testing Accuracy of DNABERT Transformer.}
\label{fig:figure 14}
\end{figure}

Figure~\ref{fig:figure 14} shows proposed model achieves 100\% training accuracy, which suggests that it may be overfitting. However, testing accuracy stabilizes at around 95\%, indicating good generalization. Increasing the number of estimators beyond 20-30 yields little improvement. Overall, the model performs admirably, with high accuracy and no excessive overfitting.

    
\subsection*{E. Discussions & Comparisons}

In this section, an in-depth comparative analysis of DNABERT transformer + Ensemble method is performed against other state-of-the-art models, evaluating its performance on UCI splice junction dataset. The examination includes various aspects like network architecture and performance metrics.

\subsubsection*{Comparison with Other Networks}

a. Network Architecture\\
This work commences an analysis by comparing the architecture of proposed work with other prominent models. 


\begin{table}[h]
\caption{Network Architecture Comparison}
    \label{tab:6}
    \centering
    \begin{tabular}{|l|l|}
        \hline
        \textbf{Reference} & \textbf{Architecture} \\
        \hline
        \citep{ji2021dnabert} & DNABERT Transformer \\
        \hline
        \citep{choi2023transformer}   & AC-Caps \\
        \hline
        Proposed Work & DNABERT Transformer + Stacking ensemble method \\
        \hline
    \end{tabular}
    
\end{table}

Table ~\ref{tab:6} highlights the architectures used in other works and the proposed work.


b. Model parameters\\
This work persists with a study of the model parameters. Table~\ref{tab:7} outlines the model parameters employed in the proposed method. The Transformer Encoder has 12 layers, a hidden size of 768, 12 attention heads, and an FFN size of 3072, with the activation function GLEU. The AdamW optimizer is used, with a learning rate of 4e-4 and a warm-up phase of 15\% of the total steps. The output layer contains a Random Forest classifier with 100 estimators, a random state about 42, and five-fold cross-validation.

\begin{table}[h]
 \caption{Model parameters of DNABERT Transformer}
    \label{tab:7}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row height
    \begin{tabular}{|l|l|l|}
        \hline
        \textbf{Structure} & \textbf{Parameters} & \textbf{Value} \\
        \hline
        \multirow{Tokenization} & K-mer size & 6 \\
        \hline
        \multirow{}{}{Transformer Encoder} 
        & Number of layers & 12 \\
        & Hidden size & 768 \\
        & Number of attention heads & 12 \\
        & Intermediate size (FFN) & 3072 \\
        & Activation Function & GELU \\
        & Dropout rate & 0.1 \\
        & Attention dropout & 0.1 \\
        & Maximum sequence length & 512 \\
        \hline
        \multirow{Training} 
        & Batch size & 2000 \\
        & Optimizer & AdamW \\
        & Learning rate & 4.00E-04 \\
        & Warm-up steps & 15\% of total steps \\
        \hline
        \multirow{Feature Extraction} & Output feature size & 768 \\
        \hline
        \multirow{Classification Layer (Stacking Ensemble)} 
        & Number of estimators & 100 \\
        & Random state & 42 \\
        & Cross Validation & Five folds \\
        \hline
    \end{tabular}
   
\end{table}

\begin{table}[h]
  \caption{Model parameters of DISTILBERT Transformer.}
    \label{tab:8}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row height
    \begin{tabular}{|l|l|l|}
        \hline
        \textbf{Structure} & \textbf{Parameters} & \textbf{Value} \\
        \hline
        \multirow{Embedding Layer} 
        & Vocabulary size & 30522 \\
        & Embedding dimension & 768 \\
        & Position embedding size & 512 \\
        \hline
        \multirow{{Transformer Encoder} 
        & Number of layers & 6 \\
        & Hidden size & 768 \\
        & Intermediate size (FFN) & 3072 \\
        & Number of attention heads & 12 \\
        & Attention head size & 64 \\
        & Dropout rate & 0.1 \\
        & Activation function & GLEU \\
        & Number of parameters & 66M \\
        \hline
        \multirow{Output Layer} 
        & Number of estimators & 100 \\
        & Random state & 42 \\
        & Cross Validation & Five folds \\
        \hline
    \end{tabular}
  \end{table}

Table~\ref{tab:8} outlines the model parameters employed in the proposed method. The Transformer Encoder has multiple layers, with a hidden size of 768 and an FFN size of 3072. It has 12 attention heads, each with a size of 64, and a dropout rate of 0.1. The activation function is GLEU. The embedding layer has a variable vocabulary size, a 768-dimensional embedding, and position embeddings. The model contains approximately 66M parameters. The output layer contains a Random Forest classifier with 100 estimators, a random state about 42, and five-fold cross-validation.\\
c. Performance Metrics\\
Moving beyond architecture and model parameters, proposed work evaluates the performance metrics of these architectures on UCI splice junction dataset, emphasizing their accurately splicing capabilities.


\begin{table}[h]
    \caption{Performance Metrics Comparison.}
    \label{tab:9}
    \centering
    \renewcommand{\arraystretch}{1.2} % Adjust row height
    \footnotesize % Reduce font size if needed
    \begin{tabular}{|p{4cm}|p{6cm}|p{2cm}|}
        \hline
        \textbf{Reference} & \textbf{Network} & \textbf{Accuracy (\%)} \\
        \hline
        \citep{ji2021dnabert} & DNABERT Transformer & 92.3 \\
        \hline
        \citep{choi2023transformer} & AC-Caps & 92.5 \\
        \hline
        Proposed Work & DNABERT Transformer + Stacking Ensemble Method & 95.0 \\
        \hline
    \end{tabular}
\end{table}

These metrics demonstrate the proposed model's performance in accuracy evaluation, with a 9\% accuracy rate. This outperforms other models, such as DNABERT Transformer (92.3\%)\citep{ji2021dnabert} and AC-Caps (92.5\%)\citep{choi2023transformer}, demonstrating the ensemble method's efficiency. Table ~\ref{tab:9}shows the results, which highlight the proposed approach's superior predictive capability.

d. Dataset comparison\\
Proposed work uses the Gene Sequences Dataset, \citep{regan2021splice} uses the Homo sapiens Splice Site Dataset to classify splice junctions. Proposed methodology uses the DNABERT Transformer and an ensemble approach for classification, achieving 95\% accuracy. The classification task in \citep{regan2021splice} is performed using the Homo sapiens Splice Site Dataset and a Long Short-Term Memory Recurrent Neural Network .


\begin{table}[h]
    \caption{Comparison of Network Performance on Different Datasets}
    \label{tab:10}
    \centering
    \footnotesize % Optional: reduce font size to fit the page
    \renewcommand{\arraystretch}{1.2} % Adjust row height
    \begin{tabular}{|p{2.5cm}|p{3cm}|p{4cm}|p{1.5cm}|p{1.5cm}|}
        \hline
        \textbf{Reference} & \textbf{Dataset} & \textbf{Network} & \textbf{Accuracy (\%)} & \textbf{F1-Score (\%)} \\
        \hline
        
        \multirow{}{}{\cite{regan2021splice}} 
        & \multirow{}{}{Homo-Sapiens Splice Site Dataset} 
        & Long Short-Term Memory & 91.34 & 90.85 \\
        \cline{3-5}
        & & CNN-LSTM & 91.25 & 89.90 \\
        \cline{3-5}
        & & GRU & 91.97 & 91.10 \\
        
        \hline
        
        \multirow{}{}{Proposed Work} 
        & \multirow{}{}{UCI-Splice Junction Dataset} 
        & DNABERT Transformer + Ensemble Method & 95.00 & 94.00 \\
        \cline{3-5}
        & & DNABERT + Random Forest & 95.00 & 93.50 \\
        \cline{3-5}
        & & DNABERT + XGBoost & 91.00 & 90.00 \\
        
        \hline
    \end{tabular}
\end{table}


An overall accuracy of 91.34\% over 50 runs and an average F1-score of 91.3\% is achieved by \citep{regan2021splice}. The accuracy quartiles were 90.78\%, 91.22\%, and 91.39\%, and the F1-score quartiles were 90.85\%, 91.25\%, and 91.97\%. The higher accuracy and F1-scores mentioned in Table ~\ref{tab:10} ~\ref{tab:10}for proposed work demonstrates the supremacy of the proposed DNABERT based approach over LSTM for splice junction classification. 

e. Analysis

\item The proposed model outperforms DNABERT Transformer (92.3\%) \citep{ji2021dnabert} and AC-Caps (92.5\%) \citep{choi2023transformer} in sequence analysis, with an accuracy of 95\%.
\item The addition of the ensemble method improves predictive performance, resulting in robust and reliable results across datasets.
\item DNABERT Transformer was significant in achieving these results, employing its deep learning architecture to capture complex sequence patterns and improve classification accuracy.\\
 Leveraging Attention Mechanism\\

One of the critical components that facilitated the success of DNABERT is the use of the attention mechanism. This attention mechanism allows the model to concentrate on the most pertinent subsequences of the DNA sequences, benefitting both sequence classification and feature extraction. DNABERT uses attention economically by dynamically adjusting the importance of different sections of the input, allowing it to perform well on a variety of genomic datasets. 

To summarize, DNABERT is a robust and versatile model that efficiently meets the requirements of complex DNA sequence analysis challenges. Its attention mechanism gives the model the capability to perform complex tasks in understanding and analyzing genomic data. This versatility marks DNABERT as a model, used in different branches of bioinformatics like feature extraction or sequencing classification.
\section*{Conclusion \& Future Scope}
This work examined the two-layer approach for DNA splice junction classification with the proposed UCI splice junction dataset of 3190 sequences. The results were validated against state-of-the-art methods where the DNABERT Transformer + Stacking ensemble method outperformed the others with 95\% accuracy, followed by the DISTILBERT Transformer + Stacking ensemble with 91\% accuracy. Both performed better than existing models. This confirms that ensemble techniques and transformer-based models are effective for predicting DNA splice junctions. The central premise of these models are to integrate the feature extraction abilities of transformers with the classifying strength of Stacking ensembles. The two pre-trained models DNABERT and DISTILBERT, yield the highest results among all other trained classifiers. In addition, the model accuracy may be further improved and made more generalizable by investigating hybrid ensemble architectures and deep learning classifiers. Adapting the model to biological problems would require the datasets to be enlarged to include a more diverse set of genomic sequences. In addition, the application of explainability methods, including attention mechanisms, could improve understanding of DNA splicing and aid in the interpretation of results. Further work will focus on the real-time application of this model to tailored medicine, genomic counter detection, and bio-pharmaceutical design, in an attempt to increase the model's usefulness in the fields of genomics and medical diagnostics.

\section*{Ethics declarations}
Ethics approval and consent to participate
This article does not contain any studies with human participants or animals performed by any of the authors.
\section*{Consent for publication}
Not applicable.
\section*{Consent to Participate declaration}
Not applicable.
\section*{Competing interests}
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The authors declare no conflict of interest.

\section*{Authors' contributions}
K.S., P.T., B.A., S.A., M.M., K.M., P.N., H.B.V., T.R.V.L. worked on Conceptualizations, methodology, software, formal analysis, data curation, and writing—original draft preparation, while S.A.H., S.M. participated in writing—review and editing, validation, supervision, project administration. All authors have read and agreed to the published version of the manuscript.

\section*{Funding}
The authors receive no funding from their institutes.

\section*{Data Availability Statement}
The data used for this work is avaliable at Kaggle website, \url{https://www.kaggle.com/datasets/drpardhuthottempudi/dna-sequence-data} 

\bibliographystyle{johd}
\bibliography{sample}

\end{document}