Sentiment analysis, also known as opinion mining, is an essential task in natural language processing (NLP) that entails recognizing and classifying the emotional tone conveyed in a text. It finds extensive uses in domains like social media observation, customer response evaluation, public sentiment assessment, and news investigation. Although considerable advancements in sentiment analysis have occurred for high-resource languages such as English, Chinese, and Hindi, numerous Indian regional languages, including Gujarati, continue to be underrepresented in NLP studies owing to a scarcity of labeled datasets and linguistic tools. Gujarati is spoken by more than 60 million individuals globally, but there are very few sentiment-labeled datasets available to the public for the language. Gujarati's morphological richness, unrestricted word order, and lack of commonly accessible NLP tools make it a special challenge for sentiment analysis. Model development and benchmarking are further complicated by the fact that the Gujarati datasets that are currently available are small, frequently domain-specific, and infrequently sentiment-labeled.Current initiatives in Gujarati sentiment analysis have mainly concentrated on limited datasets, specialized corpora (such as film or product evaluations) [3][8], or social media material like tweets [9]. These are either restricted in scope or fail to generalize across areas.

Additionally, the lack of publicly accessible, extensive, labeled datasets for Gujarati greatly hinders the creation and assessment of machine learning models. To tackle these issues, we suggest a hybrid annotation framework that methodically integrates rule-based lexicon annotation with manual and pseudo-labeling methods to create a large-scale sentiment-labeled dataset for Gujarati news headlines. Initially, we created a sentiment lexicon featuring Gujarati terms, along with their synonyms and antonyms, enhanced by utilizing resources such as Hindi WordNet and a Hindi-Gujarati dictionary. This dictionary was utilized to automatically label more than 21,000 headlines with sentiment tags. Although this rule-based approach offered a solid foundation, it had difficulty with subtle or unclear headlines.

We subsequently assessed various machine learning models on this combined dataset, such as Logistic Regression, Naive Bayes, Support Vector Machine, Random Forest, Gradient Boosting, Bagging, and AdaBoost. Of these, the Random Forest classifier utilizing TF-IDF features attained the highest performance, with an accuracy of 88.54%—a notable enhancement from the 72.75% baseline established during the rule-based annotation stage.

Current Gujarati sentiment research lacks scalable annotation pipelines and large, domain-independent datasets, and is restricted to lexicon-based approaches (Shah & Swaminarayan, 2021; Shah et al., 2022) or small Twitter corpora like GSAC (Gokani & Mamidi, 2023). By presenting a hybrid annotation method that combines machine pseudo-labeling, manual labeling, and confidence-based filtering to produce a high-quality sentiment corpus, our work fills this gap. To the best of our knowledge, this is the first hybrid technique for low-resource languages and the largest Gujarati sentiment dataset.This paper makes the following key contributions:

The remainder of this paper is structured as follows: Section 2 discusses related work. Section 3 describes the dataset and annotation methodology. Section 4 presents the feature engineering and model training techniques. Section 5 reports experimental results and analysis. Section 6 concludes the paper and outlines directions for future research.

Sentiment analysis (SA) is an extensively researched task in natural language processing that identifies the sentiment of text and has been frequently utilized in social media, customer feedback, and news analysis for high-resource languages (Pang & Lee, 2008; Akhtar et al., 2016). Initial research predominantly utilized lexicon-based methods, employing pre-established sentiment word dictionaries with polarity ratings to categorize text (Shah & Swaminarayan, 2021). Although these approaches do not need labeled training data and are effective for domain-specific tasks, they frequently struggle to grasp contextual polarity and yield unsatisfactory results in morphologically complex or agglutinative languages (Chen & Skiena, 2014). Techniques in machine learning, including Naïve Bayes, Logistic Regression, Support Vector Machines, and Random Forest classifiers, enhanced sentiment classification capabilities by utilizing labeled datasets (Cortes & Vapnik, 1995). These models identify more intricate patterns but need extensive annotated data, which is frequently lacking for low-resource languages like Gujarati (Mehta & Rajyagor, 2021). To overcome this limitation, hybrid methods that integrate lexicon-based rules and machine learning have surfaced, successfully utilizing the accuracy of lexicon scoring alongside the generalization abilities of supervised models (Breiman, 2001).

Early attempts concentrated on limited datasets like film reviews (Shah et al., 2022) and specially created lexicons (Shah & Swaminarayan, 2021), yet these resources were not made publicly available, hindering reproducibility and benchmarking. The Gujarati Sentiment Analysis Corpus (GSAC) is a recent addition that offers a manually annotated collection of 6,575 tweets and has set baseline results through the use of traditional and transformer-based models (Gokani & Mamidi, 2023). Nevertheless, GSAC is confined to social media language and is still quite limited in scale, which constrains its use for sentiment modeling that is not dependent on a specific domain.

To address these challenges, the present study presents a hybrid annotation framework that merges expert manual annotation with machine-generated pseudo-labels refined by confidence scores. This method allows for the development of a sizable, domain-agnostic sentiment dataset (~ 105,000 Gujarati news headlines) and provides notably enhanced classification precision, tackling the lack of high-quality annotated data for sentiment analysis in Gujarati, and this method can be easily replicable by other low-resource languages.

From the original dataset we have randomly selected 11,625 headlines for manual annotation. We first created annotation schema [Gokani, Monil, and Radhika Mamidi. "GSAC: A Gujarati Sentiment Analysis Corpus from Twitter." Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. 2023.] and tested it by annotating some part of dataset. Once the schema and dataset was finalized, we given task of annotation to 4 annotators with guidelines like projects, awards, ceremonies, sport success and good government policies, news with positive sentiments should be labelled as positive. Reports of violences, death, corruption, natural calamities or public rage, and news with negative sentiments should be label as negative. The headlines that do not fall under the above two criteria all should be marked as neutral. Additionally, if the news is not clear whether it is praising or criticizing all should be neutral.

With this annotation guidelines 4 annotators annotated headlines. The selected annotators are native speakers of Gujarati language with well-known knowledge of linguistic features of language. Initially all four of them seated together for annotating 100 headlines after all discussion and correction on 100 headlines they completed assigned task of annotating remaining headlines.

Each of the four annotated the news based on above guidelines separately and then kappa statistics was calculated to know inter-annotator score and the results are as shown in Table 1. The results shows that strong agreement across the annotator with Cohen’s kappa values greater than 0.70. for the training purpose annotator 1 and annotator 2 dataset was used as Kappa value greater than 0.88.

Manual annotation is time consuming task and requires trained people for labeling. In such cases, semi-supervised learning offers good alternative. It combines manually annotated dataset with automatically annotated dataset. We have expanded our dataset by pseudo labeling that includes following steps:

The 11625 headlines annotated by annotator used as a standard dataset on which different machine learning algorithm were applied like Naïve Bayes, Logistic Regression, Support vector Machines and Random Forest using TF-IDF and N-Gram. From these Random Forest achieved the best results in terms of accuracy. This created model is used as baseline model.

Once the baseline model was evaluated then it was used to label remaining 93959 news headlines as manual annotation is an unpractical approach due to resource constraints. To bridge this gap, we used Baseline model to predict the labels of remaining headlines it acts like automatic annotator. To ensure the quality of predefined labels we have introduced hybrid annotation verification framework pipeline as shown in Fig. 2. This pipeline includes confidence thresholding mechanism, stratified human sampling and cross validation.

Feature engineering plays important role in machine learning. We used apt feature engineering techniques for ML algorithm to work. In this we work with TF-IDF, count vectorization and N-grams text representation technique. Before moving on to transformer-based models in subsequent work, we concentrated on classical ML models to create solid baseline findings for the new dataset, guaranteeing reproducibility and interpretability.

Various classical machine learning models underwent performance evaluation when trained on a hybrid annotated dataset. All models were implemented using the Scikit-learn library and trained on an 80 − 20 train-test split of the final labeled corpus consisting of around 105,000 samples. The models employed were as follows:

Logistic Regression A linear model that performs well on high dimensional sparse data. It gave stable performance with TF-IDF features.

Naive Bayes (Multinomial) Being simple and fast in its classification task, this model was used with the Count Vectorizer and TF-IDF options.

Random Forest Classifier An ensemble where decision trees constitute the basic estimator. The model gave the best results overall, especially when used with TF-IDF and N-Gram features.

Gradient Boosting Classifier More computationally expensive but had balanced results with a moderate increase in the F1-score for minority classes. Macro-level metrics, including Accuracy, Precision, Recall, and F1-Score, were used for evaluations of all models.

Besides base classifiers, we also examined various ensembles that combine several weak learners (generally decision trees) to give improved performance and reduced variance. The following ensemble methods were considered:

Bagging Classifier (Decision Tree) Bagging or Bootstrap Aggregating reduces overfitting by training several decision trees on random subsets of training data (with replacement) and aggregating their predictions. We employed the BaggingClassifier under Scikit with the base estimator being DecisionTreeClassifier.

AdaBoost Classifier (Decision Tree) AdaBoost creates a chain of decision trees, where each new tree focuses beyond the ones misclassified by the previous so that we used Scikit-learn's AdaBoostClassifier with DecisionTreeClassifier as a weak learner set to max_depth = 1.

The suggested hybrid annotation approach notably enhanced model performance, raising overall accuracy from 72.75% (rule-based baseline) to 88.54%. Of all the models tested and feature engineering methods, the Random Forest classifier using TF-IDF features attained the best accuracy, highlighting the success of merging human-annotated data with machine-generated pseudo-labels. Logistic Regression also competed well with TF-IDF and Bag-of-Words (BOW), reaching accuracies over 85%. Conversely, 2-Gram features consistently performed worse in all models, with Bagging Decision Tree showing the lowest accuracy at 40.20%. These findings confirm the effectiveness of the hybrid annotation pipeline in producing high-quality annotated data for sentiment analysis in Gujarati.

Table 3 displays the comprehensive performance metrics of the model using various feature extraction techniques, while Fig. 3–9 illustrate the model accuracies for each feature representation: Fig. 3 for Logistic Regression, Fig. 4 for Naïve Bayes, Fig. 5 for Random Forest, Fig. 6 for Gradient Boosting, Fig. 7 for Bagging Decision Tree, Fig. 8 for AdaBoost Decision Tree, and Fig. 9 for the overall comparison of all machine learning algorithms. The visualizations clearly demonstrate that features based on TF-IDF consistently surpass those based on BOW and N-Gram across classifiers, validating TF-IDF as the best representation for this dataset.

Gujarati news headlines are relatively short, which contributes to the comparatively poor performance of 2-Gram features (lowest accuracy of 40.20% with Bagging and 65.27% even with Random Forest). Since many bigrams arise infrequently and are unable to reliably capture sentiment patterns, headlines are usually brief (average length 7–10 words), which results in data sparsity when utilizing 2-Gram features. Furthermore, single-word characteristics (1-Gram) are more useful in Gujarati due to its morphological richness, where sentiment cues are frequently represented through single keywords or suffixes.

However, Random Forest was able to capture complicated decision boundaries because TF-IDF weighting of unigrams improved discrimination between sentiment-bearing words and common stopwords. This combination yielded the best accuracy of 88.54%, which can be explained by the fact that ensemble learning algorithms like as Random Forest handle high-dimensional TF-IDF representations well and benefit from feature variety.

Contextual models like transformers are necessary, as we found that the majority of misclassifications happened in headlines that were unclear or humorous. The scalability of the suggested pipeline was demonstrated by the increased pseudo-label quality achieved by applying a 90% confidence criterion, which reduced manual labor by almost 60% while preserving > 90% labelling accuracy.

This study presents a comprehensive and scalable hybrid annotation framework for sentiment analysis in Gujarati, a language with limited annotated resources. We began with a rule-based sentiment lexicon approach, which achieved a respectable benchmark accuracy of 72.75% using Random Forest and N-Gram features. While this method provided a strong foundation, it lacked the flexibility to handle nuanced and ambiguous cases often found in natural language.

To overcome these limitations, we developed a novel hybrid annotation pipeline that combines manual labeling, pseudo-labeling using a baseline classifier, and confidence-based filtering. By setting a high-confidence threshold (≥ 90%) and validating results through stratified human sampling and cross-validation, we ensured that the final annotations were both high-quality and scalable. This process enabled us to create the largest sentiment-labeled Gujarati dataset to date, containing over 1,05,000 headlines.

Our experiments with multiple machine learning models—ranging from Logistic Regression to advanced ensemble techniques like Random Forest and Gradient Boosting—demonstrated that models trained on this hybrid dataset achieved substantial improvements, with the highest accuracy of 88.54% obtained using TF-IDF with Random Forest.

The key novel contributions of this work include:

This work's hybrid annotation process, which combines human labeling with confidence-based pseudo-labeling, is unusual because it makes it possible to quickly create high-quality sentiment corpora for low-resource languages. We constructed the largest Gujarati sentiment dataset (~ 1,05,000 headlines) using this method, and we saw a notable increase in accuracy from 72.75–88.54%. This approach reduces the need for manual annotation while maintaining data quality, and it can be readily modified for additional low-resource languages.

Upon publication, an online repository will make the dataset, annotation standards, and lexicon resources utilized in this study publicly accessible. In addition to facilitating resource reuse and extension for related research in sentiment analysis and low-resource language processing, this guarantees reproducibility of results.

This study has some limitations even though it produced the biggest sentiment-labeled Gujarati news dataset and introduced a scalable hybrid annotation framework. Due to its limitation to the news domain, the dataset might not accurately represent sentiment expressions present in product evaluations or social media. Despite a 60% reduction in human labor, the hybrid annotation procedure still needs a seed dataset that has been manually classified. Additionally, the potential of deep learning architectures was not investigated because trials were restricted to classical machine learning models. In order to improve contextual understanding, future work will concentrate on expanding the dataset to multiple domains, investigating transformer-based models (e.g., IndicBERT, mBERT, XLM-R), and looking into more complex tasks like aspect-based sentiment analysis and emotion classification for Gujarati and other low-resource languages.