Multi-rater evaluation systems are fundamental to numerous domains where subjective judgments must be aggregated reliably such as medical diagnosis, academic peer review, and crowdsourced data annotation. A central challenge emerges when objective ground truth is unavailable: how to assess individual rater reliability and construct consensus estimates that are statistically robust and computationally efficient?

Recent advances in Large Language Models (LLMs) have introduced a new paradigm—using multiple models as automated raters for semantic evaluation tasks [1, 2]. While the “LLM-as-a-Judge” paradigm offers scalability, it inherits the core challenge of validating consensus and model reliability without ground truth.

This problem extends beyond LLM evaluation to any scenario where multiple evaluators of varying quality must be aggregated. Traditional approaches assume either (1) availability of ground truth for calibration, or (2) equal reliability across all raters. Both assumptions are frequently violated in practice. When ground truth exists, the problem reduces to supervised learning; our work addresses the more challenging scenario where it does not.

Mathematical Problem Formulation. Given M evaluation targets and N raters, we observe an evaluation matrix , where represents rater j's score for target i on a bounded ordinal scale. In the absence of ground truth, we must: (1) estimate a consensus score x̃_i for each target i, (2) quantify each rater's consistency with this consensus, and (3) select an optimal reference set that balances reliability and computational cost.

Methodological Contributions. We develop a statistical framework that addresses these challenges through:

Unlike approaches that rely on mean aggregation and treat ICC solely as a terminal summary without budget-aware core selection, this work integrates a robust median-based consensus (50% breakdown point), multi-metric consistency profiling (R², variance of deviations, Spearman rank), and an explicit cost-constrained core optimization (Eq. 1). Beyond empirical validation, we state theoretical properties − robustness of median and ICC monotonicity for nested reference sets − yielding predictable behavior along the ‘3→5→9 models’ trajectory under minimal rank-correlation conditions for added raters. This integration of consistency, cost, and theory enables practical design of small model cores that retain ‘excellent’ ICC while offering a controlled quality–cost trade-off for scalable annotation without ground truth.

Empirical Validation Domain. We validate the framework on semantic similarity for automated accessibility testing, a task where traditional systems fail and human annotation is prohibitively expensive [3–6]. Specifically, we examine WCAG 2.5.3 (Label in Name) compliance, where visible labels must semantically correspond to accessible names − a task that defies simple string matching and can involve cases ranging from benign clarifications to critical contradictions, including potential Accessibility Cloaking Attacks [7]. LLMs have demonstrated promising capabilities in accessibility-related tasks including alternative text generation, content simplification, and interface evaluation [8–13], yet their deployment faces practical obstacles: high computational demands, API costs, latency, and provider dependency [11]. Our framework addresses these challenges by identifying minimal model cores that maintain statistical reliability while reducing computational requirements by up to 67%. Using 17 LLMs as automated raters on ~ 14,384 synthetic English/Ukrainian text pairs, we evaluate semantic correspondence on a [-1,1] scale. While the application domain involves web accessibility, the statistical methodology is domain-agnostic.

Generalizability. The framework generalizes to diverse NLP tasks requiring consensus without ground truth, including sentiment analysis, translation quality assessment [14], and content moderation scoring [15].

The mathematical formulation (Section 1.3) remains invariant across domains, requiring only ordinal outputs and scale boundedness. For categorical outputs, extensions using Fleiss' κ and weighted consensus schemes are straightforward. The remainder of this paper is organized as follows: Section 1 presents the methodology including dataset construction (1.1), evaluation protocol (1.2), theoretical properties (1.3), and consistency metrics (1.4); Section 2 reports empirical results; Section 3 discusses implications and limitations; Section 4 concludes with methodological generalizations.

The analysis of LLM consistency in the task of semantic similarity assessment was conducted based on the metrics described in the methodology.

Cross-correlation and Selection of the Reference Core of Models. Figure 1 shows cross-correlations identifying a stable core {1–9}. Notably, intra-developer correlation did not exceed inter-developer correlation, indicating architectural diversity.

Inter-model Reliability (ICC). When analyzing the group of 9 most consistent models, the calculated intraclass correlation coefficient (ICC) using the Average Random Raters scheme (ICC2k) was 0.977 with a 95% confidence interval [0.96, 0.98] (after rounding), using 86.4% of data without missing values. The obtained value indicates a high level of inter-model consistency, which is classified as “excellent” reliability.

Separately, an optimal core (based on the balance of quality indicators and cost) of three models (anthropic/claude-3.5-haiku, deepseek/deepseek-r1, and google/gemini-2.5-flash-preview) was identified, for which the ICC was 0.955 (CI95% [0.95, 0.96]) based on 88.1% of values without missing data. This is also a very high indicator, while such a core requires significantly fewer computational resources. Although the 3-model core's ICC was slightly lower (0.955 vs. 0.977), its reliability remained "excellent," making it a computationally efficient choice for resource-constrained scenarios.

These values exceed the 0.90 threshold for “excellent” reliability and empirically validate Proposition 2: the 9-model core includes the 3-model optimized core (), with as predicted

Qualitative Indicators of Models Relative to the Reference Core. Using the defined core of three models to compute the consensus, the relative quality of other models was evaluated. Table 2 demonstrates the indicators of evaluation bias (), “model noisiness” (σ²_j), and consensus consistency (R²).

For this table with both negative and positive values show systematic deviation from the core value (model bias), σ²ⱼ shows unsystematic deviation ("noise"). Models with low or negative R² represent a "minority report" or a fundamentally different interpretation of the task. While not suitable for the reference core, analyzing their outputs can provide insights into ambiguous cases or alternative semantic perspectives.

The results show that more powerful models tend to demonstrate better consistency with the consensus and less “noisiness”. Negative R² values for models gpt-4.1-nano and ministral-8b indicate that their predictions deviate from consensus more than a naive constant baseline, confirming systematic misalignment. These models should be excluded from the reference set per the optimization criterion in Eq. (1). The strong negative correlation (Pearson ρ = -0.83) between variance σ²ⱼ and R²ⱼ across all models demonstrates that consistency with consensus inversely relates to evaluation noise: lower-variance models achieve higher explanatory power.

Our findings confirm that LLMs can effectively automate semantic similarity evaluation for the WCAG 2.5.3 criterion. The developed taxonomy of semantic changes and the methodology for generating synthetic data using LLMs allowed for the creation of relevant datasets for further analysis. While manual verification of every generated instance is impractical, the framework's reliance on synthetic data is methodologically robust. By ensuring that our dataset covers a comprehensive taxonomy of 5 classes and 10 subclasses of inconsistencies, we systematically account for a wide range of potential failure modes. The adequacy of this approach is validated through the inter-class comparison of mean reliability scores, which allows us to confirm that the framework correctly differentiates between varying types and severities of rater disagreement. This structured, variance-aware evaluation serves as a proxy for correctness in the absence of absolute ground truth.

The use of a semantic similarity scale in the range [-1.0..1.0] proved effective for differentiated assessment: it allows not only to determine the degree of semantic proximity but also identifying cases of opposite meaning. This is critically important for identifying potentially dangerous or disorienting discrepancies and can be considered as an indirect indicator of the presence of Accessibility Cloaking Attacks where content is presented differently to users and assistive technologies.

The LLM-as-a-judge approach and the calculation of the intraclass correlation coefficient (ICC) confirmed a high degree of consistency among leading LLMs in assessing semantic similarity on the proposed scale. This suggests that LLMs can be used as a reliable tool for data annotation in cases where manual labeling is too resource-intensive. The ability of models with a large number of parameters to demonstrate strong cross-correlation may indicate that, despite differences in architecture and training data, they perform natural language processing tasks with similar quality, and their responses tend toward the true value.

An important observation is that a model's ability to generate syntactically correct responses (as shown by the high percentage of successful generations in Table 1 for many models) does not always directly correlate with the semantic quality and consistency of its evaluations. For example, the amazon/nova-micro-v1 model demonstrated high reliability in forming responses; however, its semantic consistency with the reference core was lower compared to some other models with similar or even lower generation success (Table 2). This highlights the need to distinguish a model's formal operability from its semantic analysis quality.

Despite the framework's success, it is important to acknowledge the practical limitations of using LLMs, such as high computational requirements and API-related costs, which remain obstacles for widespread, real-time implementation. While this research establishes a solid baseline for quality assessment, the results also highlight the need to develop more resource-efficient solutions.

Methodological Generalizability. While demonstrated on WCAG 2.5.3 semantic similarity, the proposed framework applies to any ordinal-scale multi-model evaluation task: (1) Sentiment analysis consensus [-1 to 1], (2) Text quality scoring [0 to 1], (3) Translation adequacy rating [1 to 5], (4) Any NLP annotation requiring inter-rater reliability. The mathematical formulation (Eq. 1) remains invariant to domain, requiring only: ordinal model outputs, scale boundedness, reference set selection criteria. Future work includes extending the framework to handle categorical (e.g., via Fleiss' κ) and multimodal (e.g., joint text-image) evaluation tasks.

This study demonstrated that LLMs can reliably evaluate semantic similarity for web accessibility, specifically for the WCAG 2.5.3 criterion. This research yielded the following significant outcomes:

1) A formalized statistical framework for consensus estimation and reliability assessment in multi-model LLM systems, with theoretical guarantees (Propositions 1–2) and optimal core set selection (Eq. 1).

2) The optimized 3-model core reduces the number of required API calls by 67% (from 9 to 3 models) compared to the full core, while the ICC2k decreases by only 2.2% (from 0.977 to 0.955), demonstrating favorable cost-quality trade-off.

3) The methodology generalizes to multi-rater consensus problems in NLP, enabling scalable annotation and model comparison without ground truth.

Despite the demonstrated potential, the use of LLMs is associated with high computational costs and expenses. The obtained results created a foundation for further research aimed at developing more economical and efficient solutions.

Research context. This research was conducted as part of the research project of Sumy State University titled “Information technology models and methods for analysis and synthesis of structural, information and functional models, objects and automated processes” (state registration number 0120U103071).