Trustworthiness of large language models can be approached from two complementary points: as a set of intrinsic socio-technical properties sustained by institutional safeguards, and as a socio-psychological judgement - perceived trustworthiness.

Socio-technical. At the system and institutional levels, socio-technical dimensions represent a validated and auditable foundation for determining whether a system can be considered trustworthy. Recently proposed TrustLLM framework offers the most comprehensive attempt to date to consolidate these socio-technical dimensions into an integrated evaluative structure. Synthesising insights from more than 600 studies, it identifies eight dimensions that define the trustworthiness of LLMs: truthfulness, safety, fairness, robustness, privacy, machine ethics, transparency, and accountability (Huang et al., 2024) (Fig. 1). TrustLLM provides conceptual clarity of LLM trustworthiness from as a system and also grounds each trustworthiness dimension in empirical evaluation, using over 30 benchmark datasets and systematically testing 16 mainstream LLMs. In doing so, it translates broad normative and technical concerns into measurable outcomes that can be compared across systems, establishing both the scope and the limits of what can be known about the trustworthiness of current LLMs. Yet, while these dimensions serve as objective criteria, they are limited by the extent to which users perceive them in everyday interactions. In other words, benchmark scores and audit results establish the warrant for trust, but what actually guides user behaviour is the uptake of that warrant - their perceived trustworthiness of LLMs. When objective properties and user perceptions converge, reliance is well-calibrated; when they diverge, patterns of over-trust or under-trust emerge. In this sense, the framework delineates the structural grounds of trustworthiness but leaves open the equally important question of how those grounds are taken up psychologically by users in real interactions with LLMs.

Socio-psychological. From psychological perspective, perceived trustworthiness refers the beliefs and expectations people develop about whether the necessary conditions for trustworthy performance are in place (Alarcon et al., 2016). These beliefs are influenced by prior experiences (Schwerter and Zimmermann, 2020), contextual cues (Van Der Biest et al., 2025), and institutional assurances (Li et al., 2024), and it is ultimately these perceptions, rather than formal audit results in a system as trustworthy or not. Crucially, many of these beliefs are implicit: they are formed automatically, outside conscious awareness, yet guide behaviour in systematic ways (Cyrus-Lai et al., 2022). Psychological models of dual-process cognition (Hochman, 2024) and the heuristic-systematic model (Chaiken and Ledgerwood, 2012) show that such implicit evaluations can serve as fast heuristics for action influencing whether users accept or contest a system, often before deliberate reasoning intervenes (Forscher et al., 2019). The content of these judgements is influenced by factors such as repeated interaction with a system (Glickman and Sharot, 2024), prevailing cultural narratives (Tao et al., 2024), design features that signal safety or authority (Kostick-Quenet and Gerke, 2022), and the broader regulatory environment (Kattnig et al., 2024).

The socio-psychological perspective therefore emphasises that perceived trustworthiness emerges from a combination of implicit and explicit judgements. A substantial body of psychological and organisational research shows that these judgements consistently centre on a set of recurring attributes that people use to evaluate whether an agent, institution, or system is trustworthy (Daronnat et al., 2021). One of the most fundamental of these is truthfulness, often conceptualised as integrity, honesty, or accuracy (David et al., 2015a),(David et al., 2015b). Across models of interpersonal and organisational trust, integrity is identified as a core antecedent of perceived trustworthiness, with clear effects on willingness to rely on others or on systems (Mayer et al., 1995b),(Colquitt et al., 2007) while evidence of deception or unreliability rapidly undermines it (Levine, 2014). In this sense, truthfulness constitutes a foundational dimension of perceived trustworthiness, anchoring people’s intuitive expectations about whether a system is worthy of reliance.

Beyond truthfulness, perceived trustworthiness also depends on how information is collected, stored, and protected. In interpersonal settings, respecting confidentiality is central to maintaining trust, whereas breaches of privacy often irreparably damage it. In digital environments, privacy has been formalised in theories such as the privacy calculus model, which proposes that individuals balance anticipated benefits of disclosure against perceived risks of data misuse, with trust mediating this trade-off (Culnan and Armstrong, 1999). Empirical studies consistently reported that when users believe their data are safeguarded, they are more likely to form trusting beliefs, disclose information, and continue engagement (Xu et al., 2009). Conversely, perceived risks of surveillance undermine trust and trigger avoidance behaviours (Malhotra et al., 2004). Thus, privacy protection shapes perceived trustworthiness by assuring users that their personal information will not be misused or exposed. Importantly, perceptions of privacy often intersect with concerns about fairness, since individuals evaluate not only whether their data are secured, but also whether data practices and outcomes are distributed and applied in an equitable manner across users and groups.

Perceptions of fairness have long been recognised as central to the psychology of trust. Organisational justice theory distinguishes between distributive fairness (the equity of outcomes), procedural fairness (the impartiality and consistency of decision-making processes), and interactional fairness (the quality of interpersonal treatment), with all three dimensions shown to predict perceived trustworthiness of authorities and institutions (Colquitt et al., 2001),(Colquitt et al., 2013). In digital and automated decision-making environments, fairness is similarly salient: users are more likely to trust systems they perceive as unbiased and equitable in treatment or outcomes (Lee, 2018). Violations of fairness, whether through biased outputs, inconsistent processes, or unequal access, consistently undermine perceived trustworthiness and reduce willingness to rely on a system (Grgic-Hlaca et al., 2018). In this sense, fairness operates as a core evaluative lens through which individuals interpret whether trustworthy performance is present, complementing privacy by extending concerns from the security of personal information to the equitable treatment of persons and groups.

The concept of fairness perception does not exist in isolation but often interact with broader concerns about harm and protection. When outcomes are perceived by individuals as biased or unjust, this can amplify doubts about a system’s ability to operate safely, since unfair treatment is frequently interpreted as a risk to personal or collective well-being. In psychological and socio-technical models of trust, perceptions of safety are closely linked to risk assessment and vulnerability. Classic trust definitions emphasise that trust entails a willingness to accept vulnerability in situations of uncertainty (Mayer et al., 1995c). The Trust-Confidence-Cooperation (TCC) model further specifies that trust reduces perceived risk and supports cooperative behaviour in domains such as environmental management and health communication (Earle and Siegrist, 2008). Recent psychological research further supports the centrality of perceived safety to trust and trustworthiness. A multidimensional model of perceived personal safety argues that safety confidence which represents the belief in one’s ability to remain protected or escape harm is a component influencing overall safety judgements (Syropoulos et al., 2024). Rooted in classic notions of the “fight or flight” response (Cannon, W. B., 1932), this work shows that perceived safety is not only about the absence of external threats but also about the confidence that one can withstand or avoid them. Such perceptions are integral to trustworthiness because they determine whether users view a system as exposing them to manageable or unacceptable risks, and they form the basis upon which expectations of robustness are built.

Interestingly, in person perception research, trust depends partly on predictability and consistency of perceived actions and behaviours (i.e., the expectation that behaviour will generalise across time and situations (Rempel et al., 1985). Classic attribution theory argues that when outcomes are consistent and caused by stable, competent dispositions, observers infer reliability and form trusting beliefs (Weiner, 1985). In parallel, moral-social judgement theories show that trust also hinges on accountability - the sense that agents are answerable for outcomes and can give adequate reasons (Tetlock, 2002). Experimental work on trust repair likewise finds that appropriate accounts (apology, explanation, or remedy) can restore perceived trustworthiness after failures, especially when the violation is competence-based (Huo et al., 2022). Together, these literatures indicate that perceived trustworthiness increases when an agent (or system) performs reliably (robustness) and remains answerable for what happens (accountability).

For perceived trustworthiness, it is not enough that reliability and accountability exist; they must also be perceptible to others. Transparency provides this perceptual bridge by reducing the psychological costs of uncertainty and suspicion while enhancing the rewards of confidence and predictability, thereby enabling robustness and accountability to be recognised as trustworthy qualities. Classic and modern psychological theories of interpersonal perception show that observability is critical: when people perceive others as open, their uncertainty diminishes and confidence in the other’s intent increases (Wheeless and Grotz, 1977),(Yang et al., 2021). In contrast, perceived secrecy or ambiguity raises anxiety and suspicion, degrading perceived trustworthiness (Larzelere and Huston, 1980). In addition, the Anxiety-Uncertainty Management (AUM) theory posits that reducing uncertainty through transparent signals is essential to maintaining confidence in social interaction where transparent communication serves to manage anxiety and ambiguity (Neuliep, 2017), thereby stabilising perceived trustworthiness. For instance, transparent communication of health guidelines during the COVID-19 pandemic reduced public anxiety and increased trust and compliance with protective behaviours (Lee and Li, 2021). Thus, transparency operates as a psychological mechanism that allows qualities such as reliability and accountability to be recognised and assessed. By reducing uncertainty and anxiety, it creates the conditions under which these structural features are interpreted as evidence of perceived trustworthiness.

Taken together, the evidence indicates that perceived trustworthiness is a socio-psychological construct through which people organise diverse cues such as truthfulness, privacy, fairness, safety, robustness, accountability, and transparency into an overall expectation of whether reliance is justified (Fig. 1). Because perceived trustworthiness functions as an overarching judgement that integrates multiple cues, advancing the field now requires instruments that can render this latent construct measurable from the human perspective. A validated measure would provide three core benefits. First, it would allow systematic analysis of how people perceive trustworthiness across its key dimensions, including how judgments vary across individuals, contexts, and tasks. Second, it would enable perceived trustworthiness to be transformed from a diffuse intuition into a construct that can be compared, modelled, and linked to behaviour, making it possible to test whether reliance is appropriately calibrated or misplaced, and to identify conditions that lead to over- or under-trust. Third, it would establish perceived trustworthiness as a multidimensional construct that can be tracked over time, allowing researchers to quantify its impact on behaviour, evaluate its stability and change, and generate cumulative evidence across studies and populations.

For large language models, the need for such measurement is especially important. Their outputs are probabilistic, open-ended, and often presented with a fluency that can obscure underlying uncertainty. Users must rely on perception to decide whether to accept, verify, or reject responses, and miscalibration in these judgments can have direct consequences. LLMs are also used across highly diverse contexts from casual information seeking to professional and educational settings where perceptions of trustworthiness may differ substantially. Without instruments that capture how people perceive trustfulness, privacy, fairness, safety, robustness, transparency, and accountability in these systems, it is not possible to determine whether reliance is proportionate, whether cues are interpreted as intended, or whether safeguards are effective.

The psychometric evaluation of the PT-LLM-8 scale was conducted in several steps. First, we examined the latent structure of the scale through exploratory and confirmatory factor analyses. Next, we assessed reliability using indices such as internal consistency, composite reliability, convergent validity, item-level diagnostics, and measurement invariance. Finally, we evaluated validity in relation to external criteria, including personality characteristics, LLM competence, and LLM usage characteristics. Figure 1 illustrated these steps.

Data Preprocessing. After initial inspection and removing participants who did not complete or failed attention checks, 752 participants were included in the final analysis (Mean age = 28.58, SD = 6.11, 50.3% males, 48.8% females, 0.9% missing responses for gender).

Descriptive statistics. The mean, standard deviation, and distribution of participants’ responses were calculated for each item to examine central tendency, variability, and overall response patterns.

Item quality check procedures. Before conducting the main analyses, we evaluated item quality by examining the assumption of monotonicity. This assumption requires that respondents with higher levels of the latent construct (e.g., perceived trustworthiness of LLMs) are more likely to endorse higher response categories. Assessing monotonicity in the current study served two purposes. First, it strengthened our reliability and dimensionality evaluations, as violations may indicate that an item does not operate consistently across the trait continuum. Second, because such violations can signal multidimensionality or measurement error, early detection allows problematic items to be identified before they affect further analyses. The assessment was implemented using the mokken package in R. For each item, we obtained a scalability coefficient (H), the proportion of violated monotonicity comparisons, the maximum standardized violation (zmax), and an indicator flag when violations exceeded critical thresholds.

Exploratory Factor Analysis. The purpose of this analysis was to examine the latent dimensionality of the trust items and to identify an interpretable factor structure that could be cross-validated in an independent subsample using confirmatory factor analysis (CFA). The dataset (N = 752) was randomly divided into two independent subsamples of equal size. The first subsample (n = 376) was used for exploratory factor analysis (EFA), while the second was reserved for CFA. This split-sample design ensured that the factor structure identified in the EFA could be validated on an independent dataset.

Prior EFA, sampling adequacy was assessed using the Kaiser-Meyer-Olkin (KMO) statistic, which was computed both at the overall scale level and for individual items. Bartlett’s test of sphericity was applied to confirm that the observed correlations differed significantly from an identity matrix, supporting the suitability of the data for factor analysis

To determine the appropriate number of factors to retain, several complementary criteria were applied. First, a scree plot of eigenvalues was inspected to identify points of inflection. Second, Horn’s parallel analysis was performed, comparing empirical eigenvalues to those generated from random data under maximum likelihood (ML) extraction. Third, a non-parametric bootstrap procedure with 2,000 resamples was implemented to compute mean eigenvalues and 95% confidence intervals, providing an additional robustness check on factor retention decisions

Factor extraction was performed on the item correlation matrix using maximum likelihood estimation, which allows the computation of model fit indices and is robust under the assumption of multivariate normality. Competing solutions specifying one, two, and three latent factors were estimated. To allow for correlated dimensions, an oblimin rotation was applied. For each model, rotated factor loadings, item communalities, and uniqueness values were reported. When multiple factors were specified, factor correlation matrices (Φ) were also estimated.

Model fit was evaluated using a range of indices, including the chi-square goodness-of-fit test, degrees of freedom, p-value, the root mean square error of approximation (RMSEA), the TuckerlLewis Index (TLI), the root mean square residual (RMSR), and the Bayesian Information Criterion (BIC). These indices provided information on both absolute and comparative model fit, enabling the identification of the most parsimonious and theoretically interpretable factor structure.

Confirmatory factor analysis (CFA). The purpose of this analysis was to test the fit of a one-factor model of perceive trustworthiness of LLMs, identified in the exploratory phase, on an independent subsample. Analyses were conducted in R using the lavaan package, with descriptive statistics and correlation diagnostics computed in psych. Items were recoded as numeric, and rows containing only missing responses were removed. Pairwise correlations and Mardia’s coefficients were inspected to evaluate distributional assumptions.

A single latent factor representing Perceived trustworthiness of LLMs was defined by the eight observed trustworthiness items. The primary model was estimated using robust maximum likelihood with robust (Huber–White) standard errors and a scaled test statistic (MLR). Missing data were handled with full information maximum likelihood (FIML). The latent factor variance was fixed to 1 for identification (std.lv = TRUE). Global model fit was assessed using the chi-square test, comparative fit index (CFI), Tucker–Lewis index (TLI), root mean square error of approximation (RMSEA with 90% CI), standardized root mean square residual (SRMR), Akaike Information Criterion (AIC), and Bayesian Information Criterion (BIC). Both conventional and robust versions of the indices were reported where available. Standardized factor loadings, standard errors, z-values, and R² values were computed for each item. Residual correlation matrices and standardized residuals were examined. Modification indices were inspected (threshold ≥ 10) to identify potential sources of misfit. Checks for Heywood cases (negative residual variances) were also performed.

To assess stability of parameter estimates, a nonparametric bootstrap procedure was implemented on complete cases. A total of 1,000 bootstrap resamples were drawn, with models estimated by ML in each resample. For each item, distributions of standardized factor loadings and R² values were summarized with means, medians, and 95% percentile confidence intervals. Distributions of key fit indices (CFI, TLI, RMSEA, SRMR) were likewise summarized across bootstrap samples.

Reliability analyses. The aim of this analysis was to evaluate the reliability and internal consistency of the Perceived Trustworthiness of LLM Scale (PT-LLM-8) using the full dataset (N = 752). Cronbach’s α was computed for the entire eight-item scale on the full sample to evaluate internal consistency. Item-total correlations and α-if-item-deleted statistics were examined to assess the contribution of each item to the scale. Composite reliability (CR) was calculated using standardized factor loadings and measurement error variances from the CFA model, treating all available data. CR accounts for differential item contributions and provides a more robust estimate of reliability than α. Average variance extracted (AVE) was computed from CFA estimates. An AVE value of 0.50 or greater was taken to indicate that the latent factor explained more than half of the variance in its indicators, supporting convergent validity. Standardized residuals, squared multiple correlations (R²), and modification indices were inspected across the whole sample to identify items with weaker loadings, lower communalities, or localized misfit.

Measurement Invariance Analysis. To further support the validity of the PT-LLM-8 scale, measurement invariance tests were conducted to stablish the construct structure is equivalent across male and female groups. These tests employed multigroup confirmatory factor analysis (MG-CFA) to assess whether the factor structure of PT-LLM-8 was consistent between genders. The process involved fitting a series of nested models with increasing constraints: starting with configural invariance, where no restrictions were applied and parameters were freely estimated for both groups; followed by metric invariance, which constrained factor loadings to be equal; then scalar invariance, which additionally constrained item intercepts; and finally, strict invariance, where loadings, intercepts, and residual variances were all constrained to be equal across groups.

Model invariance was determined by comparing fit indices across these models, with invariance accepted when changes in CFI were less than 0.01, and changes in RMSEA less than 0.015, based on guidelines from (Chen, 2007). Model fit was evaluated against the specified criteria, and the comparison between models focused on differences in fit indices. For example, a significant decline in fit, exceeding the cutoff thresholds, indicated a lack of invariance. Furthermore, the chi-square difference test was used to compare the models. If the resulting p-value exceeds the significance level of α = 0.05, we conclude that there is no significant difference between the models, and the more constrained model is considered acceptable.

The models were estimated using maximum likelihood with robust standard errors (MLR) to account for potential multivariate non-normality. These analyses were performed using R 4.4.2 (R Core Team, 2022) with the lavaan package (v0.6-19).

External Validation. We conducted external validation for the PT-LLM-8 scale by examining the relationships and effects of several theoretically relevant variables on the perceived trustworthiness of LLM scores. Predictor variables included self-efficacy, self-esteem, and the Big Five personality traits, Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, chosen based on their theoretical relevance to trust and technology adoption.

A multiple linear regression model was fitted using ordinary least squares (OLS) as the estimation method to evaluate the influence of the predictor variables on perceived trustworthiness. The significance of each predictor was assessed through t-tests. In this analysis, the dependent variable was the total score obtained by summing the responses to the items on the PT-LLM-8 scale.

Prior to the regression analysis, we examined the pairwise relationships among variables through Pearson correlation coefficients to ensure linearity and to detect potential multicollinearity issues. To assess multicollinearity, we used the common threshold of 0.9; correlations exceeding this value would indicate problematic multicollinearity among independent variables, which could distort the regression estimates (P.Vatcheva and Lee, 2016). Furthermore, we calculated the variance inflation factor (VIF) for each predictor to provide additional evidence of the absence of multicollinearity in the linear regression model. As a common guideline, VIF values exceeding 5 are considered indicative of problematic multicollinearity (O’brien, 2007). This process ensured that the assumptions underlying the regression analysis were adequately met, providing a robust external validation of the PT-LLM-8 scale.

Additional analyses. To further assess the external validity of the PT-LLM-8 scale, we conducted two additional analyses examining the relationship between LLM usage characteristics, self-reported helpfulness of LLMs for personal and professional tasks and user’s competence with LLMs. We tested two multiple regression models. The first model included four predictors: LLM competency, perceived helpfulness in personal and professional contexts, and frequency of LLM use. The second model focused on characteristics of LLM usage, including method of interaction (text, voice, both text and voice, or other), the device primarily used to access the LLM (PC, mobile phone, tablet, smart speaker such as Amazon Echo or Google Nest, dedicated AI device such as Humane AI Pin or Rabbit R1, or other), and the location where the individuals access the LLM (stationary settings (e.g., home, office, school), on the move (e.g., commuting, travel), both stationary and mobile, or other).

We employed the ordinary least squares (OLS) estimation method to compute regression coefficients and evaluate the significance of each predictor. To ensure the validity of the regression models, we assessed multicollinearity among the numerical predictors in the first model by calculating the variance inflation factor (VIF). In our analysis, all predictors demonstrated VIF values below this threshold, indicating that multicollinearity was not a concern.

All analyses in the present study were conducted using R (Version 2024.12.1 + 563). Codes for these analyses are available on OSF (https://osf.io/ku5sz/?view_only=794eea00b5f149b49b874c385c07651c).

The mean, standard deviation, skewness, and kurtosis of each item of the PT-LLM-8 scale were assessed to examine the distribution of responses and identify any potential deviations from normality. Spearman’s rank correlation coefficients were calculated to examine the relationships between the items of the scale. Table 2 presents the descriptive statistics and correlation coefficients across the items of the PT-LLM-8 scale.

The item means ranged from 5.91 to 7.95, with standard deviations between 1.71 and 2.68. The items “handle data securely” and “good across topics” had the lowest and highest mean scores, respectively. The absolute values of skewness for all items ranged from 0.43 to 1.27 and kurtosis from .08 to 2.15. All correlations between the items of the PT-LLM-8 scale are significant at the p < .001 level, indicating strong positive relationships among the different aspects of trustworthiness.

The aim of the exploratory factor analysis (EFA) was to examine the latent structure of the eight-item Perceived Trustworthiness scale, in order to test whether the items are best represented by a single common factor. The analysis was conducted on the Spearman correlation matrix using maximum likelihood (ML) extraction. This approach was chosen to reduce the influence of non-normality in item distributions, while ML extraction provides likelihood-based indices of model adequacy.

Preliminary diagnostics indicated that the data were highly suitable for factor analysis. The Kaiser-Meyer-Olkin (KMO) statistic was .93, with individual item MSAs ranging from .92 to .94, exceeding the recommended .80 threshold for meritorious adequacy. Bartlett’s test of sphericity was highly significant, χ²(28) = 1860.29, p < .001, N = 376, rejecting the null hypothesis of an identity matrix and confirming that the items shared substantial common variance.

The eigenvalues of the Spearman correlation matrix revealed a single dominant factor. The first eigenvalue was 5.15, accounting for 64.3% of the total variance. All subsequent eigenvalues were well below 1.0 (λ₂ = 0.67, 8.4%; λ₃ = 0.53, 6.6%; λ₄ = 0.39, 4.9%), indicating they explained only minor amounts of variance. The scree plot displayed a sharp break after the first factor, followed by a long flat tail, consistent with unidimensionality (Fig. 2). Parallel analysis supported this conclusion. The observed first eigenvalue (4.75) far exceeded the simulated random mean (0.60). By contrast, the second observed eigenvalue (0.21) was only slightly larger than the random mean (0.14) and fell below the 95th percentile benchmark. The third and higher eigenvalues were smaller than their simulated counterparts (e.g. observed = 0.10 vs random = 0.10), providing no evidence for additional factors.

The 1-factor ML solution produced strong, clean loadings across all items (|λ| range = 0.663–0.851, median = 0.774 (Table 3). Communalities were generally high (h² range = 0.440–0.725, median = 0.599 (Table 2), indicating that the single factor captures a substantial proportion of each item’s variance. Global fit for the 1-factor model was: χ²(20) = 91.71, p < .001; RMSEA = .098; TLI = 0.945; RMSR = 0.041; BIC = -26.88. Although the chi-square test was significant and RMSEA approached .10, such indices are known to be overly sensitive to large samples and low degrees of freedom and therefore should not be treated as definitive indicators of misfit (Schermelleh-Engel, K., Moosbrugger, H., & Muller H., 2003),(Kenny et al., 2015). Taken together, these results show strong item saturation, adequate communalities, and converging retention evidence justify the interpretation of the Trustworthiness scale as essentially unidimensional, consistent with best-practice recommendations for EFA (Costello and Osborne, 2005).

The aim of the confirmatory factor analysis (CFA) was to test a unidimensional measurement model for the eight-item Trustworthiness scale in the UK independent subsample (N = 376), using robust maximum likelihood with full information maximum likelihood (MLR with FIML) estimation.

The one-factor model showed good fit on robust indices: χ²(17) = 63.59, p < .001, CFI.robust = 0.96, TLI.robust = 0.93, RMSEA.robust = 0.09 (90% CI [0.08, 0.13]), and SRMR = 0.03. The robust CFI exceeded the conventional 0.95 benchmark, and the SRMR indicated very small average residuals. Although the RMSEA was near 0.10, this value requires cautious interpretation because RMSEA is biased upward in models with few degrees of freedom (here df = 17). Simulation studies show that RMSEA can “over-reject” correctly specified low-df models, making fixed cut-offs inappropriate in such contexts (Marsh et al., 2004),(Kenny et al., 2015),(Groskurth et al., 2023). Evaluating model fit holistically and given strong theoretical expectations and EFA evidence of unidimensionality, the model was judged acceptable.

At the item level, all standardized factor loadings were significant (z ≥ 11.3, p < .001) and substantial in size (Table 4). Loadings ranged from 0.646 (listen to feedback and address it) to 0.851 (factually accurate), with a median of 0.76. The factor explained a sizeable proportion of variance in each item, with R² values between 0.42 and 0.57 (median 0.50). The complementary uniqueness values (1 − R²) confirm that residual variance was moderate and well within expectations, with no Heywood cases.

Robust estimation was complemented by nonparametric bootstrapping (1,000 resamples, complete cases), which yielded closely aligned results (Table 4). Across all items, bootstrap confidence intervals were narrow, never crossed zero, and converged closely with robust point estimates. This convergence demonstrates that the factor solution is stable, replicable, and not influenced by estimation method.

Because the RMSEA estimate in the adjusted one-factor model was 0.09, a Monte Carlo simulation study was conducted to evaluate whether this value reflected substantive model misfit or sampling variability inherent in low-degrees of freedom models. The simulation generated 2,000 synthetic datasets from the fitted adjusted CFA model (N = 376) under the assumption that this model represented the true population structure. Each dataset was re-estimated using MLR with the same model specification, and robust fit indices were recorded.

Across replications, the median robust RMSEA was .06 (95% range ≈ .05-.08), with CFI and TLI consistently high (median CFI = .97, TLI = .95). The observed RMSEA of .09 in the empirical data fell near the upper tail of this simulated distribution, indicating that such values are expected by chance even when the model is correctly specified. In contrast, when the restricted one-factor model without residual covariances was fitted to the same simulated datasets, median RMSEA increased to .11 and CFI dropped to .91, demonstrating that omitting substantively justified residual correlations produces systematic deterioration in model fit.

Sensitivity analyses varying the sample size (N = 200–750) further showed the instability of RMSEA in smaller samples. At N = 376, the distribution of simulated RMSEA values remained wide, occasionally producing estimates near .09-.10 despite the model being correctly specified. Larger sample sizes narrowed the RMSEA distribution but retained a slight positive bias relative to CFI and SRMR.

Overall, the Monte Carlo results confirm that the elevated RMSEA observed in the empirical data does not indicate serious misspecification but reflects the known upward bias of RMSEA in low-df models with moderate sample sizes. This finding supports the interpretation of model fit holistically, with stronger emphasis on the robust CFI and SRMR, which consistently indicated good fit, and aligns with prior theoretical expectations and EFA evidence of unidimensionality.

We evaluated internal consistency for the eight-item PT-LLM-8 scale using the full dataset (N = 752). Cronbach’s alpha computed from the covariance matrix was α = 0.902, and the standardised alpha computed from the correlation matrix (i.e., as if all items had SD = 1) was αstd = 0.907. The small difference shows that high reliability is not an artefact of unequal item variances. The mean inter-item correlation was 0.550, indicating strong coherence without redundancy. Non-parametric bootstrapping confirmed the stability of α: the bootstrap 95% percentile interval was [0.889, 0.914].

Composite Reliability (CR) was 0.908, meaning about 91% of the variance in the optimally weighted composite is attributable to the common factor rather than measurement error. Average Variance Extracted (AVE) 0.552, indicating that, on average, over half of each item’s variance is explained by the latent factor which is consistent with satisfactory convergent validity.

Per-item diagnostics show uniformly good relationships with the scale total (Table 5). Crucially, neither α-if-deleted nor ωt-if-deleted suggested that dropping any single item would help: the largest α-if-deleted was 0.897 (which is lower than α = 0.902; Δ = -0.005) and the largest ωt-if-deleted was 0.902 (lower than ωt = 0.908; Δ = -0.006). Together the results demonstrate that all eight items contribute meaningfully, and none depress scale reliability.

To evaluate whether the PT-LLMs scale functions equivalently across gender groups, measurement invariance testing was conducted. This process assesses whether the scale's factor structure, item loadings, and intercepts are consistent between females and males, ensuring that any observed differences in scores are attributable to genuine variations in the underlying construct rather than measurement bias. The analysis involved a series of nested models, beginning with the configural model (M1), followed by the metric (M2), scalar (M3), and strict invariance models (M4), to determine the level of invariance supported by the data. The results of these tests are reported in Table 6.

The configural model demonstrated acceptable fit based on CFI = .957, TLI = .929, and SRMR = .035. The c² test was significant (c²(34) = 172.47; p < .001).

Next, metric invariance was tested by constraining factor loadings to be equivalent in the two groups. The metric model showed a good fit to the data: CFI = .956; TLI = .940; SRMR = .047. The c² test was significant (c²(41) = 172.47; p < .001) and RMSEA index was .096 which is above the conventional threshold of .08. After estimating the metric model, we compared it against the configural model. The results showed that the chi-square difference test was not statistically significant; Δχ² = 9.996, df = 7, p = .189. The absolute difference in CFI and RMSEA were small 0.01 and 0.015, respectively. This result suggests that after constraining the factor loadings to be equal across gender groups, the model fit did not change substantially. In other words, the constrained model (i.e., metric model) fits the data equally well.

After establishing metric invariance, scalar invariance was tested by constraining the item intercepts to be equivalent in the two groups of males and female. The constraints applied in the metric invariance model were retained. The model fit indices indicate a good fit for the scalar model: CFI = 0.957; TLI = 0.950; RMSEA = 0.088; SRMR = 0.047. The chi-square test for scalar model was significant (c²(48) = 185.49; p < .001). The comparison between metric and scalar models revealed minimal differences: the changes in CFI and TLI were less than 0.01, and the changes in RMSEA and SRMR were negligible, suggesting that the scalar invariance assumptions hold. Specifically, the non-significant change in fit between the scalar and metric models (Δχ²(7) = 3.027, p = .882) supports the presence of scalar invariance. This suggests that constraining the item intercepts across gender groups does not significantly influence the model fit, indicating that the PT-LLMs construct has equivalent factor loadings and intercepts for both men and women.

Finally, the strict model was conducted by further constraining the item residuals across males and females. The model demonstrated a good fit: CFI = 0.955; TLI = 0.955; RMSEA = 0.083; SRMR = 0.047. The chi-square test for strict model was also significant (c²(56) = 200.59; p < .001). Comparing the strict to scalar invariance model reveals that strict model did not worsen the fit (Δχ²(8) = 15.106, p = .057). Therefore, we can conclude that the PT-LLMs scale indicates strict invariance across the female and male groups.

Overall, these results indicate that the scale demonstrates configural, metric, scalar, and strict invariance across gender groups and constraining the factor loadings, item intercepts, and item residuals across groups does not significantly affect the model fit. These findings confirm that the PT-LLMs scale measures the same construct equivalently for males and females.

To assess the external validity of the PT-LLM-8 scale, we examined its associations with several theoretically relevant variables. Specifically, a linear regression analysis was conducted with the PT-LLM-8 score as the dependent variable and the following external variables as independent predictors: self-efficacy, self-esteem, extraversion, agreeableness, conscientiousness, neuroticism, and openness.

The absolute correlations among the external validators ranged from 0.012 to 0.608. We examined the presence of multicollinearity in the regression model using variance inflation factor (VIF). The VIF values for predictors ranged from 1.047 to 2.031, which are below the threshold of 5. Therefore, multicollinearity was not a concern in this analysis, and the regression estimates can be considered stable and reliable.

The regression model explained approximately 10.4% of the variance in PT-LLM-8 scores (R² = 0.104) by external variables, indicating a modest level of predictive power. The results of multiple linear regression fit are presented in Table 7.

Among the predictors, self-efficacy emerged as the strongest positive correlate of perceived trustworthiness in LLMs, suggesting that individuals with greater confidence in their own capabilities tend to view LLMs as more trustworthy. Two personality characteristics, agreeableness and conscientiousness, were also significantly positively associated with PT-LLM-8 scores, consistent with the notion that cooperative and dependable individuals are more inclined to attribute trustworthiness to LLMs. In contrast, extraversion was negatively related to PT-LLM-8, indicating that more outgoing individuals tended to rate LLMs as less trustworthy. The remaining predictors (self-esteem, neuroticism, and openness) did not show significant associations, although the directions of the effects for neuroticism and openness were negative.

Overall, these findings suggest that the PT-LLM-8 scale is meaningfully related to relevant external variables, supporting its validity as a measure of perceived trustworthiness in LLMs.

To further explore the factors influencing perceived trustworthiness of LLMs, two additional regression analyses were conducted. The first analysis examined whether self-report of user’s competence with LLMs, LLMs’ help in personal and professional contexts, and frequency of LLMs use could predict PT-LLM-8 scores. The second analysis investigated whether contextual variables such as location of use, device type, and interaction method serve as significant predictors of trustworthiness perceptions. The results of both regression models are reported in Table 8.

The regression model explained approximately 24.3% of the variance in PT-LLMs scores, as indicated by the R² value of 0.243. In the first set of predictors, LLMs’ help in personal contexts was the strongest predictor for perceived trustworthiness of LLMs. This suggests that individuals who seek LLMs’ help in their personal lives tend to have a higher perception of LLMs trustworthiness. Competency in using primary LLM was also significantly positively related to PT-LLM-8, indicating that individuals who are more competent in using their primary LLM consider LLMs to be more trustworthy. LLMs’ help in professional tasks also showed a strong positive relationship with PT-LLM-8, suggesting that users who rely on their primary LLM for assistance in their professional tasks are more likely to perceive LLMs as trustworthy. The frequency of LLMs use demonstrated a non-significant association with PT-LLM-8; however, it had a positive impact on PT-LLM-8 which indicates that individuals who use LLMs more frequently, tend to have higher perception of LLMs trustworthiness. For checking the multicollinearity among predictors, we examined the VIF values which ranged from 1.15 to 1.35. This suggests that there is no problematic multicollinearity among the predictors. Overall, based on the results of the first model, competency in using primary LLM, along with LLMs’ help in personal life and professional tasks significantly contribute to prediction of perceived trustworthiness of LLMs. LLMs users with higher scores on these attributes perceive LLMs as more trustworthy.

Next, we examined the relationship between PT-LLMs and the second set of predictors, which included location of use, device type, and interaction method. Since only a small number of participants selected certain choices, we considered only the relevant viable categories as follows. For the location of use, the categories “on the move” and “both” were combined, resulting in two categories retained for analysis: “Stationary” (n = 487, p = 64.76%) and “Stationary and on the Move” (n = 265, p = 35.24%). Regarding device type, the categories of PC and Laptop were merged, as were Tablets and Mobile devices, creating two categories: “PC/Laptop” (n = 530, p = 70.48%) and “Mobile” (n = 222, p = 29.52%). For the interaction method, the categories “text” and “text and images” were combined, as were “voice” and “text and voice,” resulting in two categories: “Text” (n = 662, p = 88.03%) and “Text and Voice” (n = 90, p = 11.97%).

Since all the predictors in the second regression analysis are categorical, one category within each predictor will serve as the reference category, and the results for the other categories are interpreted in comparison to this baseline (see Table 8).

Results showed that the typical location of LLMs use was a significant predictor of PT-LLM-8, indicating that individuals who used their primary LLM both in stationary and on the move locations exhibited higher perception of LLMs trustworthiness, compared to those using their primary LLM in a stationary setting. Furthermore, the device type was not significantly associated with PT-LLM-8. Additionally, the interaction method with primary LLMs was the most influential factor on PT-LLMs. The findings showed that users who interact with their primary LLM through both text and voice are more likely to perceive LLMs as trustworthy compared to those interacting with PLLM via text alone.

Overall, the second regression analysis suggests that location of use and interaction method are associated with variations in PT-LLMs scores.

Personality traits are systematically linked to differences in perceiving the trustworthiness of others. Individuals with higher self-esteem tend to view others as more trustworthy, reflecting a general positivity bias in social perception (Murray et al., 1996). Similarly, higher self-efficacy has been linked with greater confidence in judging others as reliable and trustworthy, consistent with the idea that efficacy fosters approach-oriented social appraisals (Caprara et al., 2011). Among the Big Five, agreeableness is most robustly associated with perceiving others as trustworthy, as agreeable individuals are more inclined to expect benevolence and honesty (Graziano and Eisenberg, 1997),(Thielmann and Hilbig, 2015b). Extraverts also show a tendency to make more positive trustworthiness attributions, likely due to their sociability and reward orientation (Paunonen, 2003). Conscientiousness has been linked to more cautious but generally positive trust evaluations, reflecting norms of reciprocity and reliability (Thielmann and Hilbig, 2015b). In contrast, neuroticism predicts greater suspicion and lower perceptions of others’ trustworthiness, aligning with heightened threat sensitivity and interpersonal insecurity (Robinson et al., 2025). Openness to experience shows weaker but often positive links, with open individuals more willing to extend trust and interpret unfamiliar others as trustworthy (McCrae and Costa, 1997).

These established associations between personality traits and perceived trustworthiness provide anchor points for validating our scale, as they represent theoretically grounded and empirically consistent patterns of behaviour against which our findings can be evaluated. Consistent with these expectations, higher PT-LLM-8 total score showed positive relationship with higher scores on self-efficacy, agreeableness and conscientiousness. This convergence suggests that the scale is capturing judgments of LLMs in ways that mirror well-established processes in human trust perception. The negative association with extraversion, in contrast to its typically positive relation with interpersonal trust (Paunonen, 2003), indicates that extroverts may apply different evaluative standards to LLMs than to humans, perhaps because their trust is grounded in social cues and reciprocal interaction - features less salient in machine communication. Similarly, the absence of significant associations with neuroticism and openness indicates that some dispositional predictors of human trust are less influential in judgments of trustworthiness of LLMs. Interestingly, the lack of association between self-efficacy and perceived trustworthiness of LLMs suggests that efficacy-based confidence in social judgment does not extend to human-machine contexts. Whereas interpersonal trust often requires appraising the reliability of another person’s actions (Zhang, 2021), trust in LLMs is based on perceptions of technical reliability and consistency rather than one’s own competence in handling social exchanges. In other words, self-efficacy is important when trust involves managing interpersonal interactions, but it is less relevant when judging a system whose behaviour cannot be influenced or controlled by the individual.

Taken together, the findings provide strong support for the construct validity of the Perceived Trustworthiness of LLMs scale. The observed associations with traits such as agreeableness and conscientiousness replicate well-established patterns from the human trust literature, indicating that the scale captures a recognisable trust-related construct. At the same time, the divergences, such as the negative association with extraversion and the absence of links with neuroticism, openness, and self-efficacy, suggests that judgments of LLM trustworthiness are not reducible to interpersonal trust mechanisms. Instead, they reflect evaluations that are grounded in dispositional tendencies yet influences by the unique, non-reciprocal nature of human-machine interaction.

For practitioners, it is essential to understand how perceived trustworthiness of LLMs is influenced by user experiences, since this determines how the PT-LLM-8 can inform design and implementation strategies. The present findings show that trustworthiness scores were more strongly associated with users’ perceptions of how useful LLMs were in personal and professional contexts, and with their sense of competence in using their primary system, than with how frequently they used it. This points to an important distinction between exposure and efficacy in perceived trustworthiness of LLM. Research on technology acceptance and trust in automation shows that exposure alone has limited impact on developing trust in AI unless it is accompanied by successful outcomes (Hassan et al., 2019),(Kelly et al., 2023). In contrast, efficacy as individual’s judgements of own capability to organise and execute actions required to attain designated performances (Bandura, A, 1986) is self-referential and affects both the beliefs people form and the behaviours they pursue. Applied to LLMs, this means that perceptions of trustworthiness are grounded not in the sheer number of interactions but in whether users judge themselves competent in eliciting useful outputs. From a practical standpoint, this distinction implies that increasing the availability or frequency of LLM use is unlikely to enhance perceived trustworthiness unless systems simultaneously support user efficacy. Designing interfaces that provide clear prompts, transparent feedback, and corrective guidance can enhance the sense of efficacy and thereby bolster perceived trustworthiness. Training and onboarding materials that show users how to phrase queries effectively, and adaptive support that adjusts to different levels of experience, can be the ways to achieve this. In deployment contexts of the PT-LLN-8 scale, should be monitored alongside user efficacy, because increases in exposure without corresponding growth in efficacy may produce disengagement or scepticism rather than stronger trustworthiness perceptions.

Our findings also indicate that perceptions of LLM trustworthiness vary with the context of use. Participants who reported using their primary LLM in both stationary and mobile settings show higher PT-LLM-8 scores than those who restricted use to stationary settings. This is consistent with recent studies reporting that LLMs are received a more positive evaluations when they are associated with their availability and flexibility in everyday life (Dang and Li, 2025). Interaction modality was also influential: users who engaged through both text and voice reported markedly higher scores than text-only users. This result is consistent with recent evidence that voice interaction reduces delays in conversation and creates a greater sense of responsiveness, leading users to perceive the system as more reliable (X. Zhang et al., 2024),(Wang et al., 2023). Device type, by contrast, was not a significant predictor. This aligns with evidence that evaluations of AI trustworthiness are grounded in perceptions of capability, reliability, and transparency rather than in the hardware through which the system is accessed (Dang and Li, 2025). Taken together, these findings suggest that perceived trustworthiness increases when LLMs can be used effectively across multiple contexts and modalities, whereas the platform of access plays little role in such judgements.

The development and validation of the PT-LLM-8 have significant implications for research, practice, and system design, addressing research gaps identified in previous studies (Afroogh et al., 2024),(Durán and Pozzi, 2025). For researchers, this scale offers a validated instrument for comparing perceived trustworthiness of LLM across studies, populations, and use cases. Its demonstrated measurement invariance allows for group comparisons between gender without introducing bias from the measurement itself. For practitioners working in organisational or educational settings, the scale can be deployed to monitor perceptions of LLM trustworthiness over time, thereby enabling institutions to detect whether new deployments, policy changes, or training interventions are influencing user evaluations in positive or negative directions (Roski et al., 2021). For LLM developers and engineers, the findings highlight areas where design interventions may be most effective. The strong associations observed between perceived trustworthiness, perceived usefulness, and user competence suggest that technical enhancements should be complemented by features that actively support user efficacy, such as clearer guidance on prompt formulation, more accessible explanations of outputs, and error-correction mechanisms that transparently incorporate user feedback. This aligns with recent findings that showed that in education setting trust in LLM was maintained after receiving instructional guidance, but decline after practical engagement potentially due system responses fall short of expectations (Kumar et al., 2024).

Engineers should also note that multimodal interaction (encompassing both text and voice) and flexible use across contexts (stationary and mobile) were associated with higher trustworthiness, indicating that investment in seamless integration of modalities and robust performance across diverse environments may yield greater benefits than hardware optimisation alone. Finally, for policymakers and regulators, the inclusion of laws and regulations as a dimension of perceived trustworthiness indicates the importance of enforceable oversight frameworks (Ingrams et al., 2022),(Kleizen et al., 2023),(Sarker, 2024). Users interpret compliance with regulation as a proxy for ethical assurance, thus, clear communication regarding such compliance may directly contribute to perceptions of trustworthiness in applied settings (Taillandier et al., 2025),(Wester et al., 2024).

Future research should extend the validation of the PT-LLM-8 beyond the present British sample to include diverse cultural and demographic groups. Although the scale was deliberately designed with outcome-focused and broadly interpretable items (e.g., accuracy, fairness, safety, compliance with regulations), it is reasonable to expect cultural variation in how these dimensions are weighted and understood. For instance, in regulatory environments with visible oversight, adherence to laws may be a primary marker of trustworthiness, whereas in contexts with weaker regulation, users may place greater emphasis on transparency or privacy. Similarly, expectations of fairness and accountability are shaped by cultural and institutional norms. A recent study comparing British and Arab populations found that participants from Arab cultural backgrounds generally expressed more positive attitudes toward AI systems, including higher levels of trust (Alshakhsi et al., 2025). Another study reported no significant gender differences in perceived trust across either group, but noted that UK females expressed greater privacy concerns than males, a pattern not observed among Arab participants (Rahman et al., 2025). Incorporating such cultural variability into future validation efforts would help ensure that the PT-LLM-8 remains robust and interpretable across populations.

A further direction is to use the PT-LLM-8 alongside socio-technical evaluations of LLMs. The present study approached trustworthiness of LLM from a socio-psychological perspective, capturing user judgements in everyday interaction. By contrast, frameworks such as TrustLLM conceptualise trustworthiness at the system level, emphasising technical robustness, safety, and compliance with ethical design standards. Comparing these perspectives could reveal gaps between socio-technical compliance and socio-psychological perception, for example, whether regulatory safeguards recognised by developers translate into perceived legitimacy for end-users. Such comparisons would help identify where system design and user trust diverge and inform more integrated approaches to LLM governance and deployment.

The present findings should be interpreted in light of several limitations. First, all validation analyses were based on a sample of British adults, and the psychometric parameters established here reflect the characteristics of this group. The scale’s performance may differ in other age groups, cultural contexts, or populations with varying levels of digital literacy. Applications beyond the current sample should therefore be approached with caution and accompanied by further validation. Second, while the sample size was sufficient and included a range of ages, the exclusion criteria and quality checks led to a relatively engaged and technologically literate participant pool. This limits generalisability to populations with more casual or sceptical patterns of LLM use. Finally, although the PT-LLM-8 demonstrated unidimensionality and reliability, reliance on self-report methods introduces potential biases related to self-perception, response style, and situational factors. Future work should complement self-report measures with behavioural indicators or longitudinal designs to provide a more comprehensive understanding of perceived trustworthiness of LLM.