Currently, integrated speaking tasks are widely used in many large-scale high-stakes tests, such as, the TOEFL iBT speaking tests, the Versant™ Speak Test, the TEM-4 Oral Test, the CET-SET, and the CELST. Various theoretical motivations have driven the emergence of integrated speaking tasks. For instance, communicative language ability should target at the capacity to implement knowledge in communicative language use (Bachman, 1990); integrated tasks entail the amalgamation of multiple L2 skills (Huang and Hung, 2018). Among the various studies conducted on integrated speaking tasks, those on TOEFL iBT integrated speaking tasks accounted for a great portion. They focused mainly on the comparison between different types of TOEFL iBT integrated speaking tasks and independent speaking tasks (Huang et al., 2018; Zhang et al., 2022) or between TOEFL iBT integrated speaking tasks and other academic speaking tasks (Brown and Ducasse, 2019; Farnsworth, 2013); rating issues (Wei and Liosa, 2015); textual properties and strategy uses (Crossley and Kim, 2019; Inoue and Lam, 2021; Zhang et al., 2021); and source material use (Frost et al., 2019; Frost et al., 2021). Apart from the studies conducted on TOEFL iBT integrated speaking tasks, there are also some studies performed on other integrated speaking tasks and their research perspectives include source material use (Kormos et al., 2022; Lin, 2023; Pusey, 2020), rating (Hirai and Koizumi, 2013; Kim, 2015), affective factors (Ishikawa, 2020), strategy use (Rukthong, 2021; Rukthong and Brunfaut, 2020), fluency of oral products (Suzuki and Kormos, 2023), test fairness (Yan et al., 2019), and task acceptance (Zhang and Zhang, 2022), etc.

Compared with the large number of studies on integrated speaking tasks conducted internationally, researches on such tasks in Chinese context remains relatively limited (Jin, 2012; Rui and Ji, 2017; Zeng, 2011), which is not well aligned with its substantial population of ESL learners and the great effort devoted to English teaching and learning. Jin (2012), Zeng (2011), and Zhou (2005) probed into the influences of source material input in integrated speaking tasks on college students’ oral performance and the results confirmed its positive impacts in relieving anxiety and classroom reticence, facilitating autonomous learning, and promoting oral language output. Meanwhile, Zhang and Elder (2009) examined the reliability and validity of CET-SET from multi-perspectives, namely, authenticity, interactiveness, fairness, and washback effects. Recently, Tsang and Lee (2023) proved that foreign language-related emotions (anxiety, boredom, and enjoyment), speaking motivation, and spoken input beyond the classroom connected directly to Year 3 and Year 4 EFL leaners’ speaking proficiency in Hongkong primary schools, with enjoyment and spoken input beyond classroom serving as direct predictive power. Among them, those particularly related with the present study were those on the National Matriculation English Test (Shanghai Version) (hereafter, NMET(SH)) (Hou, 2018; Liu and Chen, 2018; Xu, 2016, 2021; Zhang, 2019), especially when the characteristics of test takers (e.g. overall language ability, number of participants) were taken into consideration. Their studies confirmed that the NMET(SH) assessed students’ ability of applying various sources of information into completing the integrated listening-speaking tasks and exerted positive washback effects.

It is early since World War II that political needs have been exerting various kinds of influences on the form and scoring of speaking test (Fulcher, 2003, p.1). China is a case in point. General Senior High School Curriculum Standards (hereafter, GSHSCS) (Ministry of Education of the PRC, 2020) provided a comprehensive guidance on the general teaching goals, content standards, and teaching and assessment approaches for senior high school English courses. According to the GSHSCS (2020), the general aim of senior high school English curriculum is to help students to cultivate and develop students’ subject core competencies which include language abilities, cultural awareness, thinking capacity, and learning ability (Ministry of Education of the PRC, 2020, p.5). Meanwhile, students should be able to acquire English learning resources through multiple channels, choose appropriate strategies and methods, monitor, evaluate, reflect on, and adjust learning content and progress (Ministry of Education of the PRC, 2020, p.6).

CELST, a response of the aim of cultivating senior high school students’ comprehensive English ability, has been implemented since 2011. There are three sub-tasks in CELST, namely, reading aloud, role play, and story retelling. Test design of CELST works for the selective purposes. According to the Test Syllabus and Sample Paper Disk for Computer-based English Listening and Speaking Test (EEA-GD, 2016), CELST aims at assessing students’ ability of accomplishing communicative tasks in specific contexts by using English, acquiring and applying their knowledge of phonology, vocabulary, grammar, etc. to comprehend and express effectively. Over the past decade, the number of candidates taking the CELST has consistently exceeded 630,000 annually. In both 2022 and 2023, this figure reached 710,000, reflecting the sustained demand for English proficiency assessment in Guangdong province’s higher education admissions process. These figures also indicate that a substantive study of CELST is of great significance considering the large candidate population and the competitive nature of the college entrance examination.

Large amount of test takers it owns, few studies were conducted on it. Cheng’s (2011) doctoral dissertation focused solely on the validity issues of the story-retelling task in CELST, leaving the first two sub-tasks unexamined. So do the original researches conducted by Wang et al. (2018) and Xu and his co-workers (2019, 2020, 2021, 2023a, 2023b, 2024, 2025). Zhan and Wan (2016) investigated into Senior III students’ attitudes, test preparation practices and test taking processes when completing CELST. Zhou and Zeng (2016) compared the rating results between human raters and computer automated scoring of CELST by using many-facet RASCH models and found that despite of the differences in rater severity between these two scoring approaches, computer automated scoring was of better inner-consistency due to lower bias rates.

To sum up, the above reviewed validation studies on integrated speaking tasks are helpful in conceptualizing construct and the validation procedure. However, language test taking is a process that is task-bound and context specific (Cohen, 2014). Test response is a function not only of the items, tasks, or stimulus conditions, but also of the participants’ responding and the context of measurement (Messick, 1987). In this light, different types of integrated speaking tasks would set different requirements on the amount, degree, and form of information in the listening/video clip that can be integrated into the test-takers’ oral response. Besides, though diverse results have been reported for different types of integrated speaking tests, researches on CELST remain scare, especially the first two sub-tasks. What’s more, most existing studies have largely relied on correlational approaches, such as examining the relationship between textual features and test performance or between strategy use and test performance. Few, if any, have probed into its construct validity directly through in-depth analysis of test-takers’ actual performance. On top of that, as a large-scale high-stake test, studies addressing Chinese senior high school EFL learners’ listening-speaking performance in NMET context are in dire need concerning the great influence it exerts on education both nationally and internationally. Therefore, collection of validity evidence of CELST using a more mathematically-grounded validation testing theory and methodology is of great necessity for the sound and comprehensive interpretation of its test scores as well as test constructs.

Theoretical models serve various functions in test validity development and validation, such as score interpretation, test development, curriculum design, etc. (Luoma, 2004, p.96). The necessity of grounding language test development and application in a theory of language proficiency calls for the incorporation of a theoretical framework that defines what language proficiency is (Bachman, 1990, p.81). Besides, a task-specific theoretical model does not only represent and encompass task construct but also function as the foundation of evaluation and assessment (Luoma, 2004, p.107).

Enlightened by the various frameworks related to listening and/or speaking, it could be concluded that a model of listening-speaking ability should firstly take an integrative, interactive, or communicative stand, for language use both in natural and academic contexts are inherently integrative, interactive, or communicative. Meanwhile, it should also serve as a guideline for the assessment of both language product parameters and speaking process parameters, for the inclusion of the former could guide leaners’ self-learning, teachers’ teaching, and raters’ rating practices, and the encompass of the latter could help examine participants’ cognitive and strategic processes, which would help finding factors influencing task completion (Bachman, 1990). Therefore, an operationalized working model of the construct of CELST will be developed based on the analysis of existing theoretical models, the Test Specifications, and the specific test purposes of CELST (EEA-GD, 2016), which will be presented in Section 2.5.1.

Being aware of the shift from multiple types of validity to a unitary understanding and from a focus on prediction to one on explanation, Messick (1987) conceptualized validity as a unitary concept and proposed a comprehensive framework consisting of six interrelated aspects, namely, content, substantive, structural, generalizability, external, and consequential validity. Among them, the external aspect of construct validity, which includes convergent and discriminant evidence obtained through multitrait-multimethod comparisons, as well as evidence of criterion relevance applied utility (Messick, 1987), delineates the extent to which the construct represented in the assessment accounts for the external pattern of correlations.

Besides, Chapelle put that a validity argument should integrate both evidence and rationales to support conclusions about the appropriateness and soundness of score-based inferences and uses of a test (1999, p.263) and held that validation should begin with formulating a hypothesis concerning the appropriateness of testing outcomes (1999, p.259). For the external aspect of validity, Chapelle (1999) stated that data analyzed through a multitrait-multimethod research design could be used to examine the relationships between the target test and other tests or quantifiable performance indicators. Extending back, influenced by psychological structuralism and the conception of explaining performance through the systems and subsystems of underlying process, Embretson (1983) introduced construct modeling to construct validation study. According to construct modeling, the internal structure and substance of a test can be addressed more directly by means of causal modeling of item or task performance. Embretson (1983) suggested that the construct validation research of identifying the theoretical mechanisms underlying task performances (a task decomposition process), that is, construct representation, and detecting the relationships of a test to others (e.g. strength, frequency, and pattern of significant relations with other measures), that is, nomothetic span, should be separated. It could be achieved by mathematical modeling, psychological modeling, or multicomponent latent trait modeling which combines the features of the former two ones (1983, p.181).

Moreover, according to the Standards for Educational and Psychological Testing (hereafter the Standards) (AERA et al., 2014), which provides criteria for the development and evaluation of tests and testing practices and guidelines for assessing the validity of test scores interpretations for their intended uses, evidence based on test content may involve logical or empirical analyses to determine how adequately the test content of a test represents the targeted domain in relation to the proposed interpretations of test scores (AERA et al., 2014, p.14). However, Xu et al. (2024) pointed out that, to date, no public report has examined the content validity of the CELST.

Following the guidance of the Standards, decomposing into coded construct-relevant textual measures and strategic measures, and employing a multicomponent model based on the testing purpose of CELST, the present study would explore to what extent these data-driven measures represent and are related to the specific inferences to be made from test scores.

Firstly, the first author of this paper rechecked whether total scores, sub-scores for each subtask, and the corresponding video tape recordings were included and the results were confirmed. Secondly, during the process of data confirming, it was found that the frequency for several coding parameters were zero for all participants. Thus, the parameter number of internals was deleted due to zero occurrence across all three subtasks. Thirdly, observed variable SpM was no longer used due to its high correlation with the observed variable MSpM which has more linguistic implication.

The inter-coder reliability Cranach’s α value for all parameters were over 0.8 (see Appendix D), indicating that the two coders were consistent with each other in the coding process. This further guaranteed the reliability of the coding results in the main analysis.

Based on the results of previous literature on integrated speaking assessment and second language assessment, a set of indexes would be used in several alternative hypothesized models of CELST to explore into the relationship between the theoretical model proposed for CELST and the working construct (see Fig. 3) from the perspective of language competence parameters, addressing Research Question 1. As stated in Section 3.2.3, five factors, namely, accuracy, complexity, coherence, fluency, and phonology, were employed as latent factors in the competence model of CELST. Initially, the researchers assessed the model fitness index of hypothesized competence Model 1 using traditional second-order SEM. However, the analysis failed to converge due to the exceeding of the maximum number of iterations.

A detailed inspection of the data found that most of the value of CpT is smaller than 2 (CpT appeared less than twice in each participant’s performance), indicating inappropriateness of using CpT as a measure of differentiating senior high school students’ oral proficiency. Iwashita et al. (2008: 45) also raised doubts on using of ratio measures to represent complexity. Since with participants’ overall oral language proficiency going up, NoT and NoC in their oral production would increase, but not the value of the ratio. Besides, the negative correlation between CpT and the total score (-.018) also violates the common sense, because more CpT usually indicates high overall language proficiency and score. Therefore, the observed variable NoC and NoT are used instead of CpT and MLT to measure language complexity. Besides, variable RMS would be discarded in the new model due to its non-significant relationship with total score (-.063). However, the new model still couldn’t converge. Probing into the observed variables in the new model, those whose correlation with the total score is smaller than 0.3, to be specific, DysM and RRSU, were deleted. Thus, a hypothesized exploratory bi-factor SEM (Model 2) (see Fig. 4), instead of the traditional SEM, was proposed based on the data feedback of model 1.

Results of bi-factor ESEM indicated that the model (Fig. 4: Model 2) fits the data well. The Chi-square in the present model was 116.489, with 85 degrees of freedom. χ²/df was 1.37, indicating a good fit between the present model and the data. The value of RMSEA and SRMR were much smaller than 0.5, also indicating well model fit. Hitherto, all these three absolute model fit indexes demonstrated that Model 2 fits the present data well. CFI and TLI are two indexes of comparative fit. The higher these two values are, the more fit the model would be. They both range from 0 to 1 and a value larger than 0.9 indicates well model fit. These two model fit indexes also indicated that Model 2 fits the present data well. Besides, almost all observed variables loaded on the general factor, listening-speaking ability, and several of them clustered and loaded on one domain specific factor such as accuracy, complexity, coherence, fluency and phonology. Such kind of loading pattern confirmed the use of exploratory bi-factor ESEM analysis of the present data. However, probing into the details, some minor issues still deserve further in-depth speculations. For one thing, RMP (ratio of mispronounced phonemes) loaded on overall listening-speaking ability with a positive value, which is against the theory. As is known to all, the more mistakes one makes in terms of phonemes mispronounced, the poorer one’s oral English would be. For another thing, RUiF (0.674) did work effectively on its a domain specific factor, phonology, but not on the overall general listening-speaking ability. Finally, the factor loading of PRO on its domain specific factor was relatively lower (-0.204) than the accepted value (0.3) symbolizing meaningful causal relationship between observed variable(s) and latent variable, while its contribution to the general listening-speaking ability factor was rather well (-0.747). Therefore, modifications were to be made. Firstly, the index RMP was discarded. Even though the factor loading values in the new modified model (Model 2.1) were somewhat better than those in Model 2, RUiF still could not load on general listening-speaking ability, which is against the hypothesis of bi-factor ESEM analysis. Besides, PRO still loaded poorly on phonology (-0.207), though significant. However, several modification indexes did not imply any modification on PRO. Thus, RUiF was deleted in the modified Model 2.2 to see whether the deletion of such an observed variable would result in better model fit and interpretations.

As could be seen in Fig. 5 (Model 2.2), all observed indicators in the new model loaded on the overall listening-speaking ability and their own domain specific latent factors. As a whole, the finalized model presented a well level of model-to-data fit (χ² = 79.97, df = 60; χ²/df = 1.33; RMSEA = 0.04; SRMR = 0.02; CFI = 1.00; TLI = 0.976). All factor loadings of the observed variables on their corresponding latent variables were significant at the 5% level.

As illustrated in Section 2.4.1 and Section 3.2.3, the communicative strategies adopted in the present study are consisted of two macro-parameters, to be specific, reductive strategies and achievement strategies, representing two different model approaches. Hierarchical multiple regression is often used when the independent variables are entered cumulatively according to a specified hierarchy by the researchers based on theoretical grounds (Gaciu, 2021; Pallant, 2020). Therefore, hierarchical multiple regression was employed to examine whether and to what extent the two different macro-parameters of communicative strategies used by participants influenced and predicted their overall performance, as well as their performance on individual aspects separately. Assumptions of conducting a multiple linear regression analysis like the examination of sample size, multicollinearity and singularity, outliers, normality, linearity, etc. were all confirmed. Firstly, a total of 360 participants met the sample size requirement on conducting multiple linear regression analysis (N ≥ 50 + m8; m refers to the number of independent variables) (Tabachnick & Fidell, 2013). Secondly, with all Pearson correlation values allocated between − 0.2 and 0.3, Tolerance values of more than 0.10, and VIF values well below the cut-off of 10, non-collinearity could be achieved. Thirdly, visual inspection of normal probability plot of the regression standardized residual confirmed the normality of the data. (see Appendix E)

As shown in Table 2, R square of the reductive model is 0.065. The value was upgraded to 0.168 after the four achievement strategy parameters entered. To be specific, the value of R square changed was 0.103. That is to say, the present communicative strategy model only explained 16.8 per cent of the variance in participants’ score. Results from ANOVA indicated that this model did achieve statistical significance (F = 33.35, Sig.=.00). Detailed inspection of the coefficients analysis demonstrated that apart from App, all the other five parameters made significant unique contributions to the prediction of participants’ overall CELST score (Sig.<.05).

Such a loading pattern (see Fig. 5: Model 2.2) confirms that CELST does examine participants’ ability of fulfilling three different types of listening-speaking tasks by using and applying their knowledge of accuracy, complexity, coherence, fluency, and phonology, which is in accordance with its test purposes, testing students’ ability of accomplishing tasks in specific contexts by acquiring and applying their knowledge of English phonetics, vocabulary, grammar, etc. to comprehend and express in particular (EEA-GD, 2016). The relatively high loadings on general listening-speaking ability also reflect that CELST implements the requirements of the GSHCSC that senior high school English courses should stress on cultivating students’ ability of acquiring and processing information by means of English, analyzing and solving problems in English, especially the ability of thinking and expressing in English; thus, developing students’ overall English language using ability.

Besides, the different numbers of observed indicators of these five domain specific factors and their different weights not only manifest the comprehensiveness of CELST but also indicate its emphasis on the dimensions of language ability might diverse in different sub-tasks, that is, contextual specificity of its three subtasks. As manifested by Model 2.2, accuracy is loaded by REFC, RCVF, and RRSC, complexity is loaded by TTR, NoC, and NoT, fluency is loaded by MSpM, AR, MLR, and DysM, phonology is loaded by PRA, PRO, and PR, and coherence is loaded by COM. Each of the loadings of accuracy and fluency has a value of over 0.4, with some larger than 0.7, indicating strong predictive power of these textual measures on accuracy and fluency. However, when it comes to complexity and coherence, the situation turns out to be not so sound, for the loading value of TTR and COM on their correspondent domain specific factor is just over 0.3, a border value. What’s worse, the factor loading of RPO on phonology is only − 0.208. These different loading values reflect, to some extent, CELST gives different weights to these five dimensions due to aiming at testing participants’ ability of obtaining and applying the information from diverse source materials and their previous background knowledge to accomplish given tasks. This might also be attributed to the integrative nature of listening-speaking task (Brown et al., 2005; Farnsworth, 2013).

Meanwhile, the three sub-tasks (reading aloud, role play, and story retelling) and their corresponding rating criteria might also play a role. The different requirements for senior high school students’ language ability development in terms of grammar, vocabulary, pronunciation, etc. of listening ability, speaking ability and affective attitude (see Section 2.2) again is one of the factors that could address the different weights. What needs to be noticed is that, phoneme omitting or swallowing, particularly the omission of the medial or final consonant in words, is a common phenomenon among students from Guangdong when speaking English (Xu et al., 2025). Such kind of consonant simplification could be possibly related with their dialect, Cantonese, which does not have complex consonant clusters at the end of syllables and tends to rely on tone change with pitch but not consonants and their endings. Therefore, to some sense, the negative significant impact of RPO on phonology and listening-speaking ability indicated that raters can recognize and evaluate participants’ phoneme omitting effectively, which is in consistency with the findings of Xu et al. (2025).

Moreover, the unidimensional factor loading of RRSU, ADD, and TEM on the general listening-speaking ability and the components of the domain specific complexity factor deserve exploration. Whether RRSU should be used as a parameter of accuracy or a factor representing content remains a question (Zhou, 2005; Brown et al., 2005). The high and unidimensional factor loading of RRSU (0.965) on general listening-speaking ability manifests that RRSU is an indispensable as well as decisive variable accounting for the overall listening-speaking proficiency; however, it is a variable that should not be regarded as a part of accuracy. Moving to coding parameters of coherence, such as, ADD and TEM, it could be due to the fact that reading aloud task only requires test takers to read after the video clip with appropriate pronunciation and intonation and no cohesive devices are needed in this task. As a result, when CELST is taken into consideration as a whole, only COM variable clustered to the domain specific coherence factor. Results of Model 2.2 also prove that it is inappropriate to use ratio measures, like CpT, to assess L2 language learners’ oral production, which gives empirical support to the doubts proposed by Iwashita et al. (2008, p.45). At the meantime, the high factor loadings of NoC and NoT on both the general listening-speaking ability factor (NoC: 0.694; NoT: 0.729) and the domain specific complexity factor (NoC: 0.692; NoT: 0.447) indicates that NoC and NoT could be used as measures representing textual complexity.

Results of hierarchical multiple regression analysis demonstrated that the combination of all the six indices explained 16.4% of the score variances. Though not very large in terms of mathematic value, such a value is considerable since language competence part was not considered in the regression model. On the one hand, the four predictors of achievement strategies contributed 10.3% out of a total of 16.4%, indicating that achievement strategies like guessing, paraphrasing, restructuring, etc. are preferred by senior high school participants compared with those of reductive strategies when they have difficulties in expressing. This is quite different from Yang’s (2009) finding that their participants tended to rely more on strategies like verbatim source use which was considered to be unfavorable for their language development. Meanwhile, it might also be due to the fact that participants in the present study were well trained before they took part in such a large-scale high-stake test so that they were quite well-familiar with how their performances will be assessed. Hence, they would not use reductive strategies, message abandon and semantic reduction, to avoid being assessed poorly. Detailed inspection of the coding data proved that no participant used message abandon in the first sub-task (reading aloud), which requires the test takers to repeat immediately what the computer just broadcasted, with the text of the content presented on the computer screen.

On the other hand, approximation was the only variable that did not make sense to score variance, compared to the other five communicative strategy variables (see Table 2, t = .16, Sig. >.05). Swain et al. (2009, p.23) also reported that approximation was the type of communicative strategy that was used least by participants. According to Fulcher (2003) and Swain et al. (2009), approximation refers to the behavior of using strategies like lexical substitution, over-generalization, and exemplification to replace an unknown word or a word that is out of memory’s reach. Probably, the vocabulary and expressions given in the source material in these three subtasks in CELST facilitated participants’ oral production in acquiring words and expressions (Brown et al., 2005; Frost et al., 2021).