Валидация содержания теста по информатике в рамках ЕНТ: экспертная оценка и психометрический анализ

Abstract

The relevance of the study is significant because the Unified National Testing (UNT) in Kazakhstan is a high-stakes exam, the results of which have a substantial impact on the educational trajectories of graduates. This necessitates ensuring the validity and reliability of the test materials, particularly for dynamically developing subjects such as computer science. Additionally, the independent verification of the test content gains importance due to the transition of computer science into the category of specialized subjects starting in 2022. The study aimed to conduct a comprehensive analysis of the content validity of the computer science test in the UNT through an expert evaluation of its alignment with the requirements of the State Compulsory Education Standard (SCES) and a psychometric assessment of the quality of the test items. The research employed mixed methods, including content analysis of 54 learning objectives, an independent expert evaluation with the participation of six experts, qualitative analysis using thematic coding, and quantitative assessment using the Rasch model based on a sample of 116 students. The results of the study showed that the structure of the learning objectives was dominated by tasks focusing on understanding (25%) and application (30%), while there was a lack of tasks at higher cognitive levels, such as analysis (15%) and evaluation (10%). Expert analysis confirmed the representativeness of the content with a Kappa agreement coefficient of 0.80, but identified the need to include modern topics such as 3D modeling and information security. Psychometric analysis demonstrated high reliability of the test with α = 0.94 and the alignment of most items with the Rasch model, while two problematic items were identified with an Outfit MNSQ value above 1.5. Based on the findings, recommendations for the modernization of the test were developed: the formulations of the learning objectives were refined, new thematic blocks were added, and problematic items were corrected. The comprehensive approach to validation proved effective in improving the quality of measurement materials. The results can be applied in the practice of developing and updating test specifications.