The hidden patterns of alignment: exploring variability between generative AI and human.

Sleep health notes

Sleep health notes will be added by the Sleepatch editorial team.

中文解读

中文解读待补充：本站会优先为失眠研究、睡眠质量改善、昼夜节律等高价值睡眠研究添加中文说明。

Original abstract

<h4>Objective</h4>Aligning medical education curricula with standardized competency-based assessments is a multifaceted task that requires accuracy and consistency. While generative artificial intelligence (GenAI) models offer potential assistance, their performance compared to that of human subject matter experts remains unclear. This study investigates the consistency, accuracy, and strategies of human- and AI-generated alignments of pharmacological topics with the United States Medical Licensing Examination (USMLE) Step 1 content.<h4>Methods</h4>A comparative case study examined curricular alignment results produced by GenAI models and human experts. Agreement percentages and pairwise correlation coefficients were calculated to assess consistency and consensus within and between the two groups, highlighting similarities and differences in alignments. We also examined challenges, strategies, and the impact of time constraints on the alignment task.<h4>Results</h4>GenAIs exhibited higher inter-rater reliability in within-group comparisons, with exact agreement percentages ranging from 60% to 79% and stronger correlations. In contrast, human raters demonstrated greater variability, with exact agreement percentages between 14% and 66% and weaker correlations. While partial agreement improved alignment for both groups, GenAIs maintained higher overall consistency and consensus. On the other hand, only two human raters and all GenAI models exhibited high inter-rater reliability in between-group comparisons. GenAIs generated alignments based on contextual analysis and hierarchical mapping, while humans engaged in iterative refinement using their expertise and reflection. Time constraints had a minimal impact on GenAIs, whereas it was acknowledged as a limiting factor for human performance.<h4>Conclusion</h4>GenAIs can efficiently provide accurate and consistent alignments of pharmacological content in medical curricula but lack the sophisticated decision-making capabilities of human experts. While human alignments show variability due to human experts' diverse perspectives and backgrounds, trained experts make more contextually informed decisions. A conceptual model of Curricular Human-AI Teaming (Curricular HAT) could combine these complementary strengths to enhance the workflow and outcomes of curricular alignment. The model supports the transparency and scalability of human-AI collaboration while maintaining the responsibility of human judgment and ethical use of GenAI.

Links and sources

• Official / arXiv page

Comments