Clinical note comparison and data retrieval via embedding vectors: model selection, metrics, and convergence.

چکیده اصلی

BACKGROUND: Embedding models are an integral part of generative AI architectures, transforming text into embedding vectors that represent semantic content in numerical form. Despite their central role, their performance in clinical settings remains underexplored. We evaluated embedding models across two tasks: semantic difference detection in clinical notes, and data retrieval from patient records. METHODS: Eight models were applied to synthetic discharge summaries in English, Swedish, and Finnish. Semantic sensitivity was assessed by introducing controlled perturbations (deletion, modification, and paraphrasing) at three levels of severity; cosine similarity, L1 and Euclidean distances were computed between the vectors of the original and perturbed texts. Partial vectors were compared to explore dimensionality reduction. Two models with the biggest contrast in semantic difference detection were evaluated on retrieval of relevant information from real Finnish vascular surgery records. RESULTS: Embedding vectors captured semantic differences in clinical notes: content deletion and modification produced larger increases in vector distance than paraphrasing. On average, models detected the direction of semantic change correctly, but case-level performance varied considerably. Qwen3-Embedding-8B produced no case-level (directional) errors whereas multilingual-E5-large produced them the most (12.2%). In retrieval this contrast transferred only partially and was task-dependent: sufficiency scores favoured Qwen3-Embedding-8B for the vascular-diagnosis question (2.25 vs 1.15 out of 5) but were comparable for the antithrombotic-medication question (3.25 vs 3.17 out of 5). For some models, as few as 0.6-1.2% of dimensions sufficed to replicate full-vector accuracy; principal component analysis and coordinate-level analysis did not account for this finding. CONCLUSIONS: Our results show that the choice of embedding model is important: performance differences between models can be large enough to determine whether clinically relevant information reaches the end user, and model weaknesses can be both task-specific and context-dependent.

متن کامل اصلی