Applicability of General-Purpose Language Models for Archival Description: Practical Insights and Lessons Learned

Abstract

Purpose: To test the usability and examine the limitations of general-purpose large language models (LLMs) in archival description. The study was designed as a quantitative/qualitative assessment to monitor trends in this rapidly evolving field. Methodology: The experiment involved testing five AI services on a set of archival records. The set of questions and tasks was divided into two categories: technical tasks (page counting, structure recognition, optical character recognition – OCR) and content-related tasks, such as language detection, content summarization, and title suggestions. Performance was evaluated using quantitative and qualitative methods, along with archivists' assessments. Results: A significant discrepancy was found between the models' performance across different types of tasks. The tested models proved unreliable in seemingly simple technical tasks, such as determining the number of pages or detecting graphical elements, while showing greater utility in complex content-related tasks. Discussion: The analysis highlights that the tested LLMs are currently unsuitable for automating precise technical description processes but represent a useful analytical and generative tool for producing content summaries and descriptions. By observing how AI systems perform, archivists also gain better insight into potential difficulties faced by users.