Cadeddu, Andrea; Chessa, Alessandro; De Leo, Vincenzo; Fenu, Gianni; Motta, Enrico; Osborne, Francesco; Recupero, Diego Reforgiato; Salatino, Angelo and Secchi, Luca
(2023).

Enhancing Scholarly Understanding: A Comparison of Knowledge Injection Strategies in Large Language Models.

In: Deep Learning for Knowledge Graphs 2023 (Alam, Mehwish; Buscaldi, Davide; Cochez, Michael; Osborne, Francesco and Recupero, Diego Reforgiato eds.), 6 Nov 2023, Athens, CEUR.

URL: https://ceur-ws.org/Vol-3559/paper-7.pdf

The use of transformer-based models like BERT for natural language processing has achieved remarkable performance across multiple domains. However, these models face challenges when dealing with very specialized domains, such as scientific literature. In this paper, we conduct a comprehensive analysis of knowledge injection strategies for transformers in the scientific domain, evaluating four distinct methods for injecting external knowledge into transformers. We assess these strategies in a single-label multi-class classification task involving scientific papers. For this, we develop a public benchmark based on 12k scientific papers from the AIDA knowledge graph, categorized into three fields. We utilize the Computer Science Ontology as our external knowledge source. Our findings indicate that most proposed knowledge injection techniques outperform the BERT baseline.