VALERIA RUSCIO

PhD Graduate

PhD program:: XXXVI


supervisor: Antonetta L. Bruno (Sapienza University of Rome, ISO)
co-supervisor: Fabrizio Silvestri (Sapienza University of Rome, DIAG)

Thesis title: Analyzing Large Language Models: Bridging Linguistics and Applied Mathematics

This thesis delves into the world of language processing using advanced deep learning models to uncover complex linguistic patterns. The exploration begins with an analysis of vector semantics, focusing on the strengths and limitations of static and contextualized embeddings in capturing semantic relationships, particularly in the Korean language. A thorough examination of the attention mechanism within Transformer architectures is conducted, highlighting its role in linguistic representation and the challenges in deriving a complete understanding of linguistic phenomena through it. The research extends into probing techniques aiming to uncover the types of linguistic features captured by language models and their ability to generalize across different linguistic tasks. A significant part of the research investigates the relationship between token likelihood and attention values in language models, unveiling a dynamic interaction that provides insights into how these models process language. Further, spectral analysis and signal processing are introduced as new methods to examine the inner workings of language models. These techniques provide a new perspective to understand how language models work, extract linguistic features, and generalize across different language tasks.

Research products

11573/1669797 - 2023 - Synonymy in Korean Lexicon through the lens of vector semantics
Ruscio, Valeria - 02a Capitolo o Articolo
book: Percorsi in Civiltà dell’Asia e dell’Africa II: Quaderni di studi dottorali alla Sapienza - (978-88-9377-260-0)

11573/1696195 - 2023 - Attention-likelihood relationship in transformers
Ruscio, Valeria; Maiorca, Valentino; Silvestri, Fabrizio - 04h Atto di convegno in rivista scientifica o di classe A
paper: International Conference on Learning Representations (International Conference on Learning Representations) pp. - - issn: - wos: (0) - scopus: (0)
conference: The Eleventh International Conference on Learning Representations (Kigali)

© Università degli Studi di Roma "La Sapienza" - Piazzale Aldo Moro 5, 00185 Roma