The Syntax-Semantics Group will hold its next meeting on Thursday, January 22, at 2-3pm in Room 002 of the McGill linguistics department. Online participants can join with this link: https://mcgill.zoom.us/j/84180431566
Alessio Tosolini (McGill) will be presenting “Analyzing the Linguistic Priors of Language Models with Synthetic Languages” Here is the abstract:
- While modern language model architectures are often assumed to be language-agnostic, there is limited evidence as to whether these models actually process the wide diversity of natural languages equally well. Previous work disagrees as to whether a language model’s ability to model a language depends on its morphological complexity. In humans however, it is known that children acquire aspects of language at different rates, such as Spanish’s transparent gender system being learned at a younger age compared to Dutch’s opaque gender system. We investigate the question of LMs’ abilities to learn languages as a function of morphological complexity by analyzing how well LMs learn carefully constructed artificial languages containing a variety of verbal complexity, ranging from simple paradigms to covering far more verb classes than occur in natural languages. Rather than learning all languages equally efficiently, models trained on these languages show strict preferences for processing simpler languages. Finetuning English pre-trained models did not improve LM’s modeling of verb conjugation paradigms compared to models trained from scratch. Furthermore, while some observed behaviors mimic human linguistic priors, we find that they indicate the model memorizes its training data rather than generalizes from it. Finally, this work additionally introduces a codebase through which artificial data from naturalistic synthetic languages can be generated at scale, setting the stage for further investigations about the parallels between LM language learning and human language learning.