von Liu, Ziyue und Nils Reiter. 

In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), hg. von T.Y.S.S., Santosh and Rodriguez, Juan Diego and de Gibert, Ona, S. 1177-1189, San Diego: Association for Computational Linguistics.

Abstract

Evaluating the grammatical abilities of large language models (LLMs) is
important for both NLP and linguistic theory. We investigate the ability of large language
models (LLMs) to perform acceptability judgments in a forced-choice paradigm. We evaluate
a subset of LLMs on 150 minimal sentence pairs sampled from Linguistic Inquiry and
categorized using BLiMP linguistic phenomena. Our results show that while LLMs approximate
human judgments, performance varies across models and phenomenon types, with stronger
alignment on morphosyntactic phenomena than on linguistically and semantically demanding
phenomena. Prompting strategies have minimal impact.