An Overview of Language Models: Recent Developments and Outlook
Chengwei Wei, Yun-Cheng Wang, Bin Wang, C.‐C. Jay Kuo · APSIPA Transactions on Signal and Information Processing · 2024
Not cached yet. This record was fetched live for this request and is stored after the lookup. Reopening the page should serve the cached copy.
APSIPA Transactions on Signal and Information Processing · Cambridge University Press · article · 2024
Abstract
Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner, while pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems.
This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, architectures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.
Open the resolved locator · 10.1561/116.00000010
Metadata licence: OpenAlex metadata (CC0)