大規模言語モデル

Large language model — Grokipedia Fact-checked by Grok 1 month ago Large language model Ara Eve Leo Sal 1x Acronym LLM Genre Artificial intelligence Machine learning Natural language processing Category Natural language processing Architecture Transformer Training Objective Unsupervised next-token prediction Learning Paradigm Unsupervised Typical Parameter Count Billions to trillions Largest Reported Parameters 671 billion (DeepSeek-V3) Typical Training Tokens Trillions Typical Training Compute ExaFLOP-scale Year Introduced 2018 Year Popularized 2022 First Large Scale Example GPT-3 Model That Popularized Term ChatGPT Major Developers OpenAI Google Meta AI Anthropic Notable Open Models Llama 3.1 Mistral models BLOOM DeepSeek-V3 Notable Closed Models GPT-4 Claude Gemini Key Applications Translation Summarization Question answering Core Capabilities Few-shot learning Zero-shot learning Multi-step arithmetic Chain-of-thought reasoning In Context Learning Yes Alignment Method Reinforcement learning from human feedback Scaling Laws Reference arxiv.org/abs/2001.08361 A large language model (LLM) is a transformer-based deep neural network pre-trained on vast amounts of text data to predict the next token in a sequence. This unsupervised next-token prediction endows LLMs with broad capabilities for processing and generating natural language. These models typically encompass billions to trillions of parameters, enabling them to capture intricate patterns in language syntax, semantics, and even rudimentary reasoning abilities via unsupervised next-token prediction. Empirical scaling laws demonstrate that LLM performance, measured by cross-entropy loss, follows power-law relationships with increases in model size, training dataset volume, and computational resources, underscoring the causal role of scale in enhancing predictive accuracy. [1] LLMs have achieved notable successes, including few-shot and zero-shot learning on diverse tasks such as text generation, translation, coding assistance, summarization, and question-answering, with widespread industry adoption exemplified by tools like GitHub Copilot and ChatGPT, often surpassing specialized models without task-specific fine-tuning. [2] As parameter counts exceed certain thresholds, emergent abilities manifest, where capabilities like multi-step arithmetic or chain-of-thought reasoning show non-linear improvements, transitioning from near-random to human-competitive performance on benchmarks, though some apparent thresholds reflect metric artifacts rather than fundamental shifts. [2] These phenomena arise from the models' capacity to internalize statistical regularities from training data, though they remain probabilistic approximations rather than veridical understandings of the world. [2] Despite these advances, LLMs face significant limitations and controversies, including a propensity for hallucinations—generating fluent yet factually incorrect outputs that can mislead users in high-stakes domains like science and law. [3] Such errors stem from the autoregressive training objective, which prioritizes token likelihood over truth fidelity, compounded by gaps in training data coverage. [3] Additionally, LLMs inherit and amplify biases present in their corpora, reflecting societal imbalances rather than inherent model flaws, though mitigation techniques like reinforcement learning from human feedback have shown partial efficacy in aligning outputs with preferred behaviors. The immense compute demands of training—often exceeding exaFLOP-scale operations—raise concerns over energy consumption and accessibility, yet empirical evidence affirms that continued scaling yields diminishing but positive returns in capability. [1] Definition and Core Principles Statistical and Probabilistic Foundations Large language models operate as probabilistic generative models that estimate the joint probability distribution over sequences of tokens derived from natural language corpora. At their core, these models employ an autoregressive framework, factorizing the probability of a token sequence $ s = (t_1, t_2, \dots, t_n) $ as $ P(s) = P(t_1) \prod_{i=2}^n P(t_i \mid t_1, \dots, t_{i-1}) $, where each conditional probability $ P(t_i \mid t_{<i}) $ is parameterized by a neural network, typically a transformer architecture. [4] This decomposition reflects the sequential, context-dependent nature of language generation, allowing the model to predict subsequent tokens conditioned solely on preceding ones during both training and inference. During inference, responses are generated token by token, with the next token chosen from the predicted probability distribution over the vocabulary via a decoding strategy; greedy decoding selects the highest-probability token at each step for deterministic outputs, while sampling methods draw from the distribution in a weighted manner, akin to a lottery favoring more probable tokens, to introduce variability in phrasing. [5] [6] The training objective aligns with maximum likelihood estimation, minimizing the negative log-likelihood of the observed data to fit the model's parameters $ \theta $. This equates to optimizing the cross-entropy loss $ \mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^N \log P_\theta(t_i \mid t_{<i}) $, where $ N $ denotes the total number of tokens in the training corpus. [7] Cross-entropy quantifies the expected additional bits required to encode data from the true empirical distribution using the model's approximate distribution, derived from information theory principles. [8] Gradient-based optimization, such as stochastic gradient descent variants, adjusts $ \theta $ to reduce this divergence, with billions to trillions of parameters enabling the capture of high-order statistical dependencies in data exceeding trillions of tokens. [4] Model performance is often assessed via perplexity, the exponential of the average negative log-likelihood per token, $ \mathrm{PPL} = \exp\left( -\frac{1}{N} \sum_{i=1}^N \log P(t_i \mid t_{<i}) \right) $, which interprets the model's predictive uncertainty as an effective branching factor over the vocabulary. [9] Empirical analyses reveal that perplexity scales as a power law with respect to training compute, dataset size, and parameter count, with Kaplan et al. reporting exponents around -0.076 for parameters, -0.103 for data, and smaller for compute-optimal regimes, while Chinchilla refines the joint N-D trade-off with -0.34 for N and -0.28 for D in transformer-based models trained up to 2023. [1] [10] This statistical scaling underpins non-linear ability improvements but highlights inherent limitations, as the models remain interpolative statistical approximators without explicit mechanisms for causal inference; they do not perform Bayesian updating, though they approximate probabilistic patterns from data. [11] Distinctions from Prior AI Paradigms Large language models (LLMs) fundamentally diverge from earlier symbolic AI paradigms, which relied on hand-engineered rules and logical representations to encode domain-specific knowledge, such as in expert systems like MYCIN for medical diagnosis or DENDRAL for chemical analysis. [12] In contrast, LLMs operate as statistical models trained to predict sequences of tokens from vast, unlabeled corpora, deriving capabilities through pattern recognition rather than explicit symbolic manipulation, enabling generalization to novel inputs without predefined logic. [1] This shift prioritizes empirical scaling over axiomatic reasoning, though it introduces challenges like hallucinations due to the absence of inherent causal or truth-verifying mechanisms. [13] Unlike recurrent neural networks (RNNs) and long short-term memory (LSTM) units prevalent in pre-2017 sequence modeling, transformer-based LLMs employ self-attention mechanisms that process entire input sequences in parallel, mitigating vanishing gradient issues and enabling efficient handling of long-range dependencies. [14] RNNs and LSTMs process data sequentially, leading to computational bottlenecks and degraded performance on extended contexts exceeding hundreds of tokens, whereas transformers scale to contexts of thousands or millions via positional encodings and multi-head attention. [15] This architectural innovation facilitated the pretraining of models like GPT-3, which achieved state-of-the-art results on benchmarks such as GLUE without task-specific architectures, a departure from the era's reliance on recurrent layers fine-tuned per domain. [16] A hallmark distinction lies in adherence to neural scaling laws, where LLM performance on metrics like cross-entropy loss follows power-law relationships with model parameters ( N ), dataset size ( D ), and compute ( C ), as empirically validated in models up to 175 billion parameters. [1] Prior neural networks, constrained by smaller scales (typically under 1 billion parameters), did not exhibit predictable improvements or emergent abilities—such as few-shot learning—until compute budgets exceeded 10^23 floating-point operations, underscoring how LLMs leverage unprecedented data volumes (trillions of tokens) and hardware advances absent in earlier paradigms. [17] These laws imply optimal resource allocation, balancing N and D for efficiency, unlike ad-hoc scaling in legacy systems that plateaued without analogous gains. [18] Distinctions from Human Language Processing Large language models differ from human language processing in several fundamental ways. Humans resolve syntactic ambiguities incrementally during comprehension, often encountering garden-path effects that necessitate reanalysis upon disambiguating cues, whereas LLMs process sequences deterministically via next-token prediction without analogous reanalysis dynamics. [19] Humans also demonstrate superior data efficiency, acquiring language proficiency from limited exposures augmented by social interactions and embodied cues, enabling robust generalization that outpaces LLMs' relia