Professor Song Mei, UC Berkeley
Mechanistically Demystifying Extreme-Token Phenomena in Large Language Models
Abstract
Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. In this talk, we will elucidate the mechanisms underlying extreme-token phenomena by analyzing a single-layer transformer on a toy task, revealing striking similarities to the behavior observed in pre-trained LLMs. This is a joint work with Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, and Mike Jordan.