Attention Sink Tokens: Why the First Few Tokens Skew LLM Outputs
Large Language Models don't distribute attention evenly across a prompt. Researchers have discovered that certain early tokens can act as "attention sinks," attracting disproportionate attention and influencing outputs in unexpected ways. Learn how attention sink tokens work, why they affect model