Learning to Learn

Poly-Attention — A Better Attention Mechanism That's Still Fast Poly-Attention — A Better Attention Mechanism That's Still Fast Most improvements to the Transformer's self-attention mechanism work at the level of approximation — finding faster ways to compute the same pairwise dot products. Sparse attention, linear attention, flash attention: all of these keep the fundamental computation the same and just make it cheaper. They share an assumption: pairwise token interactions are the right thing to compute, and the goal is to do it faster. A recent paper from Columbia University — Poly-Attention: A General Scheme for Higher-Order Self-Attention (Chakrabarti, Pitassi, Alman, 2026) — makes a different move. It asks whether pairwise interactions are expressive enough , proves they often aren't, and then builds a unified mathematical framework for richer attention mechanisms. The punchline is a new mechanism called tree-attention that is simultaneously ...