Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic nature of attention, however, prohibits transformers from performing this process on arbitrarily long sequences. In this wor…

View Post on Scour ↗

More from arXiv on Scour.