The attention mechanism is one of the major breakthroughs in AI Transformer theory, but it is also a performance bottleneck. Various methods can be used to optimize the attention algorithm including sparse attention, multi-query attention, and flash attention. The speed of attention can also be improved by code optimizations such as KV caching.