Lightning Attention-2 是一种新型的线性注意力机制,让长序列的训练和推理成本与 1K 序列长度的一致。
图 1
论文:Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models 论文地址:https://arxiv.org/pdf/2401.04658.pdf 开源地址:https://github.com/OpenNLPLab/lightning-attention
![图片](https://image.jiqizhixin.com/uploads/editor/2f6b0dea-3762-4cc9-9c68-2f11ce4d6aed/640.png)
![图片](https://image.jiqizhixin.com/uploads/editor/660fc34c-e5c0-4032-aeb7-7d0c5665e996/640.png)
![图片](https://image.jiqizhixin.com/uploads/editor/f0ca5c53-ad72-4320-9957-251f04757a8f/640.png)