Tag: State Space Model

All the articles with the tag "State Space Model".

• Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism

  Published: 6 May, 2025 |  at  11:15 PM

  

  87.26 🤔

  本文通过提出Gather-and-Aggregate (G&A)机制，揭示了Transformer和SSM模型在上下文检索能力上的性能差距主要源于少数关键头部的实现差异，并通过混合模型实验验证了注意力机制在改进SSM检索能力上的潜力。

• Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation

  Published: 8 May, 2025 |  at  06:13 PM

  

  86.84 🤔

  This paper proposes Recall with Reasoning (RwR), a method that enhances Mamba's long-context memory and extrapolation by distilling chain-of-thought summarization from a teacher model, achieving significant performance improvements on LONGMEMEVAL and HELMET benchmarks while preserving short-context capabilities.

• State Space Models are Strong Text Rerankers

  Published: 4 May, 2025 |  at  04:26 PM

  

  50.53 🤔

  本文通过全面benchmark比较状态空间模型如Mamba与Transformer在文本重排序任务中的性能和效率，发现Mamba模型可实现类似性能但效率较低，并强调了未来优化方向。