Posts

All the articles I've posted.

• Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation

  Published: 8 May, 2025 |  at  06:13 PM

  

  86.84 🤔

  This paper proposes Recall with Reasoning (RwR), a method that enhances Mamba's long-context memory and extrapolation by distilling chain-of-thought summarization from a teacher model, achieving significant performance improvements on LONGMEMEVAL and HELMET benchmarks while preserving short-context capabilities.

• TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

  Published: 17 May, 2025 |  at  11:21 PM

  

  86.82 🤔

  本文提出了一种基于多头张量化和Tucker分解的框架，通过强制共享高维子空间对大型语言模型的多头注意力权重进行结构化去噪和压缩，显著提升推理能力并实现高达247倍的压缩率。

• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning

  Published: 20 May, 2025 |  at  11:24 AM

  

  86.82 🤔

  本文通过创新任务设计和Pythia模型训练检查点分析，揭示上下文学习（ICL）在大型语言模型中既非纯记忆也非符号算法，而是依赖统计特性的有限泛化能力，并探讨了其训练动态和内部机制联系。

• LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

  Published: 8 May, 2025 |  at  06:16 PM

  

  86.80 🤔

  LENSLLM introduces a Hessian-based PAC-Bayes framework and NTK-based scaling model for LLM selection, achieving up to 91.1% accuracy and 88.5% computational cost reduction by modeling fine-tuning dynamics across diverse tasks.

• TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining

  Published: 11 May, 2025 |  at  11:16 AM

  

  86.74 🤔

  This paper introduces TiC-LM, a web-scale benchmark for time-continual LLM pretraining using 114 Common Crawl dumps, demonstrating that replay and autoregressive schedules can match Oracle retraining on general web data with less compute, though trade-offs persist across domains.