Tag: Efficiency
All the articles with the tag "Efficiency".
-
Incentivizing Strong Reasoning from Weak Supervision
Published: at 11:19 AM
87.07 🤔本文提出弱到强推理（W2SR）范式，通过显著较弱教师模型生成的结构化链式思维轨迹对强学生模型进行监督微调，以低成本方式显著提升其推理能力，接近甚至超越昂贵的强化学习效果。
-
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization
Published: at 11:08 AM 86.94 🤔This paper introduces a fine-tuning strategy for LLMs that leverages the unequal importance of attention matrices and customized learning rates to enhance efficiency, demonstrating through theoretical analysis and experiments on GLUE benchmarks that fine-tuning only Wq and Wv with higher learning rates for Wv can match or exceed full fine-tuning performance with fewer parameters.
-
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
Published: at 11:25 AM 86.91 🤔R-LoRA通过多头随机化（包括多头Dropout和随机初始化）增强了LoRA在多任务学习中的性能，有效提升了任务特定知识的捕获能力，同时降低了GPU内存使用和训练时间。
-
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models
Published: at 11:21 AM 86.90 🤔本文提出 ALPS 算法，通过基于权重分布的参数对齐分布分数（sPAD）定位任务敏感注意力头并剪枝，仅更新 10% 的注意力参数即在通用、数学和代码任务上实现性能提升，同时展现头部可转移性和知识遗忘缓解效果。
-
R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search
Published: at 11:22 AM 86.90 🤔R1-Compress通过块级压缩和块间搜索机制有效压缩长链式推理（Long-CoT），在减少约20% token使用量的同时保持了与基线接近的推理准确率（92.4% vs 93.0%）。