Tag: Fine-tuning
All the articles with the tag "Fine-tuning".
-
Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Published: at 11:25 AM
85.01 🤔本文通过编码-解码框架研究任务向量在上下文学习中的浮现与有效性，提出任务可解码性（TD）指标预测ICL性能，并发现微调早期层比后期层更能提升任务编码和性能。
-
Task Specific Pruning with LLM-Sieve: How Many Parameters Does Your Task Really Need?
Published: at 11:33 AM 89.05 🤔LLM-Sieve提出了一种任务特定的剪枝框架，通过联合低秩投影和遗传算法实现差异化剪枝，在保持1-5%精度损失下减少20-75%的参数，显著优于现有方法，并与LoRA微调和量化兼容。
-
When Models Reason in Your Language: Controlling Thinking Trace Language Comes at the Cost of Accuracy
Published: at 11:29 AM 86.23 🤔本文通过XReasoning基准揭示了大型推理模型在多语言推理中语言匹配与答案准确性之间的权衡，并通过提示破解和少样本后训练方法提高语言匹配率，但以牺牲准确性为代价，凸显了当前模型的局限性。
-
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
Published: at 01:15 PM 87.89 🤔本文提出P² Law作为剪枝后大型语言模型后训练的首个缩放定律，通过结合模型规模、后训练数据量、剪枝率和初始损失预测后训练损失，并在多种剪枝方法和模型上验证其有效性和部分泛化能力。
-
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
Published: at 01:15 PM 87.30 🤔本文提出LoRA-SB方法，通过基于全参数微调第一步梯度近似的初始化策略优化低秩微调，在参数量减少27-90倍的情况下，显著超越LoRA-XS并接近全参数微调性能。