Tag: Scaling Laws
All the articles with the tag "Scaling Laws".
-
From Words to Worlds: Compositionality for Cognitive Architectures
Published: at 11:24 AM
91.89 🤔本文通过设计三种任务评估大型语言模型（LLMs）的组合性能力，发现模型规模扩大通常提升组合性表现，而指令微调效果不一致，提示组合性对性能提升的解释力有限。
-
Parallel Scaling Law for Language Models
Published: at 11:28 AM 89.78 🤔本文提出并行扩展（PARSCALE）方法，通过增加训练和推理时的并行计算流（P）来提升语言模型能力，理论和实验表明P流相当于参数扩展O(log P)，并在低资源场景下展现出更高的推理效率。
-
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
Published: at 01:15 PM 87.89 🤔本文提出P² Law作为剪枝后大型语言模型后训练的首个缩放定律，通过结合模型规模、后训练数据量、剪枝率和初始损失预测后训练损失，并在多种剪枝方法和模型上验证其有效性和部分泛化能力。
-
LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Published: at 06:16 PM 86.80 🤔LENSLLM introduces a Hessian-based PAC-Bayes framework and NTK-based scaling model for LLM selection, achieving up to 91.1% accuracy and 88.5% computational cost reduction by modeling fine-tuning dynamics across diverse tasks.
-
Superposition Yields Robust Neural Scaling
Published: at 11:17 PM 86.47 🤔本文通过玩具模型和实际LLMs分析，揭示了超位置作为神经扩展律的重要机制，在强超位置下损失与模型维度成反比，与特征频率分布无关，从而解释了损失随模型规模幂律下降的现象。