Tag: Pre-training

All the articles with the tag "Pre-training".

• Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

  Published: 4 May, 2025 |  at  04:33 PM

  

  69.21 🤔

  本文提出Mixture of Sparse Attention (MoSA)方法，通过专家选择路由实现基于内容的稀疏注意力，显著提高了Transformer模型在相同计算预算下的语言建模性能，并优化了资源使用。

• Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare

  Published: 4 May, 2025 |  at  04:31 PM

  

  67.24 🤔

  本文通过实证实验指导在医疗专业应用中语言模型的选择，强调微调小语言模型和领域特定预训练的显著优势，使其在特定任务上超越零-shot 大语言模型。

• X-Fusion: Introducing New Modality to Frozen Large Language Models

  Published: 4 May, 2025 |  at  04:31 PM

  

  66.52 🤔

  本文提出X-Fusion框架，通過凍結LLM參數並添加雙塔結構，高效實現多模態理解和生成，同時保留原始語言能力。

• Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

  Published: 4 May, 2025 |  at  04:27 PM

  

  66.29 🤔

  本文提出Flab-Pruner，一种结合词汇、层和FFN剪枝的统一结构剪枝方法，通过KL散度优化和自定义微调策略，在减少代码LLM参数的同时保持高性能和效率。

• Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders

  Published: 10 May, 2025 |  at  10:58 AM

  

  61.78 🤔

  This paper uses Sparse Autoencoders to identify and manipulate language-specific features in Large Language Models, introducing a monolinguality metric, demonstrating context dependency via code-switching, and enhancing steering vectors for better control over multilingual generation while revealing significant language-specific impacts through ablation studies.