Tag: Efficiency

All the articles with the tag "Efficiency".

• Large Language Model Compression with Global Rank and Sparsity Optimization

  Published: 11 May, 2025 |  at  11:14 AM

  

  77.26 🤔

  This paper introduces a two-stage LLM compression method using RPCA for low-rank and sparse decomposition and probabilistic pruning via policy gradient, outperforming state-of-the-art techniques at a 50% compression ratio while automatically adapting to layer-wise redundancy without manual thresholds or extensive fine-tuning.

• LLM-e Guess: Can LLMs Capabilities Advance Without Hardware Progress?

  Published: 12 May, 2025 |  at  11:20 AM

  

  76.16 🤔

  This paper introduces a framework to classify algorithmic innovations in LLMs as compute-dependent or compute-independent, demonstrating through small-scale GPT-2 experiments that compute-independent advancements like FlashAttention can yield up to 3.5× compute-equivalent gains even under hardware constraints, challenging the efficacy of hardware-focused AI regulation.

• COSMOS: Predictable and Cost-Effective Adaptation of LLMs

  Published: 10 May, 2025 |  at  11:05 AM

  

  75.99 🤔

  COSMOS introduces a cost-effective framework to predict performance and cost of LLM adaptation strategies like QLoRA fine-tuning and retrieval-augmented ICL, achieving high accuracy (1.09% MAE) and reducing computational costs by 92.72% across eight diverse benchmarks.

• Looped Transformers for Length Generalization

  Published: 7 May, 2025 |  at  08:42 AM

  

  74.13 🤔

  本文提出Looped Transformers方法，通过循环结构和自适应步数显著提升了Transformer在算法任务上的长度泛化能力，在多种任务中优于传统方法。

• R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training

  Published: 6 May, 2025 |  at  11:18 PM

  

  73.79 🤔

  R&B框架通过基于语义相似性的数据重新分组和梯度驱动的动态权重调整，以极低的计算开销（0.01%）在自然语言和多模态任务中匹配或超越现有数据混合策略，提升了基础模型训练效率。