Progressive Residual Warmup for Language Model Pretraining
Abstract
Progressive Residual Warmup (ProRes) improves transformer language model pretraining by implementing gradual layer activation that stabilizes training and accelerates convergence.
Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated by the logical dependency of sequentially stacked layers, we propose Progressive Residual Warmup (ProRes) for language model pretraining. ProRes implements an "early layer learns first" philosophy by multiplying each layer's residual with a scalar that gradually warms up from 0 to 1, with deeper layers taking longer warmup steps. In this way, deeper layers wait for early layers to settle into a more stable regime before contributing to learning. We demonstrate the effectiveness of ProRes through pretraining experiments across various model scales, as well as normalization and initialization schemes. Comprehensive analysis shows that ProRes not only stabilizes pretraining but also introduces a unique optimization trajectory, leading to faster convergence, stronger generalization and better downstream performance. Our code is available at https://github.com/dandingsky/ProRes.
Community
We show that explicitly scheduling residual contributions across depth improves Transformer optimization. Code is available at https://github.com/dandingsky/ProRes.
hot take: that simple per-layer residual scalar warmed up over training is enough to reorder optimization across depth and stabilize pretraining without exotic tricks. i kinda expected you’d need extra gymnastics, but letting shallow layers settle before deeper ones actually tracks how gradients flow in really big transformers. the breakdown on arxivlens was solid, nice quick walkthrough to sanity-check the idea: https://arxivlens.com/PaperView/Details/progressive-residual-warmup-for-language-model-pretraining-8624-0eb9f1f9
Thank you for the paper! I tried the proposed trick and it definitely improved the performance of our Protein Language Model... very simple and elegant idea!
Thank you for sharing! Glad it worked on PLMs!🤗
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Post-LayerNorm Is Back: Stable, ExpressivE, and Deep (2026)
- IMU-1: Sample-Efficient Pre-training of Small Language Models (2026)
- Stabilizing Native Low-Rank LLM Pretraining (2026)
- Revisiting the Shape Convention of Transformer Language Models (2026)
- ViT-5: Vision Transformers for The Mid-2020s (2026)
- Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks (2026)
- SimpleGPT: Improving GPT via A Simple Normalization Strategy (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper