Task-Aware Regularized Continual Fine-Tuning for Mitigating Catastrophic Forgetting in Large Language Models

Abstract

This paper addresses the problem of catastrophic forgetting and generalization degradation in large language models during multi-task continual fine-tuning. To solve this problem, we propose a Task-aware Regularized Continual Fine-tuning (TRCF) method. The method consists of two core mechanisms: Task-aware Parameter Selection (TaPS) and Multi-stage Regularization Alignment (MRA). TaPS analyzes the semantic relevance and parameter sensitivity between tasks to adaptively select a subset of parameters for updating. This helps suppress interference and conflict between old and new tasks and improves the model's ability to retain prior knowledge. MRA constructs a regularization evolution path across task stages to guide the model in maintaining semantic consistency in the representation space during continual fine-tuning. This enhances the stability and effectiveness of multi-stage knowledge integration. Empirical studies conducted across multiple task stages show that the proposed method outperforms mainstream continual learning approaches in terms of average retention, new task accuracy, and overall performance. Further experiments on hyperparameters and data sensitivity confirm that the framework demonstrates strong robustness and adaptability under different learning rates, regularization strengths, data scales, and input structures. The proposed method provides an efficient and scalable solution for continual task input, dynamic structure learning, and parameter evolution control. It offers both theoretical support and methodological guidance for achieving task generalization and memory stability in large-scale pretrained language models.