Structured Pruning-Driven High-Expressive Low-Rank Adapter Learning for Large Language Models

Abstract

This paper addresses the challenge of balancing expressive power and resource overhead in efficient fine-tuning of large language model parameters. It proposes a structured pruning-driven high-expressive low-rank adapter learning method. By freezing the main model parameters, low-rank incremental updates are injected only into key linear transformation layers to achieve lightweight task adaptation. To overcome the expression bottleneck caused by fixed low rank, the method employs residual injection to controllably adjust hidden representations and introduces differentiable gating vectors in the low-rank channel dimension, transforming adapter capacity allocation into a learnable structural selection process, thereby achieving interpretable sparsity at the structural unit granularity. Furthermore, a unified optimization objective is constructed, consisting of the task objective, structured sparsity regularization, and budget constraints. Channel-level structured pruning is achieved using gating sparsity induction, and the overall trainable capacity is controlled by budget consistency constraints, forming a sparse, low-rank integrated parameterization and training mechanism. This framework organically integrates structured sparsity and low-rank representation, improving the effective expression density of the update subspace while maintaining parameter efficiency. It provides a controllable, well-structured, and easily implementable technical solution for adapter fine-tuning in resource-constrained deployments.