Privacy-Oriented Text Generation in LLMs via Selective Fine-Tuning and Semantic Attention Masks

Abstract

This paper addresses the issue of sensitive entity exposure in text generation by large language models. It proposes a control method that combines selective fine-tuning with a semantic-guided masking strategy. First, a parameter selection mechanism is designed. By analyzing gradient responses in sensitive entity contexts, the method identifies a subset of parameters strongly related to sensitive expressions. Only this subset is updated during fine-tuning, allowing for fine-grained behavioral adjustment. Second, a semantic-guided masking strategy is introduced. It uses the model's semantic features to construct a probabilistic mask. This mask intervenes at the attention mechanism level to reduce the model's response strength to sensitive token positions, lowering the likelihood of explicit expression. The experimental section validates the effectiveness of the method across multiple dimensions. These include comparisons with mainstream fine-tuning strategies, ablation studies, intervention effects across structural layers, sensitivity recovery tests, and training step analysis. Results show that the proposed method significantly reduces the exposure rate of sensitive entities while maintaining or even improving contextual coherence and generation quality. It demonstrates strong practicality and stability. This method provides an effective path for secure generation control in large models. It suppresses potential privacy risks while preserving the model's expressive capabilities.