Enhancing Vision Transformers for Image Generation: A Hierarchical GAN with Triplet Attention and Consistency Regularization

Abstract

This paper proposes a novel hierarchical generator adversarial network based on Vision Transformers (ViT) for unconditional image generation. To address common challenges such as structural inconsistency and unstable training in GANs, we introduce the Triplet Attention mechanism within the generator, enhancing the structural soundness of generated images without increasing the model’s parameter size. Additionally, a consistency regularization term is integrated into the loss function, improving the training stability and robustness to noise while mitigating overfitting. The effectiveness of the proposed method is demonstrated through extensive experiments on the CIFAR-10 and STL-10 datasets, where our framework outperforms TransGAN and other CNN-based GANs in both FID and IS metrics. Despite the simplicity of our architecture, which contains only three transformer layers, we achieve promising results, laying the groundwork for further enhancements in high-resolution image generation.