To improve diagnosis accuracy for gear fault diagnosis under imbalanced and small sample conditions, a method combining the Un-threshold Recurrence Plots - Conditional Variational Autoencoder-Mean Generative Adversarial Network (URP-CVAE-MGAN) combined with Dropkey-Vision Transformer (DViT) is proposed. First, gear vibrational signals are transformed into Recurrence Plots (RP) images to extract more fault features without threshold effect. Then, a conditional variable and mean feature difference function are incorporated into VAE-GAN to improve the quality and diversity of generated samples, balancing the imbalanced and small sample sets. Dropkey is applied to the diagnosis model Vision Transformer to capture more fault information, improving diagnosis accuracy across various fault types and severities for gear. Finally, the proposed method is verified based on two datasets, demonstrating a significant accuracy improvement of up to 7.84% under the imbalanced and small samples, and confirming its feasibility and superiority.