Wind turbine gearboxes operate under alternating loads and complex noise, resulting in vibration signals characterized by strong interference, nonlinearity, and non-stationarity—factors that complicate fault diagnosis. To address this, a novel method is proposed that combines a gravitational search-enhanced whale optimization algorithm (GSWOA) with variational mode decomposition (VMD) and improved refined composite multiscale sample entropy (IRCMSE). GSWOA adaptively tunes VMD parameters, enhancing decomposition and reducing mode mixing and edge effects. IRCMSE, derived via enhanced coarse-graining, boosts sensitivity to weak fault signatures across multiple time scales. Extracted features are processed by a CNN-BiLSTM model that merges spatial and temporal learning for accurate fault classification. Experimental validation on the WFD-1000 platform confirms superior performance in signal reconstruction, feature separation, and fault identification, supporting the method’s suitability for intelligent diagnostics under complex conditions.