Surface roughness is influenced by various factors with uncertainty characteristics, and roughness reliability can be used for the assessment of the surface quality of CNC milling. The paper develops a method for the assessment of surface quality by considering the coupling effect and uncertainty characteristic of various factors. According to the milling kinematics theory, the milling surface topography simulation was conducted by discretizing the cutting edge, machining time, and workpiece. Considering the coupling effect of various factors, a roughness prediction model is established by the SSA-LSSVM, and its prediction accuracy reaches more than 95%. Then, the roughness reliability model was developed by applying the response surface methodology to achieve the assessment of surface quality. The proposed method is verified by the milling experiments. The maximum values of the relative errors between the simulation and experimental results of the surface roughness and roughness reliability are 9% and 1.5% respectively, indicating the correctness of the method proposed in the paper.