For broadband receiving systems, the imbalanced gain of front-end devices leads to the deterioration of in-band amplitude flatness of broadband systems, which seriously affects the accuracy of system measurement result. Therefore, it is necessary to compensate the distorted channel transmission characteristics to improve in-band amplitude flatness. The existing technology mainly adopts the frequency domain sampling method to design the compensation filter, but the order of the filter designed by this method is too high, usually reaching more than 200 orders. Aiming at the defects that the traditional LMS and RLS adaptive filtering algorithms can’t take into account the convergence speed and steady-state error, this paper designed two UWB flatness compensation technologies respectively based on LMS with variable step size factor and RLS with variable forgetting factor for the engineering application of test instruments. The simulation results showed that the proposed two methods can reduce the order of the filter to less than 100 orders on the basis of ensuring good amplitude flatness optimization.