The widespread deployment of small base stations in 5G heterogeneous networks(HetNet)can improve network capacity and user rates,but the dense deployment will also cause severe interference and higher energy consumption problems.In order to maximize the network energy efficiency(EE)and guarantee the user quality of service(QoS),this paper presents a resource allocation scheme that embeds energy harvester power supply in small cell base stations.Firstly,for the downlink of the network system,the spectrum and small base station transmit power allocation problem was modeled as a multi-objective optimization problem to jointly optimize system energy efficiency and user satisfaction.Secondly,a multi-objective actor-critic(MAC)resource allocation algorithm based on deep reinforcement learning was proposed to solve the established optimization model.Finally,simulation results show that compared with other traditional learning algorithms,the energy efficiency of the proposed algorithm is improved by 11.96%～12.37%,and the user satisfaction is improved by 11.45%～27.37%.