Quantum key distribution allows secret key generation with information theoretical security. It can be realized with photonic integrated circuits to benefit the tiny footprints and the large-scale manufacturing capacity. Continuous-variable quantum key distribution is suitable for chip-based integration due to its compatibility with mature optical communication devices. However, the quantum signal power control compatible with the mature photonic integration process faces difficulties on stability, which limits the system performance and causes the overestimation of a secret key rate that opens practical security loopholes. Here, a highly stable chip-based quantum signal power control scheme based on a biased Mach-Zehnder interferometer structure is proposed, theoretically analyzed, and experimentally implemented with standard silicon photonic techniques. Simulations and experimental results show that the proposed scheme significantly improves the system stability, where the standard deviation of the secret key rate is suppressed by an order of magnitude compared with the system using traditional designs, showing a promising and practicable way to realize a highly stable continuous-variable quantum key distribution system on chip.