We investigate the electron dynamics of p-type modulation doped and undoped InGaAs/GaAs quantum dots using up-conversion photoluminescence at low temperature and room temperature. The rise time of the p-doped sample is significantly shorter than that of the undoped at low temperature. With increasing to room temperature the undoped sample exhibits a decreased rise time whilst that of the doped sample does not change. A relaxation mechanism of electron-hole scattering is proposed in which the doped quantum dots exhibit an enhanced and temperature independent relaxation due to excess built-in holes in the valence band of the quantum dots. In contrast, the rise time of the undoped quantum dots decreases significantly at room temperature due to the large availability of holes in the ground state of the valence band. Furthermore, modulation p-doping results in a shorter lifetime due to the presence of excess defects.