Asphalt concrete is a composite heterogeneous material comprised of aggregates bonded together by bitumen binder. The mechanical behaviour of this material shows great dependence on loading rates and time (e.g., creep, relaxation). Such heterogeneity in the internal structure and complex behaviour of asphalt concrete present a challenge for numerical methods to fully capture its responses under different loading rates and durations, especially when large deformation and crack development are present in the material. This study proposes a modelling approach capable of capturing the behaviour of asphalt concrete under different loading rates and durations. In this approach, the heterogeneous internal structure of asphalt concrete is naturally reproduced by the discrete element method (DEM). In parallel, a new inter-particle contact model is developed for the DEM to describe the grain-level behaviour of asphalt concrete. This contact model couples a viscoelastic-damage law with a cohesive-elastoplastic-damage law, enabling the contact model to characterise the rate and time dependency, viscoelastic damage, and plastic-damage behaviour of asphalt concrete. Moreover, thanks to the use of DEM, the proposed approach can naturally capture crack initiation and development. Through comparisons and verifications with a wide range of experiments on asphalt concrete, including the relaxation test, creep test, and dynamic semi-circular bending test, the proposed approach shows its capability of overcoming the limitations of previous DEM models in reproducing the complex behaviours of asphalt material under various loading conditions. Insights into the failure mechanisms of asphalt material under complex loading conditions and its transitional behaviours from diffuse to localised failure can be thus naturally derived from the proposed DEM approach. This study demonstrates the effectiveness of the proposed approach for investigating the behaviour of asphalt concrete.