Approximately 2-15% of couples experience infertility, and around half of these cases are attributed to male infertility. We previously identified TBC1D21 as a sterility-related RabGAP gene derived from infertile men. However, the in vivo function of TBC1D21 in male fertility remains unclear. Here, we show that loss of Tbc1d21 in mice resulted in male infertility, characterized by defects in sperm tail structure and diminished sperm motility. The mitochondria of the sperm-tail had an abnormal irregular arrangement, abnormal diameter, and structural defects. Moreover, the axoneme structure of sperm tails was severely disturbed. Several TBC1D21 interactors were selected via proteomic analysis and functional grouping. Two of the candidate interactors, a subunit protein of translocase in the outer membrane of mitochondria (TOMM20) and an inner arm component of the sperm tail axoneme (Dynein Heavy chain 7, DNAH7), confirmed in vivo physical co-localization with TBC1D21. In addition, TOMM20 and DNAH7 detached and dispersed outside the axoneme in Tbc1d21-deficient sperm, instead of aligning with the axoneme. From a clinical perspective, the transcript levels of TBC1D21 in sperm from teratozoospermia cases were significantly reduced when compared with those in normozoospermia. We concluded that TBC1D21 is critical for mitochondrial and axoneme development of mammalian sperm.