Prostate cancer is a common malignancy, but only some tumors are lethal. Accurately identifying these tumors will improve clinical practice and instruct research. Aggressive cancers often have distinctive pathologies, including intraductal carcinoma of the prostate (IDC-P) and ductal adenocarcinoma. Here, we review the importance of these pathologies because they are often overlooked, especially in genomics and preclinical testing. Pathology, genomics, and patient-derived models show that IDC-P and ductal adenocarcinoma accompany multiple markers of poor prognosis. Consequently, "knowing what is growing" will help translate preclinical research to pinpoint and treat high-risk prostate cancer in the clinic.