We address the issue of modifiers of risk for persons who are, on average, at high risk of colorectal cancer due to carrying germline genetic mutations. We discuss how to estimate risk and assess modifiers using data from mutation-carrying families. A critical analytical issue is the adjustment for how families have been sampled (ascertainment). Only a few published studies have made appropriate adjustments and have mostly reported lifetime risks of less than 50%. Incomplete penetrance and differences in risk by subcategories are consistent with the existence of genetic and/or environmental risk modifiers. Data from large cohorts of carriers are required to study such modifiers with precision.