Accumulation of β-amyloid (Aβ) in the brain is associated with memory decline in healthy individuals as a prelude to Alzheimer's disease (AD). Genetic factors may moderate this decline. We examined the role of apolipoprotein E (ε4 carrier ε4 + , ε4 non-carrier ε4) and brain-derived neurotrophic factor (BDNF Val/Val, BDNF Met) in the extent to which they moderate Aβ-related memory decline. Healthy adults (n=333, M age =70 years) enrolled in the Australian Imaging, Biomarkers and Lifestyle study underwent Aβ neuroimaging. Neuropsychological assessments were conducted at baseline, 18-, 36- and 54-month follow-ups. Aβ positron emission tomography neuroimaging was used to classify participants as Aβ - or Aβ +. Relative to Aβ - ε4 -, Aβ + ε4 + individuals showed significantly faster rates of cognitive decline over 54 months across all domains (d=0.40-1.22), while Aβ + ε4 - individuals showed significantly faster decline only on verbal episodic memory (EM). There were no differences in rates of cognitive change between Aβ - ε4 - and Aβ - ε4 + groups. Among Aβ + individuals, ε4 + /BDNF Met participants showed a significantly faster rate of decline on verbal and visual EM, and language over 54 months compared with ε4 - /BDNF Val/Val participants (d=0.90-1.02). At least two genetic loci affect the rate of Aβ-related cognitive decline. Aβ + ε4 + /BDNF Met individuals can expect to show clinically significant memory impairment after 3 years, whereas Aβ + ε4 + /BDNF Val/Val individuals can expect a similar degree of impairment after 10 years. Little decline over 54 months was observed in the Aβ - and Aβ + ε4 - groups, irrespective of BDNF status. These data raise important prognostic issues in managing preclinical AD, and should be considered in designing secondary preventative clinical trials.