Objective: Although the APOE ε4 allele is associated with more rapid decline in memory in healthy older adults, the significance of elevated cerebral β-amyloid (Aβ) load for longitudinal changes in cognition is unclear. Methods: Healthy and cognitively normal older adults (n = 141; mean age 76 years) underwent PET neuroimaging for cerebral Aβ, APOE genotyping, and cognitive assessment as part of their baseline assessment in the Australian Imaging Biomarkers and Lifestyle study. Cognitive function was reassessed 18 months later. Results: Linear mixed-model analyses adjusted for baseline cognitive function indicated that, relative to individuals with low cerebral Aβ, individuals with high cerebral Aβ showed significantly greater decline in working memory and verbal and visual episodic memory at 18 months. Compared with noncarriers, APOE ε4 carriers showed a greater decline in visual memory at the 18- month assessment. No interaction between APOE ε4 and cerebral Aβ load was observed for any measure of cognitive function. Conclusions: In this prospective study of healthy older adults, high cerebral Aβ load was associated with greater decline in episodic and working memory over 18 months. The APOE ε4 genotype was also associated with a decline in visual memory, although the effect was less than that observed for cerebral Aβ load.