To clarify associations between APOE ϵ4 allele and age on longitudinal rates of β-amyloid (Aβ) accumulation within Aβ+ and Aβ-older individuals without dementia. Methods: We analyzed 595 older adults without dementia classified cross-sectionally as Aβ-(n = 325) and Aβ+ (n = 270) using longitudinal florbetapir PET. The influence of age and APOE genotype on longitudinal accumulation of Aβ was examined with linear mixed models. Results: APOE ϵ4 and older age were associated with higher risk of being classified as Aβ+ at baseline. The annual rate of Aβ accumulation was significantly greater than zero for Aβ-ϵ3 (0.0021 ± 0.0007 standardized uptake value ratio [SUVR] units) and Aβ-ϵ4 (0.0044 ± 0.0010 SUVR units), as well as Aβ+ ϵ3 (0.0141 ± 0.0019 SUVR units) and Aβ+ ϵ4 (0.0126 ± 0.0018 SUVR units). Aβ accumulation was significantly faster in Aβ-ϵ4 compared to Aβ-ϵ3 and Aβ-ϵ2. Rates of Aβ accumulation did not differ significantly between Aβ+ APOE groups. Older age was associated with higher rates of Aβ accumulation in the Aβ-group. Conclusions: APOE ϵ4 carriage and older age were predictors of longitudinal Aβ accumulation within the Aβ-group but not the Aβ+ group. APOE ϵ2 carriage was protective against longitudinal Aβ accumulation within the Aβ-group. APOE genotype in conjunction with chronologic age may aid in participant selection for primary prevention trials aimed at halting Aβ accumulation before abnormal levels are reached.