Age and APOE are the most robust risk factors for dementia and cognitive decline, but the underlying neurobiology remains unclear. We examined the extent to which the hallmark pathologies of Alzheimer's disease, Lewy body disease, and cerebrovascular diseases account for the association of age and APOE with decline in episodic memory versus nonepisodic cognitive abilities. Up to 20 waves of longitudinal cognitive data were collected from 858 autopsied participants in 2 ongoing clinical-pathologic cohort studies of aging. Neuropathologic examinations quantified measures of beta amyloid (Aβ) plaque, mesial temporal and neocortical neurofibrillary tangles, macro- and microinfarcts, and neocortical Lewy bodies. Random coefficient models estimated person-specific slopes of decline in episodic memory and nonepisodic cognition. Path analysis examined the relation of age, APOE, and the 6 pathologic indices to the slopes of cognitive decline. The effect of age on decline in episodic memory was mediated by Aβ, mesial temporal and neocortical tau tangles, and macroscopic infarcts; age on decline in nonepisodic cognition was mediated by Aβ, neocortical tangles, and macroscopic infarcts. The effect of APOE on decline in episodic memory was mediated by Aβ, mesial temporal and neocortical tangles, and neocortical Lewy bodies; APOE on nonepisodic cognition was mediated by Aβ, neocortical tangles, and neocortical Lewy bodies. There were no direct effects of age and APOE on decline after accounting for these pathologic pathways.