Childhood adversity, such as physical, sexual, and verbal abuse, as well as neglect and family conflict, is a risk factor for schizophrenia. Such adversity can lead to disruptions of cognitive function during development, undermining intellectual capabilities and academic achievement. Schizophrenia is a neurodevelopmental disorder that is associated with cognitive impairments that may become evident during childhood. The Australian Schizophrenia Research Bank database comprises a large community cohort (N = 1169) in which we previously identified 3 distinct cognitive groups among people with schizophrenia: (1) Compromised, current, and estimated premorbid cognitive impairment; (2) Deteriorated, substantial decline from estimated premorbid function; and (3) Preserved, performing in the normal cognitive range without decline. The compromised group displayed the worst functional and symptom outcomes. Here, we extend our previous work by assessing the relationship among these categories of cognitive abilities and reported childhood adversity in 836 patients and healthy controls. Exploratory factor analysis of the Childhood Adversity Questionnaire revealed 3 factors (lack of parental involvement; overt abuse; family breakdown and hardship). People with schizophrenia reported significantly more childhood adversity than healthy controls on all items and factors. People with schizophrenia in the compromised group reported significantly more lack of parental involvement and family breakdown and hardship and lower socioeconomic status than those in the deteriorated group. The cognitive groups were not related to family history of psychosis. These findings identify specific social and family factors that impact cognition, highlighting the important role of these factors in the development of cognitive and functional abilities in schizophrenia.