Autism spectrum disorder (ASD) and specific language impairment (SLI) are developmental disorders exhibiting language deficits, but it is unclear whether they arise from similar etiologies. Language impairments have been described in family members of children with ASD and SLI, but few studies have quantified them. In this study, we examined IQ, language, and reading abilities of ASD and SLI children and their first-degree relatives to address whether the language difficulties observed in some children with ASD are familial and to better understand the degree of overlap between these disorders and their broader phenotypes. Participants were 52 autistic children, 36 children with SLI, their siblings, and their parents. The ASD group was divided into those with (ALI, n=32) and without (ALN, n=20) language impairment. Relationships between ASD severity and language performance were also examined in the ASD probands. ALI and SLI probands performed similarly on most measures while ALN probands scored higher. ALN and ALI probands' language scores were not related to Autism Diagnostic Interview-Revised and Autism Diagnostic Observation Schedule algorithm scores. SLI relatives scored lowest on all measures, and while scores were not in the impaired range, relatives of ALI children scored lower than relatives of ALN children on some measures, though not those showing highest heritability in SLI. Given that ALI relatives performed better than SLI relatives across the language measures, the hypothesis that ALI and SLI families share similar genetic loading for language is not strongly supported.