The brains of males and females differ, not only in regions specialized for reproduction, but also in other regions (controlling cognition, for example) where sex differences are not necessarily expected. Moreover, males and females are differentially susceptible to neurological and psychiatric disease. What are the origins of these sex differences? Two major sources of sexually dimorphic information could lead to sex differences in brain function. Male and female brain cells carry a different complement of sex chromosome genes and are influenced throughout life by a different mix of gonadal hormones. Until recently all sex differences in the brain have been attributed to the differential action of gonadal hormones. Recent findings, however, suggest that brain cells that differ in their genetic sex are not equivalent, and that difference may contribute to sex differences in brain function. Here we discuss evidence for sex chromosome effects on both neural and nonneural systems, which together provide support for the idea that XX and XY cells differentiate even before they are influenced by gonadal hormones, and even if they are exposed to similar levels of gonadal steroids. Fortunately, new model systems for studying sex chromosome effects have recently been developed, and they should help in testing further the role of sex chromosome genes.