The specification of embryonic cells to produce the retina begins at early embryonic stages as a multi-step process that gradually restricts fate potentials. First, a subset of embryonic cells becomes competent to form retina by their lack of expression of endo-mesoderm-specifying genes. From these cells, a more restricted subset is biased to form retina by virtue of their close proximity to sources of bone morphogenetic protein antagonists during neural induction. During gastrulation, the definitive RSCs (retinal stem cells) are specified as the eye field by interactions with underlying mesoderm and the expression of a network of retina-specifying genes. As the eye field is transformed into the optic vesicle and optic cup, a heterogeneous population of RPCs (retinal progenitor cells) forms to give rise to the different domains of the retina: the optic stalk, retinal pigmented epithelium and neural retina. Further diversity of RPCs appears to occur under the influences of cell-cell interactions, cytokines and combinations of regulatory genes, leading to the differentiation of a multitude of different retinal cell types. This review examines what is known about each sequential step in retinal specification during normal vertebrate development, and how that knowledge will be important to understand how RSCs might be manipulated for regenerative therapies to treat retinal diseases.