Chromosomes within eukaryotic cell nuclei at interphase are not positioned at random, since gene-rich chromosomes are predominantly found towards the interior of the cell nucleus across a number of cell types. The physical mechanisms that could drive and maintain the spatial segregation of chromosomes based on gene density are unknown. Here, we identify a mechanism for such segregation, showing that the territorial organization of chromosomes, another central feature of nuclear organization, emerges naturally from our model. Our computer simulations indicate that gene density-dependent radial segregation of chromosomes arises as a robust consequence of differences in non-equilibrium activity across chromosomes. Arguing that such differences originate in the inhomogeneous distribution of ATP-dependent chromatin remodeling and transcription machinery on each chromosome, we show that a variety of non-random positional distributions emerge through the interplay of such activity, nuclear shape and specific interactions of chromosomes with the nuclear envelope. Results from our model are in reasonable agreement with experimental data and we make a number of predictions that can be tested in experiments.