There is evidence that pluripotency of mouse embryonic stem (ES) cells is associated with the activity of a network of transcription factors with Sox2, Oct4, and Nanog at the core. Using fluorescent reporters for the expression of Nanog, we observed that a population of ES cells is best described by a dynamic distribution of Nanog expression characterized by two peaks defined by high (HN) and low (LN) Nanog expression. Typically, the LN state is 5%-20% of the total population, depending on the culture conditions. Modelling of the activity of Nanog reveals that a simple network of Oct4/Sox2 and Nanog activity can account for the observed distribution and its properties as long as the transcriptional activity is tuned by transcriptional noise. The model also predicts that the LN state is unstable, something that is born out experimentally. While in this state, cells can differentiate. We suggest that transcriptional fluctuations in Nanog expression are an essential element of the pluripotent state and that the function of Sox2, Oct4, and Nanog is to act as a network that promotes and maintains transcriptional noise to interfere with the differentiation signals.