Accurate and consistent segmentation of infant brain MR images plays an important role in quantifying patterns of early brain development, especially in longitudinal studies. However, due to rapid maturation and myelination of brain tissues in the first year of life, the intensity contrast of gray and white matter undergoes dramatic changes. In fact, the contrast inverse around 6-8 months of age, when the white and gray matter tissues are isointense and hence exhibit the lowest contrast, posing significant challenges for segmentation algorithms. In this paper, we propose a longitudinally guided level set method to segment serial infant brain MR images acquired from 2 weeks up to 1.5 years of age, including the isointense images. At each single-time-point, the proposed method makes optimal use of T1, T2 and the diffusion-weighted images for complimentary tissue distribution information to address the difficulty caused by the low contrast. Moreover, longitudinally consistent term, which constrains the distance across the serial images within a biologically reasonable range, is employed to obtain temporally consistent segmentation results. Application of our method on 28 longitudinal infant subjects, each with 5 longitudinal scans, shows that the automated segmentations from the proposed method match the manual ground-truth with much higher Dice Ratios than other single-modality, single-time-point based methods and the longitudinal but voxel-wise based methods. The software of the proposed method is publicly available in NITRC (http://www.nitrc.org/projects/ibeat).