Climate changes will have an impact on food production and will require costly adaptive responses. Adapting to a changing environment will be particularly challenging in sub-Saharan Africa where climate change is expected to have a major impact. However, one important phenomenon that is often overlooked and is poorly documented is the ability of agro-systems to rapidly adapt to environmental variations. Such an adaptation could proceed by the adoption of new varieties or by the adaptation of varieties to a changing environment. In this study, we analyzed these two processes in one of the driest agro-ecosystems in Africa, the Sahel. We performed a detailed study in Niger where pearl millet is the main crop and covers 65% of the cultivated area. To assess how the agro-system is responding to recent recurrent drought, we analyzed samples of pearl millet landraces collected in the same villages in 1976 and 2003 throughout the entire cultivated area of Niger. We studied phenological and morphological differences in the 1976 and 2003 collections by comparing them over three cropping seasons in a common garden experiment. We found no major changes in the main cultivated varieties or in their genetic diversity. However, we observed a significant shift in adaptive traits. Compared to the 1976 samples, samples collected in 2003 displayed a shorter lifecycle, and a reduction in plant and spike size. We also found that an early flowering allele at the PHYC locus increased in frequency between 1976 and 2003. The increase exceeded the effect of drift and sampling, suggesting a direct effect of selection for earliness on this gene. We conclude that recurrent drought can lead to selection for earlier flowering in a major Sahelian crop. Surprisingly, these results suggest that diffusion of crop varieties is not the main driver of short term adaptation to climatic variation.