Actin-based vesicular trafficking of Cdc42, leading to a polarized concentration of the GTPase, has been implicated in cell polarization, but it was recently debated whether this mechanism allows stable maintenance of cell polarity. Here we show that endocytosis and exocytosis are spatially segregated in the polar plasma membrane, with sites of exocytosis correlating with microdomains of higher concentration and slower diffusion of Cdc42 compared with surrounding regions. Numerical simulations using experimentally obtained diffusion coefficients and trafficking geometry revealed that non-uniform membrane diffusion of Cdc42 in fact enables temporally sustained cell polarity. We show further that phosphatidylserine, a phospholipid recently found to be crucial for cell polarity, is enriched in Cdc42 microdomains. Weakening a potential interaction between phosphatidylserine and Cdc42 enhances Cdc42 diffusion in the microdomains but impedes the strength of polarization. These findings demonstrate a critical role for membrane microdomains in vesicular trafficking-mediated cell polarity.