Molecular dynamics simulations provide a route to studying the dynamics of lipid bilayers at atomistic or near atomistic resolution. Over the past 10 years or so, molecular dynamics simulations have become an established part of the biophysicist's tool kit for the study of model biological membranes. As simulation time scales move from tens to hundreds of nanoseconds and beyond, it is timely to re-evaluate the accuracy of simulation models. We describe a comparative analysis of five freely available force fields that are commonly used to model lipid bilayers. We focus our analysis on 1,2-dipalmitoyl-sn-glycero-3-phosphocholine (DPPC) and 1-palmitoyl-2-oleoyl-sn-glycero-3-phosphocholine (POPC) bilayers. We show that some bilayer properties have a pronounced force field dependence, while others are less sensitive. In general, we find strengths and weaknesses, with respect to experimental data, in all of the force fields we have studied. We do, however, find some combinations of simulation and force field parameters that should be avoided when simulating DPPC and POPC membranes. We anticipate that the results presented for some of the membrane properties will guide future improvements for several force fields studied in this work.