Understanding the immune responses elicited by SARS-CoV-2 infection is critical in terms of protection against reinfection and, thus, for public health policy and vaccine development for COVID-19. In this study, using either live SARS-CoV-2 particles or retroviruses pseudotyped with the SARS-CoV-2 S viral surface protein (Spike), we studied the neutralizing antibody (nAb) response in serum samples from a cohort of 140 SARS-CoV-2 qPCR-confirmed infections, including patients with mild symptoms and also more severe forms, including those that required intensive care. We show that nAb titers correlated strongly with disease severity and with anti-spike IgG levels. Indeed, patients from intensive care units exhibited high nAb titers; conversely, patients with milder disease symptoms had heterogeneous nAb titers, and asymptomatic or exclusive outpatient-care patients had no or low nAbs. We found that nAb activity in SARS-CoV-2-infected patients displayed a relatively rapid decline after recovery compared to individuals infected with other coronaviruses. Moreover, we found an absence of cross-neutralization between endemic coronaviruses and SARS-CoV-2, indicating that previous infection by human coronaviruses may not generate protective nAbs against SARS-CoV-2. Finally, we found that the D614G mutation in the spike protein, which has recently been identified as the current major variant in Europe, does not allow neutralization escape. Altogether, our results contribute to our understanding of the immune correlates of SARS-CoV-2-induced disease, and rapid evaluation of the role of the humoral response in the pathogenesis of SARS-CoV-2 is warranted.