Monoclonal antibodies targeting a variety of epitopes have been isolated from individuals previously infected with SARS-CoV-2, but the relative contributions of these different antibody classes to the polyclonal response remains unclear. Here we use a yeast-display system to map all mutations to the viral spike receptor-binding domain (RBD) that escape binding by representatives of three potently neutralizing classes of anti-RBD antibodies with high-resolution structures. We compare the antibody-escape maps to similar maps for convalescent polyclonal plasmas, including plasmas from individuals from whom some of the antibodies were isolated. While the binding of polyclonal plasma antibodies are affected by mutations across multiple RBD epitopes, the plasma-escape maps most resemble those of a single class of antibodies that target an epitope on the RBD that includes site E484. Therefore, although the human immune system can produce antibodies that target diverse RBD epitopes, in practice the polyclonal response to infection is skewed towards a single class of antibodies targeting an epitope that is already undergoing rapid evolution.