The adaptor proteins AP-2 and AP-1/GGAs are essential components of clathrin coats at the plasma membrane and trans-Golgi network, respectively. The adaptors recruit accessory proteins to clathrin-coated pits, which is dependent on the adaptor ear domains engaging short peptide motifs in the accessory proteins. Here, we perform an extensive mutational analysis of a novel WXXF-based motif that functions to mediate the binding of an array of accessory proteins to the alpha-adaptin ear domain of AP-2. Using nuclear magnetic resonance and mutational studies, we identified WXXF-based motifs as major ligands for a site on the alpha-ear previously shown to bind the DPW-bearing proteins epsin 1/2. We also defined the determinants that allow for specific binding of the alpha-ear motif to AP-2 as compared to those that allow a highly related WXXF-based motif to bind to the ear domains of AP-1/GGAs. Intriguingly, placement of acidic residues around the WXXF cores is critical for binding specificity. These studies provide a structural basis for the specific recruitment of accessory proteins to appropriate sites of clathrin-coated vesicle formation.