The worldwide decline of pollinators may negatively affect the fruit set of wild and cultivated plants. Here, we show that fruit set of the self-fertilizing highland coffee (Coffea arabica) is highly variable and related to bee pollination. In a comparison of 24 agroforestry systems in Indonesia, the fruit set of coffee could be predicted by the number of flower-visiting bee species, and it ranged from ca. 60% (three species) to 90% (20 species). Diversity, not abundance, explained variation in fruit set, so the collective role of a species-rich bee community was important for pollination success. Additional experiments showed that single flower visits from rare solitary species led to higher fruit set than with abundant social species. Pollinator diversity was affected by two habitat parameters indicating guild-specific nesting requirements: the diversity of social bees decreased with forest distance, whereas the diversity of solitary bees increased with light intensity of the agroforestry systems. These results give empirical evidence for a positive relationship between ecosystem functions such as pollination and biodiversity. Conservation of rainforest adjacent to adequately managed agroforestry systems could improve the yields of farmers.