Face processing in the human brain recruits a widespread cortical network based mainly in the ventral and lateral temporal and occipital lobes. However, the extent to which activity within this network is driven by different face properties versus being determined by the manner in which faces are processed (as determined by task requirements) remains unclear. We combined a functional magnetic resonance adaptation paradigm with three target detection tasks, where participants had to detect a specific identity, emotional expression, or direction of gaze, while the task-irrelevant face properties varied independently. Our analysis focused on differentiating the influence of task demands and the processing of stimulus changes within the neural network underlying face processing. Results indicated that the fusiform and inferior occipital gyrus do not respond as a function of stimulus change (such as identity), but rather their activity depends on the task demands. Specifically, we hypothesize that, whether the task encourages a configural- or a featural-processing strategy determines activation. Our results for the superior temporal sulcus were even more specific in that we only found greater responses to stimulus changes that may engage featural processing. These results contribute to our understanding of the functional anatomy of face processing and provide insights into possible compensatory mechanisms in face processing.