Most neuropsychological research on the perception of emotion concerns the perception of faces. Yet in everyday life, hand actions are also modulated by our affective state, revealing it, in turn, to the observer. We used functional magnetic resonance imaging (fMRI) to identify brain regions engaged during the observation of hand actions performed either in a neutral or an angry way. We also asked whether these are the same regions as those involved in perceiving expressive faces. During the passive observation of emotionally neutral hand movements, the fMRI signal increased significantly in dorsal and ventral premotor cortices, with the exact location of the 'peaks' distinct from those induced by face observation. Various areas in the extrastriate visual cortex were also engaged, overlapping with the face-related activity. When the observed hand action was performed with emotion, additional regions were recruited including the right dorsal premotor, the right medial prefrontal cortex, the left anterior insula and a region in the rostral part of the supramarginal gyrus bilaterally. These regions, except for the supramarginal gyrus, were also activated during the perception of angry faces. These results complement the wealth of studies on the perception of affect from faces and provide further insights into the processes involved in the perception of others underlying, perhaps, social constructs such as empathy.