It has been argued that through a process of internal 'simulation', we automatically map observed actions directly onto our motor system to facilitate imitation. Instead, here we show that observed actions influence the kinematic parameters of manual responses in a dynamic, context-dependent fashion. Participants observed object-directed hand actions in imitative and complementary action contexts, and performed identical (same grip-type) and non-identical (opposite grip-type) responses to a similar object, respectively. In imitative contexts, identical actions were performed optimally. In complementary contexts, however, non-identical actions were enhanced relative to identical actions. A further experiment using arrow cues instead of hand actions confirmed that these results were specific to action observation. Our findings demonstrate that action context plays a critical role in determining the relationship between action observation and execution. Crucially, this relationship is not fixed, but depends on an agent's goal when observing others act.