We hypothesized that novel stimuli represent salient learning signals that can motivate 'exploration' in search for potential rewards. In computational theories of reinforcement learning, this is referred to as the novelty 'exploration bonus' for rewards. If true, stimulus novelty should enhance the reward anticipation signals in brain areas that are part of dopaminergic circuitry and thereby reduce responses to reward outcomes. We investigated this hypothesis in two fMRI experiments. Images of complex natural scenes predicted monetary reward or a neutral outcome by virtue of depicting either indoor or outdoor scenes. Half of the reward-predicting and neutral images had been familiarized the day before, the other half were novel. In experiment 1, subjects indicated whether images were novel or familiar, whereas in experiment 2, they explicitly decided whether or not images predicted reward by depicting indoor or outdoor scenes. Novelty led to the hypothesized enhancement of mesolimbic reward prediction responses and concomitant reduction of mesolimbic responses to reward outcomes. However, this effect was strongly task-dependent and occurred only in experiment 2, when the reward-predicting property of each image was attended. Recognition memory for the novel and familiar stimuli (after 24h) was enhanced by reward anticipation in both tasks. These findings are compatible with the proposition that novelty can act as a bonus for rewards under conditions when rewards are explicitly attended, thus biasing the organism towards reward anticipation and providing a motivational signal for exploration.