Previous studies suggest the involvement in timing functions of a surprisingly extensive network of human brain regions. But it is likely that while some of these regions play a fundamental role in timing, others are activated by associated task demands such as memory and decision-making. In two experiments, time perception (duration discrimination) was studied under two conditions of task difficulty and neural activation was compared using fMRI. Brain activation during duration discrimination was contrasted with activation evoked in a control condition (colour discrimination) that used identical stimuli. In the first experiment, the control task was slightly easier than the time task. Multiple brain areas were activated, in line with previous studies. These included the prefrontal cortex, cerebellum, inferior parietal lobule and striatum. In the second experiment, the control task was made more difficult than the time task. Much of the differential time-related activity seen in the first experiment disappeared and in some regions (inferior parietal cortex, pre-SMA and parts of prefrontal cortex) it reversed in polarity. This suggests that such activity is not specifically concerned with timing functions, but reflects the relative cognitive demands of the two tasks. However, three areas of time-related activation survived the task-difficulty manipulation: (i) a small region at the confluence of the inferior frontal gyrus and the anterior insula, bilaterally, (ii) a small portion of the left supramarginal gyrus and (iii) the putamen. We argue that the extent of the timing "network" has been significantly over-estimated in the past and that only these three relatively small regions can safely be regarded as being directly concerned with duration judgements.