In instrumental conditioning, there is a rather precise definition of goal-directed control, and therefore an acute boundary between it and the somewhat more amorphous category comprising its opposites. Here, we review this division in terms of the various distinctions that accompany it in the fields of reinforcement learning and cognitive architectures, considering issues such as declarative and procedural control, the effect of prior distributions over environments, the neural substrates involved, and the differing views about the relative rationality of the various forms of control. Our overall aim is to reconnect some presently far-flung relations.