The early processing of sensory information by neuronal circuits often includes a reshaping of activity patterns that may facilitate further processing in the brain. For instance, in the olfactory system the activity patterns that related odors evoke at the input of the olfactory bulb can be highly similar. Nevertheless, the corresponding activity patterns of the mitral cells, which represent the output of the olfactory bulb, can differ significantly from each other due to strong inhibition by granule cells and peri-glomerular cells. Motivated by these results we study simple adaptive inhibitory networks that aim to separate or even orthogonalize activity patterns representing similar stimuli. Since the animal experiences the different stimuli at different times it is difficult for the network to learn the connectivity based on their similarity; biologically it is more plausible that learning is driven by simultaneous correlations between the input channels. We investigate the connection between pattern orthogonalization and channel decorrelation and demonstrate that networks can achieve effective pattern orthogonalization through channel decorrelation if they simultaneously equalize their output levels. In feedforward networks biophysically plausible learning mechanisms fail, however, for even moderately similar input patterns. Recurrent networks do not have that limitation; they can orthogonalize the representations of highly similar input patterns. Even when they are optimized for linear neuronal dynamics they perform very well when the dynamics are nonlinear. These results provide insights into fundamental features of simplified inhibitory networks that may be relevant for pattern orthogonalization by neuronal circuits in general.