Neurons recruited for local computations exhibit rhythmic activity at gamma frequencies. The amplitude and frequency of these oscillations are continuously modulated depending on stimulus and behavioral state. This modulation is believed to crucially control information flow across cortical areas. Here we report that in the rat hippocampus gamma oscillation amplitude and frequency vary rapidly, from one cycle to the next. Strikingly, the amplitude of one oscillation predicts the interval to the next. Using in vivo and in vitro whole-cell recordings, we identify the underlying mechanism. We show that cycle-by-cycle fluctuations in amplitude reflect changes in synaptic excitation spanning over an order of magnitude. Despite these rapid variations, synaptic excitation is immediately and proportionally counterbalanced by inhibition. These rapid adjustments in inhibition instantaneously modulate oscillation frequency. So, by rapidly balancing excitation with inhibition, the hippocampal network is able to swiftly modulate gamma oscillations over a wide band of frequencies.