Abstract We describe a comprehensive linear approach to the problem of imaging brain activity with high temporal as well as spatial resolution based on combining EEG and MEG data with anatomical constraints derived from MRI images. The "inverse problem" of estimating the distribution of dipole strengths over the cortical surface is highly underdetermined, even given closely spaced EEG and MEG recordings. We have obtained much better solutions to this problem by explicitly incorporating both local cortical orientation as well as spatial covariance of sources and sensors into our formulation. An explicit polygonal model of the cortical manifold is first constructed as follows: (1) slice data in three orthogonal planes of section (needle-shaped voxels) are combined with a linear deblurring technique to make a single high-resolution 3-D image (cubic voxels), (2) the image is recursively flood-filled to determine the topology of the gray-white matter border, and (3) the resulting continuous surface is refined by relaxing it against the original 3-D gray-scale image using a deformable template method, which is also used to computationally flatten the cortex for easier viewing. The explicit solution to an error minimization formulation of an optimal inverse linear operator (for a particular cortical manifold, sensor placement, noise and prior source covariance) gives rise to a compact expression that is practically computable for hundreds of sensors and thousands of sources. The inverse solution can then be weighted for a particular (averaged) event using the sensor covariance for that event. Model studies suggest that we may be able to localize multiple cortical sources with spatial resolution as good as PET with this technique, while retaining a much finer grained picture of activity over time.