Polarized light imaging (PLI) enables the evaluation of fiber orientations in histological sections of human postmortem brains, with ultra-high spatial resolution. PLI is based on the birefringent properties of the myelin sheath of nerve fibers. As a result, the polarization state of light propagating through a rotating polarimeter is changed in such a way that the detected signal at each measurement unit of a charged-coupled device (CCD) camera describes a sinusoidal signal. Vectors of the fiber orientation defined by inclination and direction angles can then directly be derived from the optical signals employing PLI analysis. However, noise, light scatter and filter inhomogeneities interfere with the original sinusoidal PLI signals. We here introduce a novel method using independent component analysis (ICA) to decompose the PLI images into statistically independent component maps. After decomposition, gray and white matter structures can clearly be distinguished from noise and other artifacts. The signal enhancement after artifact rejection is quantitatively evaluated in 134 histological whole brain sections. Thus, the primary sinusoidal signals from polarized light imaging can be effectively restored after noise and artifact rejection utilizing ICA. Our method therefore contributes to the analysis of nerve fiber orientation in the human brain within a micrometer scale.