Summary
Multiplexed fluorescence in situ hybridization techniques have enabled cell class or type identification by mRNA quantification in situ. However, inaccurate cell segmentation can result in incomplete cell-type and tissue characterization. Here, we present a robust segmentation-free computational framework, applicable to a variety of in situ transcriptomics platforms, called Spot-based Spatial cell-type Analysis by Multidimensional mRNA density estimation (SSAM). SSAM assumes that spatial distribution of mRNAs relates to organization of higher complexity structures (e.g. cells or tissue layers) and performs de novo cell-type and tissue domain identification. Optionally, SSAM can also integrate prior knowledge of cell types. We apply SSAM to three mouse brain tissue images: the somatosensory cortex imaged by osmFISH, the hypothalamic preoptic region by MERFISH, and the visual cortex by multiplexed smFISH. SSAM outperforms segmentation-based results, demonstrating that segmentation of cells is not required for inferring cell-type signatures, cell-type organization or tissue domains.