Altered nuclear shape is a defining feature of cancer cells. The mechanisms underlying nuclear dysmorphia in cancer remain poorly understood. Here we identify PPP1R12A and PPP1CB, two subunits of the myosin phosphatase complex that antagonizes actomyosin contractility, as proteins safeguarding nuclear integrity. Loss of PPP1R12A or PPP1CB causes nuclear fragmentation, nuclear envelope rupture, nuclear compartment breakdown and genome instability. Pharmacological or genetic inhibition of actomyosin contractility restores nuclear architecture and genome integrity in cells lacking PPP1R12A or PPP1CB. We detect actin filaments at nuclear envelope rupture sites and define the Rho-ROCK pathway as the driver of nuclear damage. Lamin A protects nuclei from the impact of actomyosin activity. Blocking contractility increases nuclear circularity in cultured cancer cells and suppresses deformations of xenograft nuclei in vivo. We conclude that actomyosin contractility is a major determinant of nuclear shape and that unrestrained contractility causes nuclear dysmorphia, nuclear envelope rupture and genome instability.