breakage vs breaking in "X denotes the spontaneous breakage/breaking of rotational symmetry"

Depending on wider context, you could use one of the following:

• X denotes the spontaneous breaking of rotational symmetry
• X denotes a spontaneous break in rotational symmetry

You can also use "discontinuity" to denote a region where global surface properties break suddenly (flipping basis directions, loss of smoothness etc.)