Finding the Center of Mass of a disk when a part of it is cut out.

I am assuming the mass per unit area is one for simplicity.

Using symmetry there is no need to actually compute an integral other than to determine masses.

In general, if you have two essentially disjoint sets $A,B$ then $int_{A cup B} f = int_A f + int_B f$.

If we let $bar{x}_A = {1 over int_A dm } int_A x dm$, then $bar{x}_{A cup B} = { int_A dm over int_{A cup B} dm } bar{x}_A + { int_B dm over int_{A cup B} dm } bar{x}_B$.

Furthermore, if we translate an object by $d$ we have $bar{x}_{A +{d}} = bar{x}_A + d$.

Let $A$ be the big disc with the little disk cut out and let $B$ be the little disk.

If we take the centre of the big disk at the origin, we have $bar{x}_{A cup B} = 0$, $bar{x}_B = ({R over 2},0)$, $int_{A cup B} dm = pi R^2$, $int_{B} dm = {1 over 4} pi R^2$ (and $int_{A} dm = {3 over 4} pi R^2$, of course).

Solving the above for $bar{x}_A$ gives $bar{x}_A = -({R over 6},0)$.

Here is another "physics oriented" way:

Because of symmetry the center of mass lies on the line connecting $O$ and $O'$. The mass at each point $-R leq r leq R$ on this line lies symmetric above and below which corresponds to a linear density

$$rho(r) = 2sqrt{R^2-r^2} text{ for } -Rleq rleq 0$$ and taking out the hole $$rho(r) = 2left(sqrt{R^2-r^2} - sqrt{left(frac R2right)^2-left(r-frac R2right)^2}right) text{ for } 0leq rleq R$$

Now, noting that the mass of the disk with hole is $frac 34 pi R^2$ we get

$$r_M = frac 4{3pi R^2}int_{-R}^Rrrho(r) dr = - frac 4{3pi R^2}int_0^Rrsqrt{left(frac R2right)^2-left(r-frac R2right)^2}dr $$ $$= - frac 4{3pi R^2}cdot frac{pi R^3}{8}=-frac R6$$

There is a simple solution, showing where the mass center of two bodies is located, knowing their position and their masses. (Well, on object has negative mass in our case.)

This answer addresses the question in the OP, it is closer to it (instead of the above physics approach) since it deals with the mentioned integral. We will "compute" it. (But not via polar transformation. But rather in a way fitting to the above intuition. However, it is a mathematical way to deal, since we use the linearity of the integral and proceed blindly.)

I will introduce some notations. Let $D$ be the (bigger) disc / the ball $B(0,R)$ centered in $0inBbb C$ and with radius $R$, $D'$ be the (smaller) disk $B(R/2,R)$ centered in $R/2inBbb C$ and with radius $R/2$, and $D''=D-D'$ the difference set.

(We have positioned the three centers in this way without loss of generality.)

We denote by $m,m',m''$ the total mass of the three sets. For symmetry reasons, the mass center of $D''$ is on the $Ox$ axis. Let $a=0$, $a'=R/2$, $a''=?$ be the three abscissa of the mass centers.

Then, in the spirit of the OP, where integrals are involved: $$ begin{aligned} a'' &= frac 1{m''}int_{D''}x; dx; dy\ &= frac 1{m''}int_{D}x; dx; dy - frac 1{m''}int_{D'}x; dx; dy\ &= frac 1{m''}cdot ma - frac 1{m''}cdot m'a' \ &=frac m{m''}a-frac {m'}{m''}a' \ &=frac {1}{3/4}cdot 0-frac {1/4}{3/4}cdot(R/2) \ &=frac 16R . end{aligned} $$

Place a Cartesian Coord. system, for example, at the center of the big disk. The mass of an homogeneous disk is $M=rho : A t$ where $rho$ is the density, $A$ is the area of the circle and $t$ is the thickness.

You want $M: z= sum_i m_i z_i$ for $i=1,2$, where $m_1$ is the mass of the large disk and $m_2$ is the mass of the small disk, which is negative as you are removing it and $M=m_1-m_2$ is the total mass of the resulting system. The values of $z_i$ are the position in the $x_i$ and $y_i$ position of the small masses and $z$ is the position you are looking for.

Focus only in $x$ direction as $y$ is trivial. Compute the masses of the disks $m_1$ and $m_2$ (example: $m_i= pi R*R t : rho$, you can use your integrals for this, but splitting the two regions makes it intuitive) and notice that the position of their centroids are $(x_1,y_1)=(0,0)$ and $(x_2,y_2)=(R/2,0)$. The answer is easy to find after algebraic operation.

Given the symmetry you noted, it seems not worth it to do the integral in polar coordinates when you know the center of mass will only have an $x$. Let's rotate your picture upside down. Notice that computing the integral will give us

$$iintlimits_{text{shaded region}}^{} x:dA = iintlimits_{text{small circle on right side}}^{} x:dA$$

by symmetry because $x$ is an odd function. Next, you want integrate w.r.t. $x$ first because the square roots will cancel

$$bar{x} =frac{4}{3pi R^2}int_{-frac{R}{2}}^frac{R}{2} int_{frac{R}{2}-sqrt{frac{R^2}{4}-y^2}}^{frac{R}{2}+sqrt{frac{R^2}{4}-y^2}} x:dx :dy = frac{4}{3pi R}int_{-frac{R}{2}}^frac{R}{2} sqrt{frac{R^2}{4}-y^2}:dy = frac{R}{6}$$

The center of mass can probably be computed by using difficult integrals, but it can also be computed in a very simple way.

Let's call $vec{OG}$ the center of mass, $m_{big}$ and $m_{small}$ the mass of the big and small disk respectively. By using the center of mass definition, we have,

$$ vec{OG} = frac{1}{m_{big} + m_{small}}(m_{big} vec{OO} + m_{small} vec{OO'}) $$

Since the small disk is cut out and not added we should not use a positive mass, but a negative mass for $m_{small}$, therefore, since the surface of the hole is 4 times smaller than the total surface,

begin{align} m_{small} &= -frac{1}{4} m_{big} end{align}

By substituting this value into the first equation, we obtain,

$$vec{OG} = - frac{1}{3}vec{OO'}$$