How the cosmological scale factor $a(t)$ will have maximum value followed by a contracting phase of the universe?

You don't know $Omega(t)$. How $Omega$ varies with time depends on the specific nature of $Omega$, e.g. $Omega_r propto a^{-4}$, but $Omega_m propto a^{-3}$. A universe with different amounts of radiation and matter will have different behaviors for $Omega(t)$.

However you do know that after this maximum value there will be a contracting phase. That comes from the definition of a maximum point - all nearby points have lower values than the maximum. So both approaching the maximum and going away from it, $a(t)$ must decrease. If it starts increasing again, there must be a minimum, which does not exist (from the differential equation you solved). Hence it keeps decreasing, until the assumptions that went into the calculation break down.

From: $$ frac{dot a^{2}}{a^{2}} + frac{k}{a^{2}} = frac{8pi G}{3} rho(t) $$ If you set $k=1$ and $Omega = frac{rho}{rho_{c}}$: $$ dot a^{2} + 1 = frac{8pi G}{3} rho_{c}cdotOmega cdot a^{2}. $$ At $a= a_{max}$, the expansion rate is zero $dot{a}=0$, therefore $$ 1 = frac{8pi G}{3} rho_{c}cdotOmega cdot a^{2}_{max}. $$

Now the solution is dependent on how $Omega$ scales with $a$. Let's for example examine a matter only universe, i.e. $Omega = 1/a^3$, therefore: $$ 1 = frac{8pi G}{3} rho_{c}cdot frac{1}{a^3_{max}} cdot a^{2}_{max}. $$ Et voila: $$ a_{max}= frac{8pi G}{3} rho_{c}. $$

how will I prove that this maximum value will be followed by a contracting phase to the universe?

Well if you plug any value of $a > a_{max}$ into the Friedmann equation, you will get an negative value of $dot{a}^2$. Hence we know that $a$ can not exceed $a_{max}$. But you might wonder whether $a$ can stay at $a_{max}$ without contracting? Hint: for the above example you can prove (as a home work for you) $$ ddot{a}|_{a = a_{max}} < 0. $$