How to get the same size for all the characters in an equation?

I can think of two solutions:

• Replace frac with dfrac (see the third row in the screenshot below); or

• Employ inline-fraction notation instead of frac (see the fourth row).

For the use case at hand, I'd prefer the inline-fraction solution since it avoids creating a needlessly tall matrix expression. That, or just keep using frac: I doubt that your readers will experience any problems at all deciphering what the fraction terms denote.

I would also employ pmatrix environments instead of left(begin{array}{ccc} ... end{array}right), replace some of the ^{-1} terms with ' (transpose), not employ the cdot crutches, create a shorthand macro to abbreviate the many instances of underline{b} and, most of all, not encase the contents of each and every cell in curly braces.

documentclass[a4paper,10pt]{book}
usepackage{amssymb}
usepackage{mathtools} % for 'split' and 'pmatrix' environments
newcommandub{underline{b}}

begin{document}

begin{equation}label{eq:one}
begin{split}
mathbf{X}^{-1} 
&= left(
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}
%cdot 
begin{pmatrix} 
    lambda_{1} & cdots & 0  
    vdots & ddots & vdots  
    0 & cdots & lambda_{N} 
end{pmatrix}
%cdot 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}' 
right)^{!!-1}   % end of row 1
&= 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}
%cdot 
begin{pmatrix} 
    lambda_{1} & cdots & 0  
    vdots & ddots & vdots  
    0 & cdots & lambda_{N} 
end{pmatrix}^{!!-1}
%cdot 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}' % end of row 2
&= 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}
%cdot 
begin{pmatrix} 
    dfrac{1}{lambda_{1}} & cdots & 0  
    vdots & ddots & vdots  0 & 
    cdots & dfrac{1^{mathstrut}}{lambda_{N}} 
end{pmatrix}^{!!-1}
%cdot 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}' % end of row 3
&= 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}
%cdot 
begin{pmatrix} 
    1/lambda_{1} & cdots & 0  
    vdots & ddots & vdots  0 & 
    cdots & 1/lambda_{N} 
end{pmatrix}^{!!-1}
%cdot 
begin{pmatrix} 
    ub_{1} & cdots & ub_{N} 
end{pmatrix}'
end{split}
end{equation}

end{document}