How to change the vertical position of a horizontal line in a matrix?

Another possibility, withcellspace:

documentclass[11pt, letterpaper, twoside]{book}
usepackage[T1]{fontenc}
usepackage{lmodern}
usepackage[total={6in,10in},left=1.5in,top=0.5in,includehead,includefoot]{geometry}
usepackage{amsfonts}
usepackage{mathtools}

usepackage[math]{cellspace}
setlength{cellspacetoplimit}{1.6ex}
setlength{cellspacebottomlimit}{1.6ex}

begin{document}

begin{equation}
    R = left[ begin{array}{ccc|>{$}Sc<{$}cc}
    multicolumn{6}{c} {}
    noalign{vspace{-3.2ex}}
    A & B & C & A & B &C 
    A & B & C & A & B &C 
    A & B & C & A & B &C 
    hline
    A & B & C & A & B &C 
    A & B & C & A & B &C 
    A & B & C & A & B &C [-0.6ex]
end{array} right]. 
end{equation}

end{document} 

I think that this code it is possible to create, quickly, with nicematrix package. Here I add my humble version using array enviroment.

documentclass[a4paper,12pt]{article}
usepackage{amsmath}

begin{document}
$left(begin{array}{@{}c|c@{}}
  begin{matrix}
  A & B & C
  A & B & C
  A & B & C
  end{matrix}
  & begin{matrix}
  A & B & C
  A & B & C
  A & B & C
  end{matrix} 
hline
  begin{matrix}
  A & B & C
  A & B & C
  A & B & C
  end{matrix}&
  begin{matrix}
  A & B & C
  A & B & C
  A & B & C
  end{matrix}
end{array}right)$
end{document}

Or this rudimental solution with the vertical space for the matrices [.3cm].

documentclass[a4paper,12pt]{article}
usepackage{amsmath}

begin{document}
$R=left(begin{array}{@{}c|c@{}}
  begin{matrix}
  A & B & C[.3cm]
  A & B & C[.3cm]
  A & B & C
  end{matrix}
  & begin{matrix}
  A & B & C[.3cm]
  A & B & C[.3cm]
  A & B & C
  end{matrix}[.1cm] [-.1cm]
hline
  begin{matrix}
  A & B & C[.3cm]
  A & B & C[.3cm]
  A & B & C
  end{matrix}&
  begin{matrix}
  A & B & C[.3cm]
  A & B & C[.3cm]
  A & B & C
  end{matrix}
end{array}right).$
end{document}

In case you'd like a TikZ alternative, using its matrix library:

documentclass{article}
usepackage{tikz}
usetikzlibrary{matrix}

begin{document}
begin{equation}
R = 
  begin{tikzpicture}[baseline]
    matrix (matrix) 
      [
        matrix of math nodes,
        row sep = 2ex,
        column sep = columnsep,
        left delimiter = lbrack,
        right delimiter = rbrack
      ]
      {
        A & B & C & A & B &C 
        A & B & C & A & B &C 
        A & B & C & A & B &C 
        A & B & C & A & B &C 
        A & B & C & A & B &C 
        A & B & C & A & B &C 
      };
    draw (matrix.east)  -- (matrix.west)
          (matrix.north) -- (matrix.south);
    end{tikzpicture}
.
end{equation}
end{document}

If you need to align = with the middle line in your matrix, you could yshift the baseline of your tikzpicture:

begin{tikzpicture}[baseline={([yshift=-3pt]matrix)}]

The environment {bNiceMatrix} of nicematrix is similar to the classical environment {bmatrix} of amsmath but creates PGF/Tikz nodes under the array accessible by the user.

Here I use these nodes with Tikz to draw the vertical rule (not broken) and the horizontal rule (with a yshift).

documentclass{article}
usepackage{nicematrix}
usepackage{tikz}

begin{document}

begin{equation}
R = begin{bNiceMatrix}[margin]
A & B & C & A & B &C [2ex]
A & B & C & A & B &C [2ex]
A & B & C & A & B &C [2ex]
A & B & C & A & B &C [2ex]
A & B & C & A & B &C [2ex]
A & B & C & A & B &C
CodeAfter
tikz draw (1-|4) -- (7-|4) 
            ([yshift=4pt]4-|1) -- ([yshift=4pt]4-|7) ;
end{bNiceMatrix}
end{equation}

end{document}

Remark: You need two compilations.

However, in this case, it seems that the use of the keys cell-space-limits of nicematrix does the job (no need to use Tikz).

documentclass{article}
usepackage{nicematrix}

begin{document}

begin{equation}
R = begin{bNiceArray}{ccc|ccc}[margin,cell-space-limits=6pt]
A & B & C & A & B &C 
A & B & C & A & B &C 
A & B & C & A & B &C 
hline
A & B & C & A & B &C 
A & B & C & A & B &C 
A & B & C & A & B &C
end{bNiceArray}
end{equation}

end{document}