Transforming a specific package in text-mode into math-mode

To fix the infinite loop, this should work:

documentclass{article}
usepackage{arcs}

usepackage{xpatch}
makeatletter
xpatchcmdover@under@arc
  {let rs@size@warning = @gobbletwo}
  {let rs@size@warning = @gobblethree}
  {}{fail}
makeatother

begin{document}
overarc{A}overarc{a} par
underarc{A}underarc{a}
end{document}

But I'm not capable of providing a math-mode variant. Sorry.

Update:

A poor attempt: $overarc{$A$}, underarc{$a$}$.

Or, you can use overparen and underparen provided by unicode-math package. This package requires unicode-aware engine like XeTeX or LuaTeX. Those two commands are documented in unimath-symbols.pdf, sec. 5 & 6.

documentclass{article}

% this uses otf math font "Latin Modern Math" by default
usepackage{unicode-math}

begin{document}
[
  overparen{a}  overparen{A}  overparen{AB}
  underparen{a} underparen{A} underparen{AB}
]
end{document}

There is a wonderful TeX-SX answer providing updated info about available otf math fonts.

Rather than use the arcs package, I borrowed from my answer at Big tilde in math mode, to adapt for the present purposes. See ADDENDUM for arcs package approach.

Note that I did some vertical squeezing (not in the macro definitions) in the MWE presentation to not have it run off my screen capture.

documentclass{article}
usepackage{scalerel}
usepackage{stackengine,wasysym}

newcommandreallywideoverarc[1]{ThisStyle{%
  setbox0=hbox{$SavedStyle#1$}%
  stackengine{-.5LMpt}{$SavedStyle#1$}{%
    stretchto{scaleto{SavedStylemkern.2mufrown}{.4wd0}}{.6ht0}%
  }{O}{c}{F}{T}{S}%
}}
newcommandreallywideunderarc[1]{ThisStyle{%
  setbox0=hbox{$SavedStyle#1$}%
  stackengine{1.5LMpt}{$SavedStyle#1$}{%
    stretchto{scaleto{SavedStylemkern.2musmile}{.4wd0}}{.6ht0}%
  }{U}{c}{F}{T}{S}%
}}

deftest#1{$%
  reallywideoverarc{#1},
  scriptstylereallywideoverarc{#1},
  scriptscriptstylereallywideoverarc{#1}
$parvspace{-4pt}$%
  reallywideunderarc{#1},
  scriptstylereallywideunderarc{#1},
  scriptscriptstylereallywideunderarc{#1}
$parvspace{-2pt}}

parskip 1ex
begin{document}

test{abcdefghijklm}
test{abcdefghijk}
test{abcdefghi}
test{abcdefg}
test{abcde}
test{abc}
test{ab}
end{document}

p.s. I later found I had done at least a partial answer to this sort of question here: Strange result with patch of overarc

---

ADDENDUM

Based on a comment, the OP asked me to elaborate as part of my answer. In trying to actually use the arcs package, I discovered, as did another answer to this question, that the package was not well behaved. The fix to that problem given in the other answer was to change the @gobbletwo to @gobblethree in the definition of over@under@arc. However, that still leaves the result in text mode. To achieve math mode as a default, two additional changes are needed:

renewcommand{overarc}[2][1]{over@under@arc{#1}{$#2$}z@}
renewcommand{underarc}[2][1]{over@under@arc{#1}{$#2$}@ne}

With those three changes the result will be set in math mode. However, the overarc` does not have an italic correction to the arc itself.

documentclass[a4paper,12pt]{article}
usepackage[T1]{fontenc}
usepackage[utf8]{inputenc} 
usepackage{amssymb, amsmath}
usepackage{arcs}
usepackage{xpatch}
makeatletter
xpatchcmdover@under@arc
  {let rs@size@warning = @gobbletwo}
  {let rs@size@warning = @gobblethree}
  {}{fail}
renewcommand{overarc}[2][1]{over@under@arc{#1}{$#2$}z@}
renewcommand{underarc}[2][1]{over@under@arc{#1}{$#2$}@ne}
makeatother
begin{document}
overarc{A}
overarc{AB}
underarc{A}
underarc{AB}
end{document}