how can I use math environment for a non English language?

text from amsmath allows text at the right size (you need something similar even for Latin scripts)

documentclass{article}

usepackage{amsmath}
usepackage{fontspec}
newfontfamilydevanagarifont[Script=Devanagari]{Lohit-Devanagari.ttf}


begin{document}

devanagarifont
$frac{text{/जी.श}}{text{५}}$


[frac{text{/जी.श}}{text{५}}]

end{document}


begin{document}

devanagarifont
$frac{text{/जी.श}}{text{५}}$


[frac{text{/जी.श}}{text{५}}]

end{document}

If you want to use text in math, you need to specify it as text. On the other hand, you can define the Sanskrit digits to work in text and math without further hassle.

documentclass{article}

usepackage{amsmath}
usepackage{fontspec}
usepackage{polyglossia}

setmainlanguage{sanskrit}
setmainfont{ITF Devanagari}[NFSSFamily=devn]
newfontfamily{latinfont}{Latin Modern Roman}

DeclareSymbolFont{sanskritdigits}{encodingdefault}{devn}{m}{n}
Umathcode`०="7 symsanskritdigits "0966
Umathcode`१="7 symsanskritdigits "0967
Umathcode`२="7 symsanskritdigits "0968
Umathcode`३="7 symsanskritdigits "0969
Umathcode`४="7 symsanskritdigits "096A
Umathcode`५="7 symsanskritdigits "096B
Umathcode`६="7 symsanskritdigits "096C
Umathcode`७="7 symsanskritdigits "096D
Umathcode`८="7 symsanskritdigits "096E
Umathcode`९="7 symsanskritdigits "096F


begin{document}

$dfrac{text{जी.श}}{५}$ $dfrac{३४५}{८९}$

{latinfont Text:} ०१२३४५६७८९

{latinfont Math:} $०१२३४५६७८९$

end{document}