Use only the mathbb font from TeX Gyre Pagella Math

I am working with LuaLaTeX on Debian stable, with texlive-full in the version 2018.20190227-2. My main goal is to use the Palatino derivates. For pure text, this can easily be achieved by

setmainfont{TeX Gyre Pagella}

However, I prefer a combination of mathpazo and mathabx instead of TeX Gyre Pagella Math (since some spacings look really weird in Pagella Math), whilst I rather prefer the mathbb font from Pagella Math. So my ideal combination would be the following:

• Use mathpazo and mathabx,
• Use the mathbb font from TeX Gyre Pagella Math.

Is there any possibility to achieve this? Unfortunately, mathpazo somehow ‘protects’ its mathbb font, and unicode-math overwrites too many other this, so the naïve approach

documentclass{scrartcl}
usepackage{fontspec}

usepackage{mathpazo,mathabx}

usepackage{unicode-math}
setmainfont{TeX Gyre Pagella}
setmathfont{TeX Gyre Pagella Math}

begin{document}
   Brackets are too low in Pagella Math: $(X+Y)^2=X^2+2XY+Y^2$.

   This also looks weird in Pagella Math: Let $fcolon Xto Y$ be a map.

   This is supposed to be Pagella Math: $mathbb{R}$
end{document}

will give the complete opposite: mathbb from Pazo and everything else from Pagella Math.

setmathfont{TeX Gyre Pagella Math}[range=bb]