What are the downsides of using XeTeX?

Let me give a biased answer to the question in the title with some reference to the other questions in the body of you post.

1. XeTeX is currently unmaintained or has an inactive maintainer which is functionally equivalent. Work on XeTeX was originally started by Jonathan Kew at SIL with contributions from Khaled Hosny. Jonathan has since moved on from SIL and no longer contributes. Khaled's interests also seem to have shifted towards LuaTeX with a custom Lua interface for HarfBuzz to bring complex scripts to LuaTeX. For some more background and a better overview see A.Reutenauer, “The state of XeTeX”, TUGboat, Volume 40 (2019), No. 2.

2. The current lack of maintenance means that XeTeX has some unfixed bugs. Some of them affect mathematics typesetting to an extent where I personally think, that XeTeX is no longer suitable for math typesetting. The bugs are a bit subtle, so you have to judge for yourself. This is the one I consider the most severe: https://sourceforge.net/p/xetex/bugs/87/

3. Microtypography in XeTeX is limited to protrusion. For expansion, tracking, kerning, and spacing there is simply no backend code. But then again, microtypography is usually not super important. In fact, Hans Hagen, the lead developer of LuaTeX and ConTeXt once said: “I hardly use expansion myself”.

4. XeTeX has a very funky way of producing PDF. While LuaTeX and pdfTeX directly output PDF and TeX outputs DVI by default, XeTeX does neither. Since XeTeX is derived from TeX it doesn't include backend code for PDF output, but because OpenType fonts cannot be represented in DVI, it invented its own format “extended DVI” (XDV). When you run XeTeX you will still get PDF output, but that is because as part of the run XeTeX feeds the intermediate XDV into xdvipdfmx to produce PDF.

   The xdvipdfmx program has a few quirks of its own. It is actually capable of processing a few PostScript specials, which means that you can use PSTricks with XeTeX. On the other hand, this means that it needs its own driver code for color and image placement. These drivers are usually very complex and require a deep understand of the file formats. Because the user and developer base for XeTeX is much smaller than for pdfTeX some of those drivers are of very low quality, in particular the PGF/TikZ driver. Using PGF/TikZ with XeTeX is not recommended.

5. Academic journals that accept LaTeX usually only accept pdfTeX-compatible documents. I'd say this is the main reason why people do not use XeTeX (or LuaTeX for that matter).

It used to be the case that XeTeX was the only TeX engine with HarfBuzz font shaping which was a big advantage for anyone having to typeset complex scripts like Arabic or CJK (Chinese, Japanese, Korean). In the recent past LuaTeX has also gained support for HarfBuzz under the name LuaHBTeX, so this advantage is now gone. However, there is still some macro code written explicitly for XeTeX to support complex scripts. I don't know what the progress in porting this code to LuaTeX is, but I think XeTeX usage might decline in the future.