What is band inversion and how to recognize it in band structure?

Topological materials form a broad family including insulators, semimetals, and superconductors, of which perhaps the best known are topological insulators. For concreteness, I will focus on topological insulators as these are the ones specifically mentioned in the question.

Topological insulator. A topological insulator is an insulator whose Hamiltonian cannot be adiabatically connected to the atomic limit. Adiabatic means that, when tuning some external parameter (e.g. pressure, temperature) to change the Hamiltonian, the process is slow enough that the material stays in its ground state throughout.

Qualitative picture. Consider an insulator like diamond. Imagine you pull the carbon atoms apart, taking each individual atom to one edge of the Universe, so that you end up with isolated carbon atoms -- this is what we call the atomic limit. In diamond, it is possible to do this process without closing the bulk band gap, so we say that bulk diamond is adiabatically connected with the atomic limit, and as such diamond is a normal insulator. Now repeat the same with another material, say Bi$_2$Se$_3$, which in its crystalline form is also a bulk insulator. In this case, when you pull the system apart to reach the atomic limit it is impossible to do it without closing the bulk band gap. As such, Bi$_2$Se$_3$ is not adiabatically connected to the atomic limit, and we call it a topological insulator.

So what is the reason for this picture of topological insulators? Broadly speaking, the electron wave function "twists" as you cross the Brillouin zone in a topological material. The electron wave function in the atomic limit is never twisted, so the twist needs to be undone when getting to that limit and this is what the band closure accomplishes.

Topological invariants. More rigorously, these twists can be characterized by so-called topological invariants, and their mathematical form depends on the type of topological material you are looking at, but are mostly related to Berry phase-like quantities that measure the evolution of the electron wave function as you cross the Brillouin zone. As an example, for 3-dimensional topological insulators, the topological invariant is a set of 4 numbers that can take one of two values ($mathbb{Z}_2$ classification) and can be calculated by following the evolution of Wannier charge centers across the Brillouin zone as described in this paper, or if the system has inversion symmetry by simply calculating the parity of the eigenstates at special points in the Brilloin zone as described in this paper. Another well-known example is for Chern insulators (which roughly speaking are 2D materials with topological and magnetic orders) in which the topological invariant is the Chern number (an integer, so $mathbb{Z}$ classification) obtained by integrating the Berry curvature over the Brillouin zone.

First principles calculations. So how do you figure out what the topological order of a material is? The way to do it is to calculate the corresponding topological invariant. As these calculations involve Berry phase-like quantities, the usual route in first principles calculations is to use Wannier functions, and packages that implement these calculations and have interfaces to major DFT packages include Z2Pack and WannierTools. There are also several databases (e.g. the Topologial Materials Database or Materiae) that include the topological classification of many materials based on semilocal DFT. These databases are excellent starting points to figure out the possible topological order of a material, although it is known that higher levels of theory (e.g. hybrid functionals, $GW$) may lead to different results, so I recommend further analysis.

Band inversion. Having said all this, where does "band inversion" enter? The usual strategy to get a topological insulator, and already used in the seminal paper by Kane and Mele, is to induce the wave function twist using spin-orbit coupling. This is a very simple schematic for Bi$_2$Se$_3$:

Without spin-orbit coupling (left), the "conduction band" shown in red is made of Bi $p_z$ orbitals, and the "valence band", shown in blue, of Se $p_z$ orbitals. However, the bands overlap and the system has no band gap. When spin-orbit coupling is included (right), a gap opens at the crossing points, and now we have a proper conduction band that has contributions from the band that made the valence band originally (blue) and vice versa (imagine you continue the valence band in blue with a dashed line, it would then "join up" with the part of the conduction band in blue). This is called a band inversion. In the case of Bi$_2$Se$_3$ it is very clear to identify, but in other materials it may be harder. A band inversion like this may suggest that the material has topological order, but the only way to confirm it is by calculating the topological invariant.

So finally let's consider the paper you cite. As far as I understand, in this paper they call "band inversion" a situation in which the bands overlap (left diagram). They argue that such a band overlap is not necessarily induced by spin-orbit coupling, but instead by other effects such as perhaps a structural distortion. I think this is a reasonable statement, and they provide evidence in a range of materials for this. But what is essential to understand is that this is not enough to have a topological insulator, it is still necessary to open the band gap to get to a situation like the one depicted in the right diagram, and for this you usually still need spin-orbit coupling (and this is indeed the case in the paper you cite).

Additional comments. (i) There are some additional subtleties with the above definition of adiabatic connection. We typically require that the Hamiltonian also obeys some symmetry throughout the entire adiabatic evolution, and depending on the symmetry that is obeyed we end up with different types of topological material. For example, when we say "topologial insulator" we typically understand "time-reversal invariant topological insulator", which means that time reversal symmetry is conserved throughout. Another example may be to impose a crystalline symmetry, and then we could find a topological crystalline insulator. (ii) Another very famous feature of topological insulators is the presence of surface states. From the qualitative description above it is very easy to understand where they come from: the surface of a topological insulator is really the boundary between a topologically ordered bulk and a "normal" vacuum or air outside. As such, the wave function twist needs to be undone at this boundary, so the "gap" closes and we get metallic states, which are observed as surface states in this setup. These states have some nice features because they are protected by the underlying topology, and the presence of surface states (similar to the presence of band inversion) is indicative of topological order. You still want to calculate the topological invariant though to confirm this.

P.S.: the basic features of topological materials can be understood with simple 2-band models. I am preparing a series of videos on this, and will share the link when I have them ready.