What do the off-diagonal elements of a density matrix physically represent?

A good way to think about density matrices is to think about them as Bloch Vectors (I assume you are familiar with the Bloch Sphere). This won't tackle your question head on; but I hope will give some motivation for what each of the numbers in the density matrix means.

You see, every $2text{x}2$ matrix can be decomposed into a linear superposition of $mathbb{1}, sigma_z, sigma_x. sigma_y$; they form a basis over the space of $2text{x}2$ matrices. Therefore we can rewrite every density matrix $rho$ as such:

begin{aligned}rho &={frac {1}{2}}left(I+{vec {a}}cdot {vec {sigma }}right)&={frac {1}{2}}{begin{pmatrix}1&0&1end{pmatrix}}+{frac {a_{x}}{2}}{begin{pmatrix}0&11&0end{pmatrix}}+{frac {a_{y}}{2}}{begin{pmatrix}0&-ii&0end{pmatrix}}+{frac {a_{z}}{2}}{begin{pmatrix}1&0&-1end{pmatrix}}&={frac {1}{2}}{begin{pmatrix}1+a_{z}&a_{x}-ia_{y}a_{x}+ia_{y}&1-a_{z}end{pmatrix}}end{aligned}

Where $(a_x,a_y,a_z)$ are real numbers representing the vector of our state $rho$ on the Bloch sphere. Of course there are many ways to interpret this information; the relative probability of measuring the state in any basis comes to mind. But something that is really interesting is that the shorter the Bloch vector, the more mixed our state; for a Bloch vector of norm $1$ we have a pure state.

This is a bit more general than what your question was asking about; but clearly you can see that the diagonal elements of the matrix play a role in our analysis here.

For multi Qubit systems; there are various ways of approaching the problem; one which Linear Algebra afficonatos will enjoy is using higher dimensional Bloch Bodies, and doing the same style of matrix decomposition, only now you have a larger basis, and a different geometric structure to interpret your decomposition by (not a sphere). This style of thinking about the problem emphasizes very well how mixedness changes with higher dimensional quantum systems; and in fact your maximally mixed state in a system of $n$ Qubits has Purity $frac{1}{2^n}$, this result is geometrically obvious using Bloch Bodies

A more common way to think about multi Qubit Systems is through partial traces of the overall density matrix; this has been answered many times on this SE, I will link some posts.

It can represent 'various' things depending upon the physical system and the context. For instance:

(1) For a 'closed and isolated' system (let us say a qubit), it represents the 'coherence' between the energy levels by the virtue of the relative phase of the amplitudes. For instance starting with a pure state (in this case): $|psirangle=cos theta|0rangle+ text{e}^{iphi}sin theta |1rangle$, we would have $rho_{01}=text{e}^{-iphi}cos theta sin theta $ which contains the 'relative phase' information. Here, $rho=|psirangle langlepsi|.$

(2) For a general mixed state, which can be seen as 'partially traced' version of a pure state (over a larger composite Hilbert space) over the 'environmental degrees of freedom'. In this case, the off-diagonal entries of the reduced density matrix would contain the information about the decoherence as well. For instance, $rho_{01}(t) sim text{e}^{-gamma t}$, along with the system information as in above example. Here, the decay constant $gamma$ can be seen as a coupling to the environment via the operator $sigma^-$ (in this example) which causes the decay of an excited level into the ground state by emitting a quanta to the environment. The information here was of 'decoherence'. Notice that the system evolution is taken into account in this case, unlike the case (1) of stationary qubit with no evolution.

In general, these two viewpoints are enough for the context of a qubit with $2 times2$ density matrix. For a larger Hilbert space more information can be deduced from off-diagonal terms related to various energy levels and interactions between the levels.

To put it very shortly, non-zero off-diagonal elements of the density matrix signify that your system features a quantum superposition between the elements of the basis that you chose to represent $rho$. In other words, your state is not only a statistical mixture of your basis states (which can be understood as representing your ignorance about the system), but an actual quantum superposition in which those basis elements are "coexisting". Of course there can be a combination between those two situations (different degrees of coherence/mixture), and all this is basis dependent (something that looks like a quantum superposition in one basis, can be described in terms of a single basis element in a different basis).