Derivation of Slater-Koster equations

The answer by Anyon does not address how to calculate the form of the Slater-Koster two-center integrals (2CI) but rather how to determine the direction cosines that appear in the 2CI's (the $l$,$m$,$n$'s in things like $sqrt{3}lm*(sdsigma)$, but not the $sqrt{3}$). I am currently trying to explicitly perform the same calculation to include in my comps so I can sympathize with your frustration in the lack of clarity in the literature...

Any who, the most explicit reference I have found that explains how to actually calculate the SK-matrix elements is in this paper (Takegahara et al 1980 J. Phys. C: Solid State Phys. 13 583, Slater-Koster tables for f electrons) where Takegahara extends the original SK table to f electrons.

The fundamental strategy he invokes, which is very tedious, is to calculate the form of hopping elements along the bond direction and then rotate it into a generic orientation to make use of the l,m,n- direction cosines we are familiar with.

Sorry that this is less of an answer and more of a comment, I don't have enough reputation to post any other way.

Spherical harmonics are not themselves full atomic orbitals. Consider the Hydrogen wave function, which separates into a radial part and an angular part. The latter is a spherical harmonic, but the former is some other function (in the case of Hydrogen it's a Laguerre polynomial). In general, we can approximate the angular part for other atoms with the same spherical harmonics, but we usually don't know the radial part analytically. Hence, if you want to directly evaluate Eq. (1) by integration you also need to find the radial part somewhere. The strength of Slater's and Koster's work, however, is that we can avoid this problem entirely, by hiding all radial dependence in the Slater-Koster parameters (in your example they are $V_{pdsigma}$ and $V_{pdpi}$).

For simplicity, I'll focus on the case of an $s$-$p$ overlap. This simplifies the geometry, and will let me borrow pictures and notation from this thesis. Let's say we have an $s$ orbital at site $i$ with wave function $psi_{is}$, and a $p_alpha$ orbital at site $j$ with wave function $psi_{jp_alpha}$, where $alpha in {x,y,z}$. In Dirac's bra-ket notation, the overlap between them can be written $$ E_{s,p_alpha}=langle psi_{is}|H_{2c}|psi_{jp_alpha}rangle = langle S|H_{2c}|P_alpharangle, tag{3} $$ where $H_{2c}$ is the two-center Hamiltonian, and we've introduced short-hand notation for the the two wave functions. Of course, behind the bra-ket notation you have exactly the kind of overlap integral shown in Eq. (1).

Next step is to work out the geometry. Let $vec{r}$ be the vector connecting sites $i$ and $j$, and let $vec{d}$ be a unit vector along the same direction. We decompose the $p$ orbital at site $j$ into components parallel to ($sigma$) and perpendicular to ($pi$) the vector $vec{d}$, as shown in this figure:

Figure from Anthony Carlson's 2006 MSc thesis at University of Minnesota.

In this notation, $sigma$ and $pi$ (and also $delta$) is used to denote the component of angular momentum about the axis $vec{d}$. $sigma$ means zero, $pi$ means $1$, etc. To proceed, we also define $vec{a}$ as pointing along the $p$ orbital, and $vec{n}$ as a vector perpendicular to $vec{d}$ in the plane spanned by $vec{a}$ and $vec{d}$. Then, the $p$ orbital (at site $j$) can be decomposed $$ |P_alpharangle = vec{a}cdotvec{d}|P_sigmarangle + vec{a}cdotvec{n}|P_pirangle. tag{4} $$ Then, the overlap is simply $$ langle S|H_{2c}|P_alpharangle = left( vec{a}cdot vec{d} right) langle S | H_{2c} |P_sigmarangle + left( vec{a}cdotvec{n} right) langle S | H_{2c} | P_pirangle, tag{5} $$ where the second term is zero by symmetry. Further, we can easily express this in terms of directional cosines since we can without loss of generality choose $vec{a}$ to be parallel to one of the coordinate axes. With $$ d_x=frac{vec{r}cdothat{x}}{|vec{r}|},quad d_y=frac{vec{r}cdothat{y}}{|vec{r}|},quad d_z=frac{vec{r}cdothat{z}}{|vec{r}|}, tag{6} $$ we get $$ langle S|H_{2c}|P_alpharangle = d_alpha langle S|H_{2c}|P_alpharangle = d_alpha V_{spsigma}, tag{7} $$ where all radial overlap is hidden in the parameter $V_{spsigma}$. In tight-binding models it's common to denote this overlap integral $t_{spsigma}$ if it occurs in a hopping term.

The same approach works for other orbital combinations. You just need to set up the geometry and coordinate systems properly, and know where the atomic orbitals point. (Admittedly this can become quite complicated in some systems, e.g. transition metal oxides.) Then the Slater-Koster parameters can be treated as tuning parameters - either tuned to explore possible phenomena in some system, or fit to reproduce some experiment or calculated band structure.