What does it mean to contract the indices of a Lorentz matrix?

This took me quite a while to get at the beginning of relativism class but this may help: a general matrix multiplication like we all know it looks like this: $$[A][B] = sum a_{ri}b_{ir}$$ Now the relativity version with einstein summation convenction would make this matrix multiplication look like this: $$[A][B] = a_{nusigma}b^{sigmanu}$$ As you see this is the same as the previous statement with just the summation symbol dropped, and thus with the first index representing the row and the second representing the column. Now there are possibly people who have an explanation for how a (2,0) (i.e 'covariant') tensor is different from a (0,2) (i.e 'contravariant') tensor, however I just keep the rule in my head that you can only contract covariant with contravariant indices, and thus apply a metric of the form $g_{mu mu}$ ($g^{mu mu}$) if it is necessary to lower (rise) indices. To get to the form of matrices it is necessary to convert the tensor back to a matrix-form by contracting it with a metric. I will use the minkowski metric $eta$ here for the demonstration. This brings us back to your question, your equation can be written as: $$Lambda_{sigmaalpha}eta^{mu sigma}Lambda^{nusigma}eta_{betasigma}eta_{munu} = Lambda_{sigmaalpha}eta_{betasigma}Lambda^{nusigma}delta^{sigma}_{nu} = Lambda_{nualpha}eta_{betasigma}Lambda^{nusigma} stackrel{nu leftrightarrow beta}{=} [Lambda]cdot[eta]cdot[Lambda]^T$$ Where I've used the fact that $$eta^{musigma}eta_{betasigma} = delta^{mu}_{sigma}$$ And just basic rising and lowering of indices via: $$Lambda^{mu}_{;alpha} = Lambda_{sigmaalpha}eta^{mu sigma} text{ and } Lambda_{mu}^{;alpha} = Lambda^{sigmaalpha}eta_{mu sigma}$$

I could've made an index mistake somewhere but I hope this gives a kind of understanding and I hope it helps.

First a brief introduction to tensors. An $(r,s)$ tensor $t$ on a $K$-vector space $V$ is just a multilinear map from $s$ copies of $V$ and $r$ copies of the dual vector space $V^*$ to the underlying field $K$: $$t:underbrace{V^*times...times V^*}_{r-times}timesunderbrace{Vtimes...times V}_{s-times}to K$$ The dual vector space is just the set of all linear maps from $V$ to $K$. The set of all $(r,s)$ tensors on $V$ is commonly denoted $T^r_s(V)$.

Ok, how do we get from these vectors and linear maps to these numbers $g_{munu}$ that we're working with?

Well, we do what we always intuitively do namely, choose a basis. Say we chose some basis ${e_i}subset V$ on our vector space, we can now express every vector $vin V$ as a linear combination of these basis vectors: $v=v^ie_i$. The $v^iin K$ are basically numbers and the position of the index so far is just convention. We can now do the same for the dual vector space, with a particularly clever choice of basis ${e^i}subset V^*$, such that $e^{i}(e_j)=delta^i_j$. Again, the position of the index is just convention in order not to mix up vectors and dual vectors, as these are fundamentally different objects!

With this index convention, we would now write the components of a vector as $v^i$ and of a dual vector as $v_i$.

For the tensors, it is very similar: $$ t(w_k^{(1)}e^k,...,w_l^{(r)}e^l,v^i_{(1)}e_i,...,v^j_{(s)}e_j)=w_k^{(1)}...w_l^{(r)}v^i_{(1)}...v^j_{(s)}t(e^k,...,e^l,e_i,...,e_j) $$ The $t(e^k,...,e^l,e_i,...,e_j)equiv t^{k...l}_{i...j}$ are again just numbers, independent of what vectors and covectors the tensor is acting on (just the basis we chose). Similarly to the vectors and covectors, we can also express the tensor in terms of its components: $$ t = t^{k...l}_{i...j}e_kotimes...otimes e_lotimes e^iotimes...otimes e^j $$

Ok now, what happens when we choose a different basis ${b_i}subset V$ and a corresponding dual basis ${b^i}subset V^*$? Since our new basis vectors (and new basis covectors) are still elements of the same underlying (dual-)vector space, we can express them as some linear combination: $$b_i = A_i^je_j qquad text{and} qquad b^i=B^i_je^j$$ Similar to above, we can view the numbers $A_i^j$ and $B^i_j$ as components of a tensor: $$ A=A_i^je_jotimes b^i qquad text{and} qquad B=B^i_jb_iotimes e^j $$ note: In physics, we typically don't view these transformations as tensors, however in I think this is quite useful for our discussion here. For more information see, e.g. (Is Lorentz transform a tensor?)

Now lastly what does contracting two tensors actually mean? Tensor contractions are really just defined for an individual tensor and are linear maps $C^k_l:T^r_s(V)to T^{r-1}_{s-1}(V)$ defined by: $$ T^{nu_1...nu_r}_{mu_1...mu_s}e_{nu_1}otimes...otimes e_{nu_r}otimes e^{mu_1}otimes...otimes e^{mu_s} mapsto T^{nu_1...nu_r}_{mu_1...mu_s} e^{mu_l}(e_{nu_k}) e_{nu_1}otimes...otimes e_{nu_{l-1}}otimes e_{nu_{l+1}}otimes...otimes e_{nu_r}otimes e^{mu_1}otimes...otimes e^{mu_{k-1}}otimes e^{mu_{k+1}}otimes...otimes e^{mu_s}) $$ But thats no problem, since we can make one tensor out of two using the tensor product. In your particular case we have the "tensor" $Lambda=Lambda^{nu}_{mu}e_{nu}otimes b^{mu}$ and the tensor $eta=eta_{munu}e^{mu}otimes e^{nu}$: $$ etaotimesLambda = eta_{munu}Lambda^{alpha}_{beta}e^{mu}otimes e^{nu}otimes e_{alpha}otimes b^{beta} $$ and $$ C^{alpha}_{nu}(etaotimesLambda)=eta_{munu}Lambda^{alpha}_{beta}e^{nu}(e_{alpha})e^{mu}otimes b^{beta} = eta_{munu}Lambda^{nu}_{beta}e^{mu}otimes b^{beta}equiv Lambda_{mubeta} e^{mu}otimes b^{beta} $$ In GR (and SR) space-time is a Lorentzian manifold $L$, the vector space of interest is the tangent space $T_pL$ at some point $pin L$ and the tensors you're considering have special meaning, however (as always) the maths doesn't care too much about the physics.