In amplitude amplification, isn't the speedup hindered by the realization of $S_o$?

In brassard et al. Amplitude Amplification work, they define the Q operator as

$$mathbf{Q} = -AS_{o}A^{-1}S_{chi}$$

where $S_{o}$ is an operator which flips the sign of the $|0 rangle$ state.

Which is basically a diagonal unitary matrix (in the computational basis) with -1 on the first diagonal element.

I was wondering, isn’t Quantum amplification’s quantum speedup hindered by the realization of $S_o$ when the number of qubits in the circuit is too big? Based on Barenco’s et al work (Elementary gates for quantum computation), isn´t the number of gates required for a $n$-qubit controlled gate exponential in $n$?