There's many reasons, both in theory and implementation, that make quantum computers much harder to build.

The simplest might be this: while it is easy to build machines that exhibit classical behaviour, demonstrations of quantum behaviour require really cold and really precisely controlled machines. The thermodynamic conditions of the quantum regime are just hard to access. When we finally do achieve a quantum system, it's hard to keep it isolated from the environment which seeks to decohere it and make it classical again.

Scalability is a big issue. The bigger our computer, the harder it is to keep quantum. The phenomena that promise to make quantum computers really powerful, like entanglement, require the qubits can interact with eachother in a controlled way. Architectures that allow this control are hard to engineer, and hard to scale. Nobody's agreed on a design!

As @pyramids points out, the strategies we use to correct errors in classical machines usually involve cloning information, which is forbidden by quantum information theory. While we have some strategies to mitigate errors in clever quantum ways, they require that are qubits are already pretty noise-free and that we have lots of them. If we can't improve our engineering past some threshold, we can't employ these strategies - they make things worse!

I have to disagree with the idea that the no-cloning theorem makes error correction with repetition codes difficult. Given that your inputs are provided in the computational basis (i.e. you inputs are not arbitrary superpositions, which is almost always the case, especially when you're solving a classical problem e.g. Schor's algorithm), you can clone them with controlled-Not gates, run your computation in parallel on all the copies, and then correct errors. The only trick is to make sure you don't do a measurement during error-correction (except possibly of the syndrome), and to do this all you have to do is continue to make use quantum gates.

Error correction for quantum computers is not much more difficult than for classical computers. Linearity takes can of most of the perceived difficulties.

I'd also like to mention that there are much more efficient schemes for quantum error correction than repetition codes. And that you need two pauli-matrices to generate the rest, so you need two types of repetition codes if you're going to go for the inefficient, but conceptually simple repetition code route (one for bit-flips and one for phase flips).

Quantum error correction shows that linear increase in the number of physical qubits per logical qubit improves the error rate exponentially, just as it does in the classical case.

Still, we're nowhere near 100 physical qubits. This is the real problem. We need to be able to glue a lot more semi-accurate qubits together before any of this starts to matter.

One important point is that quantum computers contain classical computers. So it must be at least as hard to build a quantum computer as it is a classical computer.

For a concrete illustration, it's worth thinking about universal gate sets. In classical computation, you can create any circuit you want via the combination of just a single type of gate. Often people talk about the NAND gate, but for the sake of this argument, it's easier to talk about the Toffoli gate (also known as the controlled-controlled-not gate). Every classical (reversible) circuit can be written in terms of a whole bunch of Toffolis. An arbitrary quantum computation can be written as a combination of two different types of gate: the Toffoli and the Hadamard.

This has immediate consequences. Obviously, if you're asking for two different things, one of which does not exist in classical physics, that must be harder than just making the one thing that does exist in classical physics. Moreover, making use of the Hadamard means that the sets of possible states you have to consider are no longer orthogonal, so you cannot simply look at the state and determine how to proceed. This is particularly relevant to the Toffoli, because it becomes harder to implement as a result: before, you could safely measure the different inputs and, dependent upon their values, do something to the output. But if the inputs are not orthogonal (or even if they are, but in an unknown basis!) you cannot risk measuring them because you will destroy the states, specifically, you destroy the superpositions that are the whole thing that's making quantum computation different from classical computation.

Ultimate Black Box

A quantum computer is by definition the ultimate black box. You feed in an input and you get a process, which produces an output.

Any attempt to open up the black box, will result in the process not happening.

Any engineer would tell you that would hinder any design process. Even the smallest design flaw would takes months of trial and error to trace down.

Simpler answer: All quantum computers are classical computers too, if you limit their gate set to only classical gates such as $X$, which is the NOT gate. Every time you build a quantum computer, you're also building a classical computer, so you can prove mathematically that building a quantum computer must be at least as hard as building a classical computer.

In 1996, David DiVincenzo listed five key criteria to build a quantum computer:

1. A quantum computer must be scalable,
2. It must be possible to initialise the qubits,
3. Good qubits are needed, the quantum state cannot be lost,
4. We need to have a universal set of quantum gates,
5. We need to be able to measure all qubits.

Two additional criteria:

6. The ability to interconvert stationary and flying qubits,
7. The ability to transmit flying qubits between distant locations.

Long Explanation