Why is LaTeX3 taking so long to come out?

Will has written a couple of short answers, I'll add a bit of detail.

There are several things that are wrong with LaTeX2e. Most obviously to end users, the kernel is rather inflexible. As a result, to get what you want you almost always need to load quite a number of packages, or do some low-level hacks of the kernel. That is not really ideal: I think most people would be happier not having to remember 'load package X to do ...' for every document they write. So the first challenge is to write a kernel which does many of the common tasks (such as those covered by KOMA-Script or memoir) without needing add-ons. The current aim is to include 'most of The LaTeX Companion': that is a lot of material.

The second challenge is that programming LaTeX2e is a mix of TeX, documented LaTeX, undocumented LaTeX and picking up stuff from packages. That is not a good thing: what is needed is documented system for programming. That is at least partly what expl3 is about: it provides a lot of lower-level programming material, although there is still lots to do.

The next things are 'big challenges' (we know how to do the two things above, it's just lots of work, whereas here I'm talking about things we have to work out how to do). Actually providing a way to make design and coding separate is not easy. The CSS model from HTML does not answer all of the issues we have in a typesetting system, so a new model is being developed (as xtemplate). There are then questions such as a better output routine, to allow things like grid typesetting, more complex column and float placement and so on. None of this is easy.

You also cannot ignore the fact that LaTeX is written by people in their spare time. There have been periods when the various people involved have had a lot of things on outside of LaTeX. It is pretty clear that some momentum was lost due to writing The LaTeX Companion, peoples jobs, other LaTeX packages, etc.

Put together, these issues mean that the project is very challenging. Looking at it today, both Will and I are active in trying to get the work done. We've both only joined the LaTeX Project relatively recently, and are putting in the serious effort that is needed. One of the aims of releasing material to CTAN as it gets to a stable state is to demonstrate that delivery is possible. For example, both expl3 and xparse are usable and do give real benefits.

One final point. LaTeX2e was released in 1994, and since then has acquired a lot of users. Even so, some people still use LaTeX2.09 concepts (such as documentstyle). LaTeX3 will face a much bigger 'hill' than LaTeX2e did, and it will only succeed if the benefits are big enough to get a critical mass of users. To do that, we need to deliver on a lot of the issues with LaTeX2e. (See for example the 'there is nothing to improve' answer.)

There is always more that can be done, and I am focused on delivery (I don't want to load lots of support packages any more than anyone else does). One place to help out would be to work on documentation. We have some stuff, but we know that it needs improving. Feel free to drop the Project a line with offers of assistance or send a pull request to LaTeX3 GitHub repository!