How do these similar engines differ so much in power output?

So I think that you have most of the reasons in your question.

But I will list a few more that I think is possible:

• Higher compression ratio
• Different inlet manifolds
• Probably larger valves
• Lighter pistons, valves, rods etc, maybe even flywheel and crank (higher red line)
• Stiffer valve springs, and different lifters (higher red line)

All things except the lighter parts will give a higher power, it can also increase the red line some, and combined with lighter parts the red line is actually increased by quite a lot. Actually the max power for the M54 engine is at 5900rpm while it's at 7900rpm in the S54 engine. Looking at the torque / power curve for the S54 it looks like the power at 5900rpm is around 275hp. So those extra revs give quite a lot!

The biggest difference is that the M3 engine uses a cast iron engine block compared to the alu-sil block of the 330. Under hard (very hard) use, alloy blocks can physically twist and deform whereas the cast iron block is "harder" so resists internal forces more effectively. This is why the Alfa Romeo twin spark engines can use oil when pushed hard.

The M3 runs a compression ratio of 11.5:1 compared to the 10.2:1 of the 330. This means that the pistons are effectively "squishing" the fuel/air mix more.

The M3 has solid lifters in its camshaft gear which means it can be revved higher (8k RPM vs. 7.5k RPM redline of the 330 with its hydraulic lifters).

The camshaft profiles on the M3 are more aggressive so it's keeping its valves open for longer with more overlap.

The M3 runs independent throttle bodies meaning that each cylinder is fed through its own butterfly and trumpet compared to the plenum chamber system in use on the 330. This is a little like the old Twin DCOE Weber carburettor upgrade of old.

The M3 has a much more performance-oriented map on its fuel management ECU. This means that it isn't tuned for general road use like the 330 but instead sacrifices some fuel economy in favour of higher power output.

The exhaust system fitted to the M3 is significantly free-er flowing than that of the 330 meaning it can expel a larger volume of exhaust gasses at the cost of a less quiet system.

You will generally find, for the performance variants of most models, there is a significant difference between the "cooking" versions and the sports-tuned versions. This is as true of the Mini VS Mini Cooper S and Golf VS Golf GTI as the Aston Martin DB9 VS the DBS and Lotus Elise VS Lotus Exige.

Cylinder capacity is only a small part of the story when it comes to engine performance and tuning.

I'm not an automotive engineer, but the differences are primarily:

• higher rpm
• higher compression ratio
• probably a better cylinder head design (valves, ports etc.)

All these things create more power for the same reason: they result in more air being displaced in a given space of time.

The other details - intake design, lifters, VANOS, injectors, ECU maps etc. all exist only to support these features.

A few things tend to limit rpm, but I believe the big ones are valve timing and intake resonance.

Valve timing as you mentioned is controlled by VANOS, but it's not the same in the M3. The base models use single VANOS (intake cam only), but the M cars use double VANOS (both cams). (I believe M VANOS operates at higher pressure so that it can react faster, although that's really a features for responsiveness rather than peak power.)

The intake manifold will always tend to resonate at one particular frequency (which translates to one particular engine speed, i.e. rpm), and this affects the volumetric efficiency of the engine. Think of it as waves interfering with each other; when they are perfectly in phase, then tend to strengthen each other (so more air gets in). When the waves are out of phase, then diminish each other, and less air gets in. Most cars are designed with fairly long intake runners, which is optimized for low to medium rpm torque, since most cars have cam profiles that are optimized for that rpm range. Some cars actually have variable intake geometry to compliment their variable valve timing. The M3 has no intake runners at all - instead each cylinder has it's own throttle body, so it's optimized for very high rpm.

Similarly, the lifters are the older, non-hydraulic type to support the goal of high rpm - hydraulic lifters are heavy and so have too much inertia.

Finally, valve ports are typically a bottleneck on engine performance, because all the air has to go through a fairly small hole. Ordinary cars don't get perfectly optimized heads, because it would be too expensive, and would probably just move the bottleneck to the intake or the cam. But performance cars tend to get very carefully optimized cylinder heads because it's worth the effort, when you have everything else in place to get high air flow.

Once you have your power goal, that tells you how much air you need to move. Then you can decide whether to achieve this goal via displacement, higher rpm, or forced induction, or some combination of these. Once you decide on that, the rest of the details - block material, mass of reciprocating parts, valve timing, intake design etc. all follow from that.