Why do I sometimes “cream” and “beat until smooth”, and sometimes not?

For the same reason that sauce recipes will sometimes direct you to cut the tomatoes, sometimes to puree them, sometimes to deseed them, etc. You are not only making a cake every time, you are making different kinds of cake, and for each kind, there is a method which will produce the desired type of cake.

It will probably be more useful to stop thinking that 'the basic rule of thumb is to “cream”'. There is no basic rule of thumb. What is needed is to mix the ingredients in a way such as to provide the desired final texture in the cake. There are perhaps 6-7 methods which can be used, and the choice of method is always an integral part of the recipe, to the point where Michael Ruhlmann considers sponge cake and pound cake to be two different cakes, made with exactly the same ingredients at exactly the same ratios, but with two different methods (the creaming and the egg-foaming method).

As a very brief description of the specific methods of creaming and all-at-once, creaming is generally used for cakes with a finer final texture that is softer, with uniform small bubbles. Festive decorated cakes and tortes are frequently made with layers using this technique. The all-in-once method, also called the muffin method, is more commonly used for everyday cakes. It gives you a somewhat irregular texture with some tooth, requires a chemical leavener, and is more tolerant of adding extra ingredients, such as fruit or vegetables in sweet loaves like banana bread, zucchini bread, etc.

There are many different ways to mix ingredients, which will affect the final texture. Here are a few that I'm familiar with, but I suspect a professional baker would know even more:

Creaming : Beating the fat and sugar together first to encorporate air, then add your other ingredients. Requires having a solid fat. Typically used for cookies and some cakes.

Muffin Method : Mixing the liquids and dry ingredients individually, then mixing the two together. Typically used for quick breads (and muffins)

Sponge: Whole eggs are whipped, then sugar and flour are added. Used for pound cake. (some argue that it's no longer a sponge if you add chemical leavener)

Chiffon : Mix together everything but the egg whites. Beat the egg whites, then fold them in. Used for chiffon cakes. (rather similar to making a souffle, but the proportions for cakes are different and includes chemical leaveners)

Angel Food : Whip your egg whites, then slowly fold in the sugar, flour and other ingredients. Used for angel food cake (and requires a special pan, so it's upside down as it cools). Unlike other cakes, there are no egg yolks or other fat.

Obviously, if you don't have a solid fat, you can't use the creaming method. So an olive oil cake simply can't use the creaming method.

Muffin method typically uses a liquid fat (there are variants that use fruit puree and no fat). I'm not familiar with the "All at once" method, but I suspect that it would also use liquid fats.

The only other solid fat methods that I'm aware of aren't used for cakes -- they're used for pie crusts, biscuits, and puff pastry.

As for the question about "beat until smooth", it again depends on ingredients and desired texture -- you typically do not do this with cakes once flour has been added. If you do, you'll develop gluten, which creates a chewy texture. This might be okay for cookies or yeast breads, but it's a problem for cakes and muffins.

If you over-beat a muffin, you'll get "tunneling", when bubbles are trapped inside the batter and end up looking like a worm has burrowed through the item. This is why muffins, brownies, and pancake recipies often tell you to stir until "just combined" or even that few flour streaks are okay.