What should the torque of an expander plug in a carbon fork be?

You use the manufacturer specified torque, which in this case is max 8 Nm but in another component it could be different. That value sounds like more than we are used to in this area of the bike, but it is presumably for a 6 mm hex fastener, in which case it isn't an unusual value - it would be moderate torque on a 6 mm bolt.

The resulting force is spread out a little more over the larger component, than it would be if it was acting at a localised pinch point in another component. 8 Nm would be too high for a stem bolt, for example, but the stem compresses inwardly, while the expander pushes outward, which is very different forcing on a carbon steerer. Carbon fibres are very strong under tension, but floppy when compressed. This means it is difficult for them to resist a clamping force, hence the cited need for an expander plug to support the tube.

When the expander is pushing outwards from inside the tube, the carbon fibres are not being compressed, but are under tension instead, which is when they are strong. Like when you wrap a tape measure around your bicep and tense the muscles. That 'weak and floppy' tape you could crumple if you squashed it, is suddenly able to resist the large force of your expanding muscles when being expanded.

So given the size of the hex key required, the need for securely fixing the plug to adjust the headset and the carbon fibres not being compressed, I am of the opinion that max 8 Nm is quite a reasonable value to torque up to, if the manufacturer's instructions are hard to come by. one could interpret it as a suggested 6-8 Nm, which means using the 6 mm hex key with a moderate force but a little sensitivity.

The expander plug has no load to bear. After the steering has been adjusted and the stem clamped you could theoretically remove the top-cap or at least loosen the bolt. In the early stages of adjustment, the plug needs just to be tight enough to prevent it from being pulled out of the steerer by the top-cap bolt.

So 5-6Nm would be a good start for tightening. Consider 8Nm the absolute upper limit. Also, before tightening the plug, put the spacers and the stem on the steerer as they will prevent the (carbon)-tube from expanding.