How would you build a big production ready image training dataset from scratch?

I'm not expert in images classification, I'm just going to give some general advice here.

The strategy should be progressive, for instance:

1. Proof of concept: devise a first draft of annotation process with some general guidelines, then take a random subset of a few hundreds images and try annotating them following the guidelines. Observe carefully the problems found at the annotation stage: any ambiguous cases, any problems with the granularity of the classes (for instance a class which is too general), potential error cases. Devise a training/testing setup to train a mockup system, test it with the small dataset, fix any bugs in the process, possibly try different methods. The goal of this stage is to iron out the different stages, possibly rule out some options which turn out to be unworkable.
2. Prototype: have a team of a few laypeople annotate a few thousands images following the updated annotation guidelines. Again pay attention to any problem, especially human errors in the process. At least a subset of the images should be annotated by several annotators, in order to detect differences. Ask annotators for feedback: were there any difficulties, was it fun, would they do it as a game or not, etc. At this stage you can start having a real ML system, albeit not optimal yet. Analyze the performance of the system on the different classes, possibly try different methods, estimate the minimum number of images per class to obtain decent performance. At this stage a lot of other tests could be done, and it should become clear how accurate the manual annotation is and whether annotators need to be compensated or not.

It's only around this stage that the final annotation process can be fully designed. Depending on the strategy, you could consider devising a strategy for iterative manual annotation: some classes are going to be quickly learned by the model, so it could make sense to use the model in order to propose for annotation at the next round the images for which the model fails. Be careful to avoid bias, keep evaluating and refining the model at every round of annotations.