Which neural network should I use to distinguish between different types of defects?

I want to teach a neural network to distinguish between different types of defects. For that, I generated images of fake-defects. The images of the fake-defect types are attached.

I tried many different network architectures now:

• resnet18
• squeezenet
• own architectures: a narrow network with broad layers and high dropout rates.

I have to say that some of these defects have really random shapes, like the type single-dirt or multi-dirt. I imagine that the classification should not be as easy as I thought before, due to the lack of repetitive features within the defects. But I always feel like the network is learning some "weird" features, which do not occur in the test set, and the results are really frustrating. I felt like teaching binary images had way better results, which should IMO be not the case.

Still, I feel like a neural network should be able to learn to distinguish them.

Which kind of network architecture would you recommend to classify the images in the attachment?