Is finetuning from a pretrained model always better than training from scratch?

At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn’t it be better to always start with a pretrained model, as the lower layers of weights has already probably learnt general patterns of images that are transferable across all data sets?

My worry is what if the dataset I want to use for finetuning is highly specialized, highly unnatural and very different from the dataset the pretrained model is trained on. Would this still mean finetuning from a pretrained model wouldn’t be the best idea? (e.g. training on X-Ray images instead of natural images for a cifar-1000 pretrained model.)