When is it time to switch to deep neural networks from simple networks in text classification problems?

I did an out of domain detection task (as a binary classification problem) and tried LR and Naive Bayes and BERT but the deep neural network didn’t perform better than LR and NB. For the LR I just used BOW and it beats the 12-layer BERT.

In a lecture, Andrew Ng suggests “Build First System Quickly, Then Iterate”, but it turns out that sometimes we don’t need to iterate the model into a deep neural network and most of the time traditional shallow neural networks are good/competitive enough and much simpler for training.

As this tweet (and its replies) indicate, together with various papers [1, 2, 3, 4 etc], traditional SVM, LR, and Naive Bayes can beat RNN and some complicated neural networks.

Then my two questions are:

1. When should we switch to complicated neural networks like RNN, CNN, and transformer and etc? How can we see that from the data set or the results (by doing error analysis) of the simple neural networks?

2. The aforementioned experiments may be caused by the simple test set, then (how) is it possible for us to design a test set that can fail the traditional models?