LWE versus neural nets

This seems quite unlikely. A recent paper has suggested a Continuous LWE problem. This problem is structurally quite similar to the standard LWE problem --- it has (quantum) worst-case to average-case reductions to hard lattice problems.

In that paper they also show that a standard problem in ML (learning mixtures of Gaussians) reduces to the continuous LWE problem. This gives you a more concrete ML problem to attack. If you could learn mixtures of Gaussians, you could reduce this to solving continuous LWE, which you could then reduce to solving worst-case lattice problems. If you can solve worst-case lattice problems, then we do not believe LWE is hard anymore.

There have also been some results showing that a broad class of learning algorithms (ones which fit into the "Statistical Query model") cannot solve certain learning problems. The LPN problem (which you can vaguely think of as "LWE with $q = 2$) has such bounds (see for example this problem). I don't know if specific ML techniques such as backpropagation fit into the SQ model though.

Well, they can be applied, but are probably unlikely to succeed, otherwise you would have a breakthrough, so maybe go for it if you have a concrete idea you can implement.

The reason is this kind of discrete problem (lattice or finite field) is chosen to be very difficult on average, and many hardness results tied to famous problems usually exist.

The ML problems are non adversarial in nature, so apples and oranges, really.