Can we use reinforcement learning and convex optimization to solve an optimization problem?

You can use RL in any step. But problem is optimality check of solution which is explained above. Also you can solve your problem directly using RL such as RL for VRP. And you can read this blog which is about RL usage. By the way your question is too broad to give detailed answer.

Not really, but approximately. By OR standards, a problem is "solved" once we manage to satisfy the KKT conditions. There is no machine learning algorithm to date that can consistently satisfy constraints. ML is designed to give pretty good approximations, and that's about it.

For instance, image recognition can be posed as a convex optimisation problem, and we all know how well ML works on those problems. That doesn't mean however that it will always work (and it doesn't), unlike an optimisation algorithm.