Is quantum computing mature enough for a computer scientist with no physics background?

It has pretty much always been like that. You can study the book by Nielsen & Chuang without knowing about physics. There is the introduction by Mermin aimed at computer scientists. There are probably lots of other resources (I'm pretty sure e.g. that Aaronson's book -- based on a CS lecture -- is perfectly suited for people without a physics background.) Overall, the physics formalism needed to understand quantum information and computation is pretty low-key, as compared to other fields of (quantum) physics. (This doesn't mean though that studying the phenomena in quantum information and computation is low-key.)

As I can relate in my experience, I will say yes. One can indeed design algorithms without physics knowledge. For me it is so far Maths concepts. I remember once I watched a course about quantum computing from Scott Aaronson and he quoted :

Quantum computing is really "easy" when you take the physics out of it.

However, if you are going to work on applications in physics or chemistry, it will always be useful to have a background of what you are going to work on.

The field is open to many backgrounds (Maths, Physics, Computer Science...). I think it is just a challenge sometimes communicating between different backgrounds but it is not impossible. Indeed, I would say it is constructive and it can be beneficial to collaborate together. But one can always relate to his preferred interpretation/concepts.

As a career, it is again according to your point of view. I think there is much work to be done in this field so do not worry about it. Do it if you feel you like it. Plus working in this field does not mean you have to restrain yourself. You will still have to work with classical algorithms and you will need coding skills.

If you are interested in learning it from a computer scientist point of view, there is this book that may be helpful: https://www.amazon.com/Quantum-Computing-Computer-Scientists-Yanofsky/dp/0521879965

Good luck on your quantum trip !

Speaking as a computer scientist without any physics background making contributions to quantum computing: yes, computer scientists without any physics background can make contributions to quantum computing. Though I think it was always that way; it has nothing to do with the field being "mature".

If you understand the postulates of quantum mechanics (operations are unitary matrices, states are unit vectors, measurement is a projection), and know how to apply those in the context of a computation, you can create quantum algorithms. The fact that these concepts were originally derived from physics is historically interesting, but not really relevant when optimizing a quantum circuit. As a concrete example: quantum physics is very heavy on calculus, but quantum computation isn't.

Quantum physics does become relevant if you are trying to design algorithms for simulating quantum systems. And some of the concepts you would learn in a quantum physics course should also appear in a quantum computation course. But overall I agree with Scott Aaronson:

[My] way to teach quantum mechanics [...] starts directly from the conceptual core -- namely, a certain generalization of probability theory to allow minus signs. Once you know what the theory is actually about, you can then sprinkle in physics to taste [...]

[quantum mechanics is] not a physical theory in the same sense as electromagnetism or general relativity. [...] Basically, quantum mechanics is the operating system that other physical theories run on as application software [...]

[...] [quantum mechanics is] about information and probabilities and observables, and how they relate to each other.

Quantum computing is totally understandable without physics background for several reasons:

1. Quantum computing is sufficiently written in abstract mathematical language (Linear Algebra).
2. Quantum computing does not include dynamical equations such as Schrodinger equations. Moreover, all operations can be understood as logical gates instead of physical entities.
3. Quantum mechanics itself made a bridge between physics and information in a such way that information scientist can understand it from his/her perspective.

However, quantum mechanics and quantum computing includes new concepts that are hard to accept by everyone including physicists. So don't worry.