Could I use the sentence like this?

You should use past perfect to talk about something that happened before another action in the past. So in this case you better avoid using the past perfect as you are stating a fact (not speaking Japanese) rather than implying the order of two actions in the past. Your first sentence is fine. An example in which you should use past perfect may be something like:

I had lived in Japan for three years when I started to speak Japanese fluently