Answering with another question for "What do you mean"

Yes, it is appropriate to use "By what?" as a response to the first question. This usage implies that the second speaker is unsure what the first speaker is referring to. I'd say that the interaction you have written sounds fluent in American English.

If you want a more formal way, you can consider using a full sentence: "What do I mean by what?"