Should one stick to American style of placing punctuation marks within quotes if one uses the American spelling?

The spelling you prefer doesn't have to determine the punctuation style, but if you want to use the British punctuation style using British spelling could work as a clue to the reader that your punctuation isn't incorrect. Or, depending on the reader, it might just make it look like your punctuation and spelling are incorrect.

My own personal rule, as an American, is to put the punctuation inside the quotes unless that results in ambiguity as to whether the punctuation is or isn't part of the quoted text. For example, the sentence:

When the dialog box appears, type "password".

seems less ambiguous than:

When the dialog box appears, type "password."

If you're following American spelling and usage conventions, then you probably ought to adhere to American punctuation as well, for sake of consistency. The Chicago Manual of Style and Robert Bringhurst's Elements of Typographic Style are two sources for advice on usage and punctuation that support American style; whereas if you devise your own hybrid style, you might wind up with a hodge-podge of exceptions and idiosyncratic affectations.

That's not illegal - you're quite free to follow your linguistic muse in the direction it leads you, but your readers may suffer for it.

It's worth noting that British style isn't necessarily easy to apply. British placement depends on whether the quoted material is complete or a fragment. Making that determination in some cases can become a matter of subjective opinion - in which case the American style starts to make a strong case for itself.

Referring to this way of placing quotation marks as the American style is a shorthand for something like the style specified by the stylesheets used by the majority of the publishers based in the U.S. (but not many publishers that are based elsewhere). When one reflects on that, one can see that this is something entirely independent of spelling, and that there is nothing inconsistent about using American spelling and the 'British' way of using quotation marks in the same text.

While American spelling is deeply ingrained in American life, this is not the case with the 'American' way of combining quotation marks with other punctuation. The 'American style' is imposed and enforced from above, by the publishers. When writing something that is to be published by a U.S. publisher, one may have no choice but to go along, but that doesn't mean that, if one regards this style as irrational, one has to follow it otherwise, even in an American context.