In "Brown Man's Burden" is the antecedent changing?

I was reading the "The Brown Man’s Burden" by Henry Labouchère.

I was a little confused because the antecedent for "you," seemed to be changing from the White men to the Brown men (Philippine natives) in the poem.

For example, in the first stanza,

Pile on the brown man’s burden;
And, if ye rouse his hate,
Meet his old-fashioned reasons
With Maxims up to date.

Here, "you," seems to be referring to the natives who anger "his hate," referring to the White men.

However, in all other parts of the poem, "you," seems to refer to the White men, as in the first stanza that states, "Pile on the brown man’s burden
To gratify your greed".

Please let me know if I am analyzing this poem correctly.