Correct usage of "verb+ing" used as an adjective

Sentence –

1. It’s literally annoying to tell someone to do all the work.

I have a doubt regarding the implication of this above mentioned sentence , who’s being annoyed here the person who’s being told to do all the work , or the person who’s telling the other one to do all the work .

From what I know , it should mean that it’s annoying to be told to do all the work . But it’s said in a different way .. and it doesn’t matter how you say it implication remains the same