A Specific Case of Infinitive as Adjective

To see him, you would have supposed he had been doing this for years.

Having first thought that to see him was an adverb, I then noticed it wasn’t exactly doing the work of an adverb and might just as well be replaced with the participle seeing him, modifying you. Is this right, or have I missed something?

Thank you.