When it's work-based, it's not only offensive to you but to the person who hired or promoted you. It insinuates that the other person couldn't see past your looks.

I suggest a response along the lines of -

"If you're really concerned that I don't deserve this promotion, I'd be happy to ask (boss) to talk you through their decision-making process."