Thank you for writing such a vulnerable piece!
I also struggled when I was the same size/slightly larger than my last ex. In theory I knew it was fine, why should I want to be smaller in an age where no lions were around to attack and it wasn't like other, bigger men were trying to intimidate him? What about feeling petite is appealing?
So there's the mental piece of 'this shouldn't matter', and then the biological piece of 'this really bothers me'.