So many of the arguments they make against trans women's existence just don't make sense when applied to trans men. A lot of us trans guys appear/present "like guys" and/or we "look trans," and realistically the anti-trans people do not want us in ladies' rooms. The anti-trans people talk big, claiming to know what we trans people "really are," yet they'd be utterly unprepared if they ever met one of us in a normal human context — and they devote especially little imagination to anticipating what a trans man looks like.
And if (as they say) one's hormones and surgeries don't make one a man or woman, then why do they care if people choose hormones and surgeries, and why do they infantilize trans men regarding these choices? Anti-trans people seem to believe that women (in which category they include trans men) are unable to make these choices. Which is misogynistic, as you say.