As you point out, if anti-trans people deliberately group trans people with the wrong gender (i.e. not the gender they identify as), then, no matter what the trans person has done to be considered a member of that gender, they would be outed every moment of the day through structural misgendering, which leaves the trans person vulnerable to discrimination and violence. Unfortunately this happens to many trans and intersex people.
To which I add—not that trans people should be required to "pass" as a man or woman nor even want or aspire to do so—but, if a trans person does pass, an anti-transgender person would have to figure out that this person is trans (how would they know?), persuade everyone around them that the person is trans (why is it anyone's business?), and then deliberately misgender the trans person even though they pass and otherwise generally conform to a social category despite their harassers' efforts to out them. So, for example: If a trans woman appears, by the social conventions that surround her, to be a woman in 100 different ways and a man in 0 ways, it would require a continuous effort from anti-trans forces to follow her around town and point and say Actually, that's a man! to everyone the trans woman might meet. (And if the anti-trans people argue, Well, there is no such thing as woman or man, just female and male, then why are they fussing over "she" and "he" pronouns? Like, they don't take a consistent position on whether society should or shouldn't be divided and categorized by physical sex. By enforcing a particular use of gendered pronouns, they are turning "sex" into "gender," and then they cannot coherently claim to aim for a society where only sex exists and gender does not. If they really believe in gender-neutrality, they should be endorsing nonbinary perspectives.)
On which point, further, the anti-trans a.k.a. "gender-critical" people have not thought through their supposedly "gender-neutral" position which would allow a person to wear any clothing they like, take any job they like, use any first name they like—while somehow requiring that society perceive them in accordance with their "birth sex" and refer to them with those binary-gendered pronouns that correspond to their binary sex at birth (assuming they even had a clear binary sex at birth). When a person changes their clothing, job, name, etc., they may pass as a member of that gender, meaning that strangers and acquaintances may organically call them "he" or "she" without being asked. Regardless of how the trans person may feel about that, it may happen. Yet anti-trans people are opposed to other people passing. Too bad for the anti-trans people, since it's like being opposed to "weather." It is a thing that happens. Banning or discouraging hormones/surgery alone does not prevent people from passing. The anti-trans people don't even seem aware of this. Or, at least, they are deliberately writing this obvious fact out of their deliberately harmful narratives. If they spent two minutes thinking about how they, personally, "naturally" decide to call someone "he" or "she," they would realize it almost never has anything to do with the person's genitals and even less so with the person's genitals at birth, and has more to do with how they perceive that person in other ways, which is about how that person "passes" to them. I think they do realize this (i.e. that passing is a thing), and they are deliberately lying about it, because they want to create a philosophical narrative of an alternate world in which all cis people "look cis," all trans people "look trans," and intersex people don't exist. I wish them bad luck with their narrative.
