Interesting. I've also struggled with finding the right words to describe bigotry.

For a while, I avoided "homophobia"/"transphobia" because it seemed to immediately derail any conversation, as the bigot would reply: How do you know I fear you? You can't see inside my private emotional state! I'm actually not afraid of you! To the contrary: I have perfectly valid, common-sense reasons for hating you! So your word 'homophobia' is a slur against me! etc. But I usually avoid interacting with bigots who are going to pull that boring tired shit anyway, and if my audience is people who already know what homophobia/transphobia is and who aren't going to pick apart existing terms on specious grounds, I don't need to waste time hedging fake arguments from bigots trying to waste my time. And so recently I've tried using it again.

One way in which I've come to be a bit more accepting of the "-phobia" terms is to think of them in the scientific sense. A hydrophobic molecule, for example, is a material that doesn't dissolve in water. The molecule isn't pulled in by water; that doesn't mean it's afraid of water. A molecule isn't conscious. In that sense, we could refer to socially "phobic" behaviors even if we completely bypass individual psychological critiques. We just look at what people actually say and do. When people avoid and resist other people, they're being "phobic" in the way a chemist might describe the interaction of two substances.

But you raise a good point that the primary meaning is psychological. Fear is one thing, while prejudice and organized hate are another thing, and conflating the two takes away a useful term that people use to describe their, well, phobias.

Finding the right word to describe bigotry is an ongoing challenge, especially because bigotry is often unconscious, and even when it's conscious, the bigot always denies what they're doing and tries to derail the conversation by resisting the label.