I think that's a good point. And I do think that issue is operational to some degree in how some audiophiles switch equipment.
But even so, I have also noticed that it's far from necessarily the case.
I say this because, having been a citizen of many different audio forums over decades, I've seen tons of threads asking about people's speaker history and what they'd settled on as their "keeper/end game" speaker. (I don't pay attention btw to people who are just currently infatuate with their new speaker thinking they have "found the one").
And when those threads/surveys come up the speakers that audiophiles have settled with, expressing long term satisfaction, have been all over the map - from original Quads, to varieties of horn speakers, to omnis, to dipoles, to wide dispersion, narrow dispersion, you name it.
As almost none of these people had the luxury of double-blind testing their speakers, surely some level of sighted (and other) bias may be in play.
But if so, whatever combination of sound/bias was in play, it still worked out to many people's satisfaction in settling on a speaker.