Excellent answers. I may have some questions, but I figure I should answer my own question as well:
My answer to the main question would be "no" (I wouldn't want all speakers to sound the same) though with caveats.
First, I very much would like significant research and engineering to continue to push the state of the art, and at least one version or aspect of that is low distortion/accuracy. I very much want neutrality/low distortion/accuracy as one of my options! (It's why I use some Benchmark equipment for instance).
And of course I would never want to make my own tastes conscribe anyone else's, so I'd want all sorts of excellent neutral gear available for those with those goals.
But personally, and I think some of this is perhaps because I still have a foot in the Old School: I have always enjoyed the Wild West aspect of high end audio, often on display at audio shows, where you can find everything from Revel flagship speakers to some hopeless new start up whose product might have some intriguing qualities.
I can see why some evaluate speakers by comparing them to some ideal in terms of measured performance. I have yet to encounter a speaker that sounds as if there were no speaker in the chain, so every speaker to me has some character or another. In that sense I am not comparing speakers to a set of measurements, but to other speakers. How does speaker A sound to me vs speaker B?
And since I've had an almost life long interest in comparing real sound to reproduced sound, I can't help evaluating speakers to one degree or another as to how it sounds further or closer to "the real thing." Some speakers do A and B more like the real thing, some others may do C and D aspects of the real thing.
So I find I end up appreciating different characteristics from different speakers. Sometimes I luxuriate in the electrostat sound, other times in an Omni, other times in a narrow dispersion speaker, other times in a wide dispersion, sometimes in an extremely even, neutral speaker, sometimes in a colored speaker that sounds really awesome with some stuff.
Reminds me of today: I was at my friend's place (reviewer) listening through some tall, narrow speakers. The sound was astonishingly clean, open, boxless, detailed, utterly detached from the speakers, with the sensation of every iota of detail evident in the tracks. So it did a lot of the detail/imaging "wowee" magic act. But when I sort of took stock and thought "ok, forget about how gee wiz the imaging and clarity might be - how compelling is the general character of the sound of each voice and instrument? In the end, it wasn't to my taste. Neato to visit, but not much to love in terms of the tonality I seek.
Whereas when he had the Klipsch La Scala speakers I just couldn't get enough of them! The measurements I've seen for the La Scala's are awful by the standards used here. But..my god... the life-like sensation they created! The density and propulsiveness of the sound - drums, guitars, sax - everything sounded more solid and moving air rather than the sort of airy holograph many speakers create. On those speakers I just wanted to keep putting different records on all day! Not sure I could live with them, but glad as heck someone is making them and I got to experience that sound.
(Same with the Shun Mook speakers I once reviewed. Much of the design brief for those speakers seems ludicrous - would be rightly scoffed at on ASR - and yet, despite that, they produced an almost unique sound that has stuck with me for years as one of my favorites).