Here is experimental evidence.
Using headphones and a "click" track.
One sample delay = 0.00002083 seconds
2 sample times delay on one channel at 48kHz starts to make a difference in the left/right location for me.
12 sample times or more makes it sound like only one side the headphone is working, though both sides are still outputting at the same level.
So, for me, today, 0.00002083 seconds (one sample time) borders on the imperceptible for location, 0.00004166 seconds delay is audible as a slight left/right location change (maybe perceived as 10 degrees off-center).
The same applies for speakers, though the numbers might be a
little different, as both ears receive signal from both speakers.
View attachment 93840