This is exactly what is happening for me. I used a Suptronics X6000 HDMI extractor which has 4 stereo outputs. Channels get messed up frequently.I don't know if it is related or not, but I modded a multichannel HDMI audio extractor to get 8 channel multiple spdif outputs from my RPI 4b via HDMI and I also had the channels messed up everytime there was a buffer underrun (I was using CamillaDSP). The only solution was to automatically restart CamillaDSP every time it happened.
I suspected the problem was the driver, but people more knowledgeable told me the driver was apparently fine...
It is quite annoying that nobody can confirm that 8-channel I2S (BCK/LRCK + 4 pairs of I/O pins) works on RPi5. The pin mapping is listed in the RP1 datasheet but that is all.