Saturation seems to occur around 1.5V, distortion just gradually increases from 70mW up to the knee around 1,5V.
This requires 4.3V voltage swing if the buffer was biased correctly (gate at around 3V DC ?)
I figure the Gate DC bias voltage is a bit too low, maybe at 1/2 voltage rail ?
Too much voltage drop across GS would then be resulting in the asymmetric behavior.
Chances are for most people the device would be usable up to 1.5V.
That said most DACs do 2V or even higher.
As said... I would try to run the device at 9V DC and if that did not work I would remove the buffers and bypass the buffer circuit.
I would not use a higher voltage given the 10V rated power supply caps. Well... assuming these aren't caps for the bias voltage otherwise 12V DC might be good enough for non critical usage (say a P.A. system)
When using it in a hifi system just remove the buffers and bypass the part where the circuit was. Just be careful when paralleling amps that the loud does not become too low in impedance for the connected source.