with low impedance sensitive headphones like IEMs, yah, i don't bother with anything more complicated than the headphone jack or cheap dongle on a battery powered player. it's less true now that Apple has beefed up their internal headphone drivers, but for high impedance and lower sensitivity headphones, a small amplifier has better dynamics, so there is still a place for headphone amps.
also, to get less distortion at the D/A stage, you need to put something between your outboard D/A stage and the headphones. enter the headphone amp.
i agree there's an outsized focus on power and, for the subjectivists, other flowery terms, attached to headphone amps, but in reality it's more that you need something that is low distortion and provides enough power at the expected distortion levels, and that is it. people shouldn't have 10 HPAs looking for something with the right clarity/air/musicality/whatever. one also doesn't need 4W at 600o for any reasonable headphone.