One thing is the impedance adapter reduces the damping factor as the impedance is added to the source's side. Might be bad if you had high output impedance DAPs (read that some ipod had 5ohm output). General rule says 10:1 is good enough. Lower damping factor means amp have lower ability to control the diaphragm especially in the bass region. Damping factor is not the same with every frequency too, before and after adding the adapter.

2nd thing is if i'm not mistaken the impedance adapter attenuates different frequency differently, thus results in different sound signature.