An amplifier works by multiplying the voltage of your audio source(CD, TV, etc...). When this voltage is applied to a load, such as a pair of headphones, then a current runs though the pair of headphones in relation to V=IZ where Z is the impedance. An ideal amp would maintain the exact same voltage regardless if the load is connected or not. However realistically the current causes a drop in voltage which leads to a different current then the one that was intended, this results in distortion. A set of high impedance headphones draws less current then low impedance headphones and therefore doesn't change the signal from the amp as much, which means lower distortion. This change in voltage is associated with a figure called the 'damping factor' which is the ratio of the headphone impedance with the output impedance of the amp.
Of course damping factor can also be improved by lowering the output impedance instead of increasing headphone impedance. However you can only push an amp as long as it can supply current. If you were to continue to lower the impedance Z you reach a point where you really can't get anymore current and wave cuts off. This cutoff is called clipping and it is a serious form of distortion. A high impedance headphone has an advantage in this case as well because of the higher impedance means a lower current draw and delays the onset of clipping.
The downside of high impedance headphones is the lower output for the same voltage. This is really only an issue if the sound level seems way too low even on max volume. In this case you need a more powerful amp to drive the headphones. Along with a limited amount of current, amps also have a maximum voltage they can reach. If you have a low powered amp with high gain then you could reach this maximum which would result in clipping again.
Just play close attention to the specs of your amp and headphones and you can avoid many of these issues.