headphone amp distorts at low volume

I connected my phone to a microphone mixer via a stereo mini to 2 male XLR adaptor. The mixers phantom power was on and now my headphone amp doesn't work without distortion until I turn it up past 3/4 volume. Also the ear bud mic and switch don't work. Nethier the speaker or Bluetooth are effected . Can this be fixed? FYI phantom power places the negative on the ground pin and the positive appears on pins 2&3 on the xlr connector.