The early generations of Sonys Xperia Z line, Z and Z1 have extremely unreliable headphone connectors. During my time as a professional service technician this was one of the most common problems on these phones. You will have to replace the headphone connector.
The repair is quite hard if you have not worked with a hot air gun before. If you know your way around one, it is quite easy as the phone is very modular and comes apart easy once you have removed the back cover.
Guide; [guide|22791]