The ribbon cable connecting the minus button of the Joy-Con to the motherboard runs over the back of the joystick. It is possible to remove the joystick without disconnecting this cable, but it's much easier to do so when it is disconnected and out of the way.

Unlock the minus button’s ZIF connector and disconnect the cable.