Mario,
You can bind an autopilot disconnect command on Xbox to stop the alarm, you never have to touch the side stick.
I had to do this with the a310 but I believe the bind is a bit different in the A320.
I currently have "Toggle Auto Pilot Disconnect" and "Set Autopilot Disconnect" bound to the same button on the controller, this allows me to fly the A310 AND the A320 and use the same button to disconnect the autopilot on both and silence the alarm.
You can do what I do and assign both of the above to the same button or I believe just bind "Set Autopilot off" and you can then disconnect the A320 with no alarms.
There's no reason not to bind both toggle autopilot off and set autopilot off on the same button though since this will allow you to disconnect the auto pilot on any high Fidelity aircraft / airliner and silence the alarms, so far this works with the a310, a320, 737, and 146.
Anyway I hope this helps you there and anyone else that's having trouble silencing the autopilot disconnect alarm.