The contacts in the switch get sloppy over time. Best to leave it in full power all the time anyway.
Since you have already pulled the panel, take the switch apart and tweek the contacts. Pretty easy fix. When you put it back together you will feel the difference in the stops in the switch.
