For me its difficult to quit. If I don't do any moderate/intense physical activity for a day, I become unbelievably restless and all I can think about is going on that next run or hitting the gym.
Basically its just another habit you need to do consistently if you want to be able to stick with it long term.