First of all, they shouldn't be changing the schedule like that. Once it's made and posted, it should be permanent, because obviously people are going to make plans on their days off. If they are going to make changes, they should make sure that it's okay to put you on that day.

It's not wrong if you don't go in. You have plans, and you shouldn't get fired.