The same thing happened in my old job. As a nonsmoker it drove me crazy. I tried to tell myself that I'd live 10-20 years longer (on average) than the people taking the breaks so it would even out eventually. But it's very unfair. It's like you're being punished for not having this destructive, deadly habit.