Maybe there is a reason why he keeps his comments to himself when he is around you. Perhaps he knows it will bother you so he is choosing to respect your beliefs and not make jokes.

I say leave it alone until he does something in front of you.